Sep 10 14:17:43.493805 ip-10-0-130-249 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 14:17:43.493816 ip-10-0-130-249 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 14:17:43.493823 ip-10-0-130-249 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 14:17:43.493975 ip-10-0-130-249 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 14:17:54.816799 ip-10-0-130-249 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 10 14:17:54.816818 ip-10-0-130-249 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 5ef7e4bf410642af9956e3c6124b647e --
Sep 10 14:19:47.853987 ip-10-0-130-249 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 14:19:48.193078 ip-10-0-130-249 kubenswrapper[2547]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 14:19:48.193078 ip-10-0-130-249 kubenswrapper[2547]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 14:19:48.193078 ip-10-0-130-249 kubenswrapper[2547]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 14:19:48.193078 ip-10-0-130-249 kubenswrapper[2547]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 14:19:48.193078 ip-10-0-130-249 kubenswrapper[2547]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 14:19:48.195378 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.195290    2547 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 14:19:48.200956 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.200930    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:19:48.200956 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.200951    2547 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:19:48.200956 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.200956    2547 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:19:48.200956 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.200960    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:19:48.200956 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.200963    2547 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:19:48.201239 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.200968    2547 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:19:48.201239 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.200973    2547 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:19:48.201239 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.200977    2547 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:19:48.201239 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.200981    2547 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:19:48.201239 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.200986    2547 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:19:48.201239 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.200990    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:19:48.201239 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.200996    2547 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:19:48.201239 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201002    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:19:48.201239 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201007    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:19:48.201239 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201011    2547 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:19:48.201239 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201015    2547 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:19:48.201239 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201019    2547 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:19:48.201239 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201023    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:19:48.201239 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201027    2547 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:19:48.201239 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201031    2547 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:19:48.201239 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201035    2547 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:19:48.201239 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201039    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:19:48.201239 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201043    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:19:48.201239 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201047    2547 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:19:48.201239 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201051    2547 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:19:48.202073 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201054    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:19:48.202073 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201058    2547 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:19:48.202073 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201062    2547 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:19:48.202073 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201066    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:19:48.202073 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201070    2547 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:19:48.202073 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201074    2547 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:19:48.202073 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201078    2547 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:19:48.202073 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201084    2547 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:19:48.202073 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201088    2547 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:19:48.202073 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201092    2547 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:19:48.202073 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201098    2547 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:19:48.202073 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201102    2547 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:19:48.202073 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201106    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:19:48.202073 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201111    2547 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:19:48.202073 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201115    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:19:48.202073 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201119    2547 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:19:48.202073 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201123    2547 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:19:48.202073 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201127    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:19:48.202073 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201134    2547 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:19:48.202907 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201142    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:19:48.202907 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201147    2547 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:19:48.202907 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201151    2547 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:19:48.202907 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201156    2547 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:19:48.202907 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201161    2547 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:19:48.202907 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201165    2547 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:19:48.202907 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201171    2547 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:19:48.202907 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201176    2547 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:19:48.202907 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201181    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:19:48.202907 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201185    2547 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:19:48.202907 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201190    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:19:48.202907 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201194    2547 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:19:48.202907 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201198    2547 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:19:48.202907 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201202    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:19:48.202907 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201206    2547 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:19:48.202907 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201210    2547 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:19:48.202907 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201215    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:19:48.202907 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201219    2547 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:19:48.202907 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201223    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:19:48.202907 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201228    2547 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:19:48.203766 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201232    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:19:48.203766 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201236    2547 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:19:48.203766 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201240    2547 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:19:48.203766 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201244    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:19:48.203766 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201248    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:19:48.203766 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201252    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:19:48.203766 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201257    2547 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:19:48.203766 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201261    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:19:48.203766 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201291    2547 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:19:48.203766 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201298    2547 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:19:48.203766 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201302    2547 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:19:48.203766 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201308    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:19:48.203766 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201313    2547 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:19:48.203766 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201317    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:19:48.203766 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201322    2547 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:19:48.203766 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201326    2547 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:19:48.203766 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201331    2547 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:19:48.203766 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201335    2547 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:19:48.203766 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201343    2547 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:19:48.203766 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201350    2547 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:19:48.204642 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201355    2547 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:19:48.204642 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201360    2547 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:19:48.204642 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201365    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:19:48.204642 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201369    2547 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:19:48.204642 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201374    2547 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:19:48.204642 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201378    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:19:48.204642 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201382    2547 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:19:48.204642 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201386    2547 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:19:48.204642 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201391    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:19:48.204642 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201396    2547 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:19:48.204642 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201401    2547 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:19:48.204642 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201405    2547 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:19:48.204642 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201410    2547 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:19:48.204642 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201415    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:19:48.204642 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.201419    2547 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:19:48.204642 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202596    2547 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:19:48.204642 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202607    2547 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:19:48.204642 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202611    2547 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:19:48.204642 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202616    2547 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:19:48.204642 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202620    2547 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:19:48.205500 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202625    2547 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:19:48.205500 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202629    2547 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:19:48.205500 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202633    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:19:48.205500 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202640    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:19:48.205500 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202645    2547 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:19:48.205500 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202649    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:19:48.205500 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202653    2547 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:19:48.205500 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202657    2547 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:19:48.205500 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202661    2547 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:19:48.205500 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202666    2547 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:19:48.205500 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202670    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:19:48.205500 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202674    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:19:48.205500 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202679    2547 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:19:48.205500 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202683    2547 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:19:48.205500 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202688    2547 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:19:48.205500 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202692    2547 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:19:48.205500 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202719    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:19:48.205500 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202723    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:19:48.205500 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202728    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:19:48.205500 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202732    2547 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:19:48.206181 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202736    2547 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:19:48.206181 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202740    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:19:48.206181 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202744    2547 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:19:48.206181 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202748    2547 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:19:48.206181 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202752    2547 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:19:48.206181 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202757    2547 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:19:48.206181 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202761    2547 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:19:48.206181 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202765    2547 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:19:48.206181 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202769    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:19:48.206181 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202776    2547 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:19:48.206181 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202783    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:19:48.206181 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202788    2547 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:19:48.206181 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202793    2547 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:19:48.206181 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202797    2547 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:19:48.206181 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202801    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:19:48.206181 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202806    2547 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:19:48.206181 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202811    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:19:48.206181 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202815    2547 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:19:48.206181 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202820    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:19:48.206762 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202824    2547 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:19:48.206762 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202828    2547 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:19:48.206762 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202833    2547 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:19:48.206762 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202837    2547 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:19:48.206762 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202841    2547 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:19:48.206762 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202845    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:19:48.206762 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202850    2547 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:19:48.206762 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202855    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:19:48.206762 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202859    2547 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:19:48.206762 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202863    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:19:48.206762 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202867    2547 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:19:48.206762 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202871    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:19:48.206762 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202875    2547 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:19:48.206762 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202880    2547 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:19:48.206762 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202885    2547 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:19:48.206762 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202889    2547 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:19:48.206762 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202893    2547 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:19:48.206762 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202897    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:19:48.206762 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202901    2547 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:19:48.206762 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202905    2547 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:19:48.207262 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202909    2547 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:19:48.207262 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202913    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:19:48.207262 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202917    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:19:48.207262 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202921    2547 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:19:48.207262 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202925    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:19:48.207262 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202929    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:19:48.207262 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202933    2547 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:19:48.207262 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202937    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:19:48.207262 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202941    2547 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:19:48.207262 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202947    2547 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:19:48.207262 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202952    2547 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:19:48.207262 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202956    2547 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:19:48.207262 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202960    2547 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:19:48.207262 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202964    2547 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:19:48.207262 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202968    2547 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:19:48.207262 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202973    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:19:48.207262 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202979    2547 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:19:48.207262 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202986    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:19:48.207262 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202991    2547 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:19:48.207945 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.202997    2547 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:19:48.207945 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203003    2547 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:19:48.207945 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203008    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:19:48.207945 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203013    2547 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:19:48.207945 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203018    2547 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:19:48.207945 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203023    2547 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:19:48.207945 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203027    2547 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:19:48.207945 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203032    2547 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:19:48.207945 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203037    2547 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:19:48.207945 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203041    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:19:48.207945 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203045    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:19:48.207945 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203049    2547 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:19:48.207945 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203053    2547 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:19:48.207945 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203057    2547 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:19:48.207945 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203061    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:19:48.207945 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203065    2547 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:19:48.207945 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203676    2547 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:19:48.207945 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203684    2547 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:19:48.207945 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203688    2547 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:19:48.208421 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203710    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:19:48.208421 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203715    2547 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:19:48.208421 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203719    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:19:48.208421 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203724    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:19:48.208421 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203728    2547 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:19:48.208421 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203731    2547 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:19:48.208421 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203736    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:19:48.208421 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203740    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:19:48.208421 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203744    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:19:48.208421 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203748    2547 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:19:48.208421 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203753    2547 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:19:48.208421 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203757    2547 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:19:48.208421 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203764    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:19:48.208421 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203770    2547 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:19:48.208421 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203774    2547 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:19:48.208421 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203777    2547 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:19:48.208421 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203782    2547 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:19:48.208421 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203786    2547 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:19:48.208421 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203790    2547 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:19:48.208421 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203793    2547 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:19:48.209046 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203797    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:19:48.209046 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203801    2547 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:19:48.209046 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203806    2547 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:19:48.209046 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203810    2547 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:19:48.209046 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203814    2547 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:19:48.209046 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203819    2547 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:19:48.209046 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203823    2547 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:19:48.209046 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203827    2547 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:19:48.209046 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203832    2547 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:19:48.209046 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203836    2547 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:19:48.209046 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203840    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:19:48.209046 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203844    2547 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:19:48.209046 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203848    2547 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:19:48.209046 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203853    2547 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:19:48.209046 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203857    2547 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:19:48.209046 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203861    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:19:48.209046 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203865    2547 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:19:48.209046 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203869    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:19:48.209046 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203873    2547 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:19:48.209046 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203877    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:19:48.209541 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203881    2547 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:19:48.209541 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203885    2547 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:19:48.209541 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203890    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:19:48.209541 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203895    2547 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:19:48.209541 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203901    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:19:48.209541 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203905    2547 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:19:48.209541 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203910    2547 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:19:48.209541 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203915    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:19:48.209541 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203919    2547 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:19:48.209541 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203923    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:19:48.209541 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203927    2547 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:19:48.209541 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203933    2547 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:19:48.209541 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203937    2547 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:19:48.209541 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203941    2547 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:19:48.209541 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203945    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:19:48.209541 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203950    2547 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:19:48.209541 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203954    2547 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:19:48.209541 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203958    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:19:48.209541 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203962    2547 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:19:48.209541 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203966    2547 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:19:48.210096 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203970    2547 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:19:48.210096 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203974    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:19:48.210096 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203978    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:19:48.210096 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203983    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:19:48.210096 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203987    2547 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:19:48.210096 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203992    2547 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:19:48.210096 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.203996    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:19:48.210096 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204000    2547 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:19:48.210096 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204004    2547 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:19:48.210096 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204008    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:19:48.210096 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204012    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:19:48.210096 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204016    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:19:48.210096 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204023    2547 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:19:48.210096 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204029    2547 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:19:48.210096 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204034    2547 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:19:48.210096 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204040    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:19:48.210096 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204046    2547 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:19:48.210096 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204051    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:19:48.210096 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204055    2547 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:19:48.210560 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204060    2547 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:19:48.210560 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204065    2547 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:19:48.210560 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204070    2547 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:19:48.210560 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204074    2547 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:19:48.210560 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204078    2547 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:19:48.210560 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204082    2547 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:19:48.210560 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204088    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:19:48.210560 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204093    2547 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:19:48.210560 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204097    2547 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:19:48.210560 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204101    2547 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:19:48.210560 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204107    2547 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:19:48.210560 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204113    2547 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:19:48.210560 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204117    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:19:48.210560 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204121    2547 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:19:48.210560 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204125    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:19:48.210560 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204129    2547 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:19:48.210560 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.204135    2547 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:19:48.210560 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204244    2547 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 14:19:48.210560 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204300    2547 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 14:19:48.210560 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204311    2547 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 14:19:48.211052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204318    2547 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 14:19:48.211052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204324    2547 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 14:19:48.211052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204329    2547 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 14:19:48.211052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204335    2547 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 14:19:48.211052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204343    2547 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 14:19:48.211052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204348    2547 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 14:19:48.211052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204353    2547 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 14:19:48.211052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204359    2547 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 14:19:48.211052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204366    2547 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 14:19:48.211052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204374    2547 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 14:19:48.211052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204379    2547 flags.go:64] FLAG: --cgroup-root=""
Sep 10 14:19:48.211052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204384    2547 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 14:19:48.211052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204389    2547 flags.go:64] FLAG: --client-ca-file=""
Sep 10 14:19:48.211052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204394    2547 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 14:19:48.211052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204399    2547 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 14:19:48.211052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204405    2547 flags.go:64] FLAG: --cluster-domain=""
Sep 10 14:19:48.211052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204410    2547 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 14:19:48.211052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204414    2547 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 14:19:48.211052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204419    2547 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 14:19:48.211052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204424    2547 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 14:19:48.211052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204439    2547 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 14:19:48.211052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204444    2547 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 14:19:48.211052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204449    2547 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204455    2547 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204459    2547 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204464    2547 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204468    2547 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204473    2547 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204479    2547 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204485    2547 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204491    2547 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204495    2547 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204500    2547 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204505    2547 flags.go:64] FLAG: --enable-server="true"
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204509    2547 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204517    2547 flags.go:64] FLAG: --event-burst="100"
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204521    2547 flags.go:64] FLAG: --event-qps="50"
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204526    2547 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204531    2547 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204536    2547 flags.go:64] FLAG: --eviction-hard=""
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204544    2547 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204551    2547 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204555    2547 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204561    2547 flags.go:64] FLAG: --eviction-soft=""
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204565    2547 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204570    2547 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204584    2547 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 14:19:48.211820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204588    2547 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 14:19:48.212452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204593    2547 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 14:19:48.212452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204598    2547 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 14:19:48.212452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204602    2547 flags.go:64] FLAG: --feature-gates=""
Sep 10 14:19:48.212452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204608    2547 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 14:19:48.212452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204613    2547 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 14:19:48.212452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204618    2547 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 14:19:48.212452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204624    2547 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 14:19:48.212452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204629    2547 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 14:19:48.212452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204633    2547 flags.go:64] FLAG: --help="false"
Sep 10 14:19:48.212452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204638    2547 flags.go:64] FLAG: --hostname-override="ip-10-0-130-249.ec2.internal"
Sep 10 14:19:48.212452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204643    2547 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 14:19:48.212452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204647    2547 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 14:19:48.212452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204651    2547 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 14:19:48.212452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204656    2547 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 14:19:48.212452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204662    2547 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 14:19:48.212452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204666    2547 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 14:19:48.212452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204670    2547 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 14:19:48.212452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204676    2547 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 14:19:48.212452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204680    2547 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 14:19:48.212452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204685    2547 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 14:19:48.212452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204690    2547 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 14:19:48.212452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204712    2547 flags.go:64] FLAG: --kube-reserved=""
Sep 10 14:19:48.212452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204717    2547 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 14:19:48.212452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204722    2547 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204728    2547 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204734    2547 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204739    2547 flags.go:64] FLAG: --lock-file=""
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204743    2547 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204748    2547 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204753    2547 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204762    2547 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204768    2547 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204773    2547 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204777    2547 flags.go:64] FLAG: --logging-format="text"
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204783    2547 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204788    2547 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204793    2547 flags.go:64] FLAG: --manifest-url=""
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204797    2547 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204804    2547 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204809    2547 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204815    2547 flags.go:64] FLAG: --max-pods="110"
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204820    2547 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204825    2547 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204830    2547 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204835    2547 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204840    2547 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204844    2547 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204849    2547 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 14:19:48.213065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204860    2547 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204865    2547 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204870    2547 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204875    2547 flags.go:64] FLAG: --pod-cidr=""
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204880    2547 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204888    2547 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204893    2547 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204897    2547 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204904    2547 flags.go:64] FLAG: --port="10250"
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204909    2547 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204915    2547 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00ff9e36109bc303c"
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204921    2547 flags.go:64] FLAG: --qos-reserved=""
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204925    2547 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204931    2547 flags.go:64] FLAG: --register-node="true"
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204935    2547 flags.go:64] FLAG: --register-with-taints=""
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204941    2547 flags.go:64] FLAG: --registry-burst="10"
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204951    2547 flags.go:64] FLAG: --registry-qps="5"
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204956    2547 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204961    2547 flags.go:64] FLAG: --reserved-memory=""
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204966    2547 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204972    2547 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204977    2547 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204982    2547 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204987    2547 flags.go:64] FLAG: --runonce="false"
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204992    2547 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 14:19:48.213679 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.204997    2547 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205002    2547 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205007    2547 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205011    2547 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205016    2547 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205021    2547 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205026    2547 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205030    2547 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205035    2547 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205040    2547 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205044    2547 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205049    2547 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205054    2547 flags.go:64] FLAG: --system-cgroups=""
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205058    2547 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205069    2547 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205075    2547 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205080    2547 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205087    2547 flags.go:64] FLAG: --tls-min-version=""
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205092    2547 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205096    2547 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205101    2547 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205106    2547 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205111    2547 flags.go:64] FLAG: --v="2"
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205117    2547 flags.go:64] FLAG: --version="false"
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205125    2547 flags.go:64] FLAG: --vmodule=""
Sep 10 14:19:48.214312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205131    2547 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 14:19:48.214990 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.205136    2547 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 14:19:48.214990 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205319    2547 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:19:48.214990 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205325    2547 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:19:48.214990 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205329    2547 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:19:48.214990 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205333    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:19:48.214990 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205338    2547 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:19:48.214990 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205342    2547 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:19:48.214990 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205346    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:19:48.214990 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205350    2547 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:19:48.214990 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205354    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:19:48.214990 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205358    2547 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:19:48.214990 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205363    2547 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:19:48.214990 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205367    2547 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:19:48.214990 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205371    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:19:48.214990 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205378    2547 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:19:48.214990 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205384    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:19:48.214990 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205389    2547 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:19:48.214990 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205394    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:19:48.214990 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205398    2547 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:19:48.214990 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205402    2547 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:19:48.215475 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205406    2547 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:19:48.215475 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205412    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:19:48.215475 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205416    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:19:48.215475 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205422    2547 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:19:48.215475 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205426    2547 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:19:48.215475 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205431    2547 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:19:48.215475 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205435    2547 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:19:48.215475 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205439    2547 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:19:48.215475 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205444    2547 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:19:48.215475 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205448    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:19:48.215475 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205452    2547 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:19:48.215475 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205457    2547 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:19:48.215475 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205461    2547 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:19:48.215475 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205466    2547 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:19:48.215475 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205470    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:19:48.215475 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205474    2547 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:19:48.215475 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205478    2547 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:19:48.215475 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205482    2547 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:19:48.215475 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205486    2547 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:19:48.215475 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205490    2547 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:19:48.216008 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205494    2547 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:19:48.216008 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205498    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:19:48.216008 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205501    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:19:48.216008 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205505    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:19:48.216008 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205509    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:19:48.216008 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205514    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:19:48.216008 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205518    2547 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:19:48.216008 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205522    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:19:48.216008 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205526    2547 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:19:48.216008 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205530    2547 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:19:48.216008 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205534    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:19:48.216008 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205538    2547 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:19:48.216008 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205542    2547 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:19:48.216008 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205549    2547 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:19:48.216008 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205554    2547 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:19:48.216008 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205559    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:19:48.216008 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205563    2547 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:19:48.216008 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205567    2547 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:19:48.216008 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205571    2547 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:19:48.216008 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205575    2547 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:19:48.216499 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205579    2547 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:19:48.216499 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205583    2547 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:19:48.216499 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205587    2547 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:19:48.216499 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205592    2547 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:19:48.216499 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205597    2547 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:19:48.216499 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205602    2547 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:19:48.216499 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205606    2547 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:19:48.216499 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205609    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:19:48.216499 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205614    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:19:48.216499 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205618    2547 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:19:48.216499 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205622    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:19:48.216499 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205626    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:19:48.216499 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205630    2547 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:19:48.216499 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205635    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:19:48.216499 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205639    2547 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:19:48.216499 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205643    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:19:48.216499 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205647    2547 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:19:48.216499 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205651    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:19:48.216499 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205655    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:19:48.216499 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205659    2547 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:19:48.217012 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205663    2547 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:19:48.217012 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205667    2547 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:19:48.217012 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205671    2547 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:19:48.217012 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205676    2547 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:19:48.217012 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205680    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:19:48.217012 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205686    2547 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:19:48.217012 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205690    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:19:48.217012 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205713    2547 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:19:48.217012 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205718    2547 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:19:48.217012 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205722    2547 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:19:48.217012 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205728    2547 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:19:48.217012 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205734    2547 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:19:48.217012 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205739    2547 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:19:48.217012 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205743    2547 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:19:48.217012 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205747    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:19:48.217012 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205751    2547 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:19:48.217012 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205758    2547 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:19:48.217012 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205765    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:19:48.217453 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205771    2547 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:19:48.217453 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.205776    2547 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:19:48.217453 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.206333    2547 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 14:19:48.217453 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.213492    2547 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 14:19:48.217453 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.213511    2547 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 14:19:48.217453 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.214133    2547 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 14:19:48.217453 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.214140    2547 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 14:19:48.217453 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214198    2547 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:19:48.217453 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214203    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:19:48.217453 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214206    2547 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:19:48.217453 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214209    2547 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:19:48.217453 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214212    2547 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:19:48.217453 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214215    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:19:48.217453 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214218    2547 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:19:48.217453 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214221    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:19:48.217843 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214223    2547 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:19:48.217843 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214226    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:19:48.217843 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214228    2547 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:19:48.217843 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214231    2547 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:19:48.217843 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214234    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:19:48.217843 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214236    2547 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:19:48.217843 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214239    2547 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:19:48.217843 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214241    2547 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:19:48.217843 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214246    2547 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:19:48.217843 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214249    2547 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:19:48.217843 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214253    2547 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:19:48.217843 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214256    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:19:48.217843 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214259    2547 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:19:48.217843 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214262    2547 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:19:48.217843 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214265    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:19:48.217843 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214268    2547 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:19:48.217843 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214270    2547 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:19:48.217843 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214273    2547 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:19:48.217843 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214275    2547 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:19:48.217843 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214278    2547 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:19:48.218358 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214280    2547 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:19:48.218358 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214285    2547 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:19:48.218358 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214288    2547 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:19:48.218358 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214290    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:19:48.218358 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214293    2547 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:19:48.218358 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214296    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:19:48.218358 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214298    2547 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:19:48.218358 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214301    2547 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:19:48.218358 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214304    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:19:48.218358 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214306    2547 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:19:48.218358 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214309    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:19:48.218358 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214312    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:19:48.218358 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214314    2547 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:19:48.218358 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214317    2547 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:19:48.218358 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214319    2547 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:19:48.218358 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214322    2547 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:19:48.218358 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214325    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:19:48.218358 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214327    2547 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:19:48.218358 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214330    2547 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:19:48.218837 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214334    2547 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:19:48.218837 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214337    2547 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:19:48.218837 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214340    2547 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:19:48.218837 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214343    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:19:48.218837 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214347    2547 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:19:48.218837 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214350    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:19:48.218837 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214354    2547 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:19:48.218837 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214356    2547 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:19:48.218837 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214359    2547 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:19:48.218837 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214362    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:19:48.218837 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214364    2547 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:19:48.218837 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214367    2547 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:19:48.218837 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214369    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:19:48.218837 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214372    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:19:48.218837 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214376    2547 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:19:48.218837 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214379    2547 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:19:48.218837 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214381    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:19:48.218837 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214384    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:19:48.218837 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214387    2547 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:19:48.219294 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214389    2547 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:19:48.219294 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214392    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:19:48.219294 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214394    2547 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:19:48.219294 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214397    2547 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:19:48.219294 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214399    2547 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:19:48.219294 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214402    2547 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:19:48.219294 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214404    2547 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:19:48.219294 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214407    2547 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:19:48.219294 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214410    2547 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:19:48.219294 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214412    2547 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:19:48.219294 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214414    2547 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:19:48.219294 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214417    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:19:48.219294 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214419    2547 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:19:48.219294 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214422    2547 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:19:48.219294 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214424    2547 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:19:48.219294 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214430    2547 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:19:48.219294 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214432    2547 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:19:48.219294 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214435    2547 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:19:48.219294 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214437    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:19:48.219294 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214440    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:19:48.219868 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214443    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:19:48.219868 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214445    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:19:48.219868 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214448    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:19:48.219868 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214451    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:19:48.219868 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214453    2547 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:19:48.219868 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214456    2547 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:19:48.219868 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214458    2547 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:19:48.219868 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214461    2547 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:19:48.219868 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214464    2547 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:19:48.219868 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214468    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:19:48.219868 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214470    2547 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:19:48.219868 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214473    2547 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:19:48.219868 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214475    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:19:48.219868 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.214480    2547 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 14:19:48.219868 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214587    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:19:48.220231 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214591    2547 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:19:48.220231 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214595    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:19:48.220231 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214599    2547 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:19:48.220231 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214602    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:19:48.220231 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214605    2547 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:19:48.220231 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214608    2547 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:19:48.220231 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214611    2547 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:19:48.220231 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214614    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:19:48.220231 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214617    2547 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:19:48.220231 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214620    2547 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:19:48.220231 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214622    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:19:48.220231 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214625    2547 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:19:48.220231 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214628    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:19:48.220231 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214630    2547 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:19:48.220231 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214633    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:19:48.220231 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214636    2547 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:19:48.220231 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214638    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:19:48.220231 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214641    2547 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:19:48.220231 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214644    2547 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:19:48.220231 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214647    2547 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:19:48.220738 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214650    2547 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:19:48.220738 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214654    2547 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:19:48.220738 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214657    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:19:48.220738 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214660    2547 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:19:48.220738 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214663    2547 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:19:48.220738 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214665    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:19:48.220738 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214668    2547 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:19:48.220738 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214671    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:19:48.220738 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214673    2547 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:19:48.220738 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214676    2547 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:19:48.220738 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214678    2547 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:19:48.220738 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214681    2547 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:19:48.220738 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214684    2547 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:19:48.220738 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214686    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:19:48.220738 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214688    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:19:48.220738 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214691    2547 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:19:48.220738 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214709    2547 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:19:48.220738 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214713    2547 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:19:48.220738 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214717    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:19:48.221200 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214721    2547 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:19:48.221200 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214725    2547 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:19:48.221200 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214729    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:19:48.221200 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214735    2547 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:19:48.221200 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214739    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:19:48.221200 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214742    2547 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:19:48.221200 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214745    2547 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:19:48.221200 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214749    2547 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:19:48.221200 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214752    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:19:48.221200 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214754    2547 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:19:48.221200 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214757    2547 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:19:48.221200 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214760    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:19:48.221200 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214762    2547 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:19:48.221200 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214765    2547 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:19:48.221200 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214768    2547 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:19:48.221200 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214770    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:19:48.221200 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214773    2547 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:19:48.221200 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214776    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:19:48.221200 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214779    2547 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:19:48.221654 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214782    2547 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:19:48.221654 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214784    2547 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:19:48.221654 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214788    2547 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:19:48.221654 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214790    2547 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:19:48.221654 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214793    2547 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:19:48.221654 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214796    2547 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:19:48.221654 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214798    2547 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:19:48.221654 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214801    2547 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:19:48.221654 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214804    2547 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:19:48.221654 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214806    2547 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:19:48.221654 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214808    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:19:48.221654 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214811    2547 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:19:48.221654 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214813    2547 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:19:48.221654 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214816    2547 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:19:48.221654 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214819    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:19:48.221654 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214821    2547 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:19:48.221654 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214823    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:19:48.221654 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214826    2547 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:19:48.221654 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214828    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:19:48.221654 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214831    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:19:48.222331 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214834    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:19:48.222331 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214836    2547 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:19:48.222331 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214839    2547 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:19:48.222331 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214842    2547 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:19:48.222331 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214844    2547 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:19:48.222331 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214847    2547 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:19:48.222331 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214850    2547 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:19:48.222331 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214852    2547 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:19:48.222331 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214855    2547 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:19:48.222331 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214857    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:19:48.222331 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214860    2547 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:19:48.222331 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214863    2547 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:19:48.222331 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214865    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:19:48.222331 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214867    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:19:48.222331 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214870    2547 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:19:48.222331 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214873    2547 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:19:48.222331 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214875    2547 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:19:48.222331 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214878    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:19:48.222331 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214881    2547 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:19:48.222331 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:48.214883    2547 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:19:48.223008 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.214888    2547 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 14:19:48.223008 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.214990    2547 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 14:19:48.223008 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.217002    2547 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 14:19:48.223008 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.217887    2547 server.go:1019] "Starting client certificate rotation"
Sep 10 14:19:48.223008 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.217988    2547 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 14:19:48.223008 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.218053    2547 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 14:19:48.223008 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.220783    2547 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 14:19:48.226146 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.226126    2547 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 14:19:48.238181 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.238160    2547 log.go:25] "Validated CRI v1 runtime API"
Sep 10 14:19:48.240768 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.240753    2547 log.go:25] "Validated CRI v1 image API"
Sep 10 14:19:48.241592 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.241577    2547 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 14:19:48.244045 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.244026    2547 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 14:19:48.246394 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.246377    2547 fs.go:135] Filesystem UUIDs: map[108db183-034d-485a-9e62-6e27777cf588:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 917cfb8d-823f-4d7f-b368-7b8066a77be1:/dev/nvme0n1p4]
Sep 10 14:19:48.246456 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.246394    2547 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 14:19:48.252175 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.252072    2547 manager.go:216] Machine: {Timestamp:2026-09-10 14:19:48.250819435 +0000 UTC m=+0.292693909 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3081990 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cebcd610abb32a12405a3f5b9de43 SystemUUID:ec2cebcd-610a-bb32-a124-05a3f5b9de43 BootID:5ef7e4bf-4106-42af-9956-e3c6124b647e Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:99:68:7a:fe:bd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:99:68:7a:fe:bd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:66:cf:d8:98:07 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 14:19:48.252175 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.252174    2547 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 14:19:48.252282 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.252255    2547 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 14:19:48.255286 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.255255    2547 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 14:19:48.255433 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.255289    2547 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-249.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 14:19:48.256301 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.256290    2547 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 14:19:48.256340 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.256304    2547 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 14:19:48.256340 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.256311    2547 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 14:19:48.256905 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.256889    2547 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 14:19:48.258018 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.258008    2547 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 14:19:48.260474 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.260463    2547 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 14:19:48.260593 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.260584    2547 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 14:19:48.260630 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.260616    2547 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 14:19:48.260630 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.260625    2547 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 14:19:48.260683 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.260646    2547 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 14:19:48.260683 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.260655    2547 kubelet.go:398] "Adding apiserver pod source"
Sep 10 14:19:48.260683 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.260664    2547 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 14:19:48.261310 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.261295    2547 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 14:19:48.261387 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.261322    2547 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 14:19:48.263772 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.263752    2547 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 14:19:48.265727 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.265713    2547 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 14:19:48.265778 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.265744    2547 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 14:19:48.265894 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.265884    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 14:19:48.265927 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.265899    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 14:19:48.265927 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.265906    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 14:19:48.265927 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.265911    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 14:19:48.265927 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.265918    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 14:19:48.265927 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.265923    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 14:19:48.266494 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.266485    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 14:19:48.266494 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.266494    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 14:19:48.266546 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.266501    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 14:19:48.266546 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.266506    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 14:19:48.266546 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.266514    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 14:19:48.266546 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.266521    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 14:19:48.268061 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.268051    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 14:19:48.268061 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.268061    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 14:19:48.270760 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.270742    2547 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-249.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 14:19:48.271313 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:48.271288    2547 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-249.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 14:19:48.271354 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:48.271304    2547 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 14:19:48.271880 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.271868    2547 server.go:1268] "Started kubelet"
Sep 10 14:19:48.271978 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.271934    2547 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 14:19:48.272308 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.272285    2547 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 14:19:48.272371 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.272329    2547 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 14:19:48.272819 ip-10-0-130-249 systemd[1]: Started Kubernetes Kubelet.
Sep 10 14:19:48.273261 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.273243    2547 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 14:19:48.277990 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.277969    2547 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 14:19:48.281195 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:48.281046    2547 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 14:19:48.281279 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.281259    2547 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 14:19:48.281825 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.281808    2547 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 14:19:48.281994 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:48.281969    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-249.ec2.internal\" not found"
Sep 10 14:19:48.282588 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.282569    2547 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 14:19:48.282588 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.282586    2547 factory.go:55] Registering systemd factory
Sep 10 14:19:48.282740 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.282594    2547 factory.go:223] Registration of the systemd container factory successfully
Sep 10 14:19:48.282740 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.282595    2547 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 14:19:48.282740 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.282596    2547 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 14:19:48.282740 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.282622    2547 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 14:19:48.282740 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.282673    2547 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 14:19:48.282740 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.282681    2547 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 14:19:48.282968 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.282825    2547 factory.go:153] Registering CRI-O factory
Sep 10 14:19:48.282968 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.282841    2547 factory.go:223] Registration of the crio container factory successfully
Sep 10 14:19:48.282968 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.282858    2547 factory.go:103] Registering Raw factory
Sep 10 14:19:48.282968 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.282874    2547 manager.go:1196] Started watching for new ooms in manager
Sep 10 14:19:48.283607 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.283591    2547 manager.go:349] Starting recovery of all containers
Sep 10 14:19:48.288988 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:48.288963    2547 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 14:19:48.289215 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:48.289187    2547 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-249.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 14:19:48.289963 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:48.289050    2547 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-249.ec2.internal.18d3fb7c4c5c2003  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-249.ec2.internal,UID:ip-10-0-130-249.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-249.ec2.internal,},FirstTimestamp:2026-09-10 14:19:48.271841283 +0000 UTC m=+0.313715755,LastTimestamp:2026-09-10 14:19:48.271841283 +0000 UTC m=+0.313715755,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-249.ec2.internal,}"
Sep 10 14:19:48.293283 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.293266    2547 manager.go:354] Recovery completed
Sep 10 14:19:48.295947 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.295929    2547 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9b52s"
Sep 10 14:19:48.297926 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.297915    2547 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:19:48.299816 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.299800    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-249.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:19:48.299916 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.299834    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-249.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:19:48.299916 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.299849    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-249.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:19:48.300363 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.300347    2547 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 14:19:48.300449 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.300363    2547 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 14:19:48.300449 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.300373    2547 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:19:48.300449 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.300381    2547 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 14:19:48.301471 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.301457    2547 policy_none.go:49] "None policy: Start"
Sep 10 14:19:48.301536 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.301476    2547 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 14:19:48.301536 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.301489    2547 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 14:19:48.301631 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:48.301552    2547 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-249.ec2.internal.18d3fb7c4e06fb54  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-249.ec2.internal,UID:ip-10-0-130-249.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-249.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-249.ec2.internal,},FirstTimestamp:2026-09-10 14:19:48.299815764 +0000 UTC m=+0.341690238,LastTimestamp:2026-09-10 14:19:48.299815764 +0000 UTC m=+0.341690238,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-249.ec2.internal,}"
Sep 10 14:19:48.301731 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.301715    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-249.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:19:48.301781 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.301742    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-249.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:19:48.301781 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.301757    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-249.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:19:48.302396 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.302381    2547 policy_none.go:47] "Start"
Sep 10 14:19:48.303730 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.303711    2547 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9b52s"
Sep 10 14:19:48.327262 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.327244    2547 manager.go:337] "Starting Device Plugin manager"
Sep 10 14:19:48.327346 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:48.327296    2547 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 14:19:48.327346 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.327310    2547 server.go:87] "Starting device plugin registration server"
Sep 10 14:19:48.327636 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.327623    2547 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 14:19:48.327723 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.327638    2547 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 14:19:48.327925 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.327828    2547 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 14:19:48.327925 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.327911    2547 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 14:19:48.327925 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.327920    2547 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 14:19:48.328471 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:48.328443    2547 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 14:19:48.328546 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:48.328494    2547 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-249.ec2.internal\" not found"
Sep 10 14:19:48.413523 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.413488    2547 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 14:19:48.414672 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.414642    2547 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 14:19:48.414773 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.414709    2547 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 14:19:48.414773 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.414741    2547 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 14:19:48.414866 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:48.414787    2547 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 14:19:48.417051 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.417029    2547 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:19:48.428578 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.428558    2547 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:19:48.429377 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.429362    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-249.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:19:48.429449 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.429390    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-249.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:19:48.429449 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.429400    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-249.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:19:48.429449 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.429421    2547 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-249.ec2.internal"
Sep 10 14:19:48.436449 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.436435    2547 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-249.ec2.internal"
Sep 10 14:19:48.436500 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:48.436456    2547 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-249.ec2.internal\": node \"ip-10-0-130-249.ec2.internal\" not found"
Sep 10 14:19:48.471803 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:48.471736    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-249.ec2.internal\" not found"
Sep 10 14:19:48.515578 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.515554    2547 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-249.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-249.ec2.internal"]
Sep 10 14:19:48.515673 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.515615    2547 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:19:48.517894 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.517878    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-249.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:19:48.517982 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.517903    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-249.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:19:48.517982 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.517913    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-249.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:19:48.519535 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.519524    2547 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:19:48.519652 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.519639    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-249.ec2.internal"
Sep 10 14:19:48.519692 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.519677    2547 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:19:48.520305 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.520287    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-249.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:19:48.520305 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.520297    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-249.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:19:48.520418 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.520316    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-249.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:19:48.520418 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.520318    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-249.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:19:48.520418 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.520325    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-249.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:19:48.520418 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.520330    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-249.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:19:48.521440 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.521425    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-249.ec2.internal"
Sep 10 14:19:48.521508 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.521449    2547 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:19:48.522168 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.522151    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-249.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:19:48.522230 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.522177    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-249.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:19:48.522230 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.522187    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-249.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:19:48.555457 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:48.555430    2547 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-249.ec2.internal\" not found" node="ip-10-0-130-249.ec2.internal"
Sep 10 14:19:48.559621 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:48.559604    2547 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-249.ec2.internal\" not found" node="ip-10-0-130-249.ec2.internal"
Sep 10 14:19:48.572702 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:48.572683    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-249.ec2.internal\" not found"
Sep 10 14:19:48.584671 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.584651    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f4b9a9d29461bb847438bd99b7b31d4-config\") pod \"kube-apiserver-proxy-ip-10-0-130-249.ec2.internal\" (UID: \"7f4b9a9d29461bb847438bd99b7b31d4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-249.ec2.internal"
Sep 10 14:19:48.584776 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.584675    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/39f59cdebad304ce873337176c339318-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-249.ec2.internal\" (UID: \"39f59cdebad304ce873337176c339318\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-249.ec2.internal"
Sep 10 14:19:48.584776 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.584711    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39f59cdebad304ce873337176c339318-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-249.ec2.internal\" (UID: \"39f59cdebad304ce873337176c339318\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-249.ec2.internal"
Sep 10 14:19:48.673161 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:48.673122    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-249.ec2.internal\" not found"
Sep 10 14:19:48.685533 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.685514    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39f59cdebad304ce873337176c339318-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-249.ec2.internal\" (UID: \"39f59cdebad304ce873337176c339318\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-249.ec2.internal"
Sep 10 14:19:48.685610 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.685542    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f4b9a9d29461bb847438bd99b7b31d4-config\") pod \"kube-apiserver-proxy-ip-10-0-130-249.ec2.internal\" (UID: \"7f4b9a9d29461bb847438bd99b7b31d4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-249.ec2.internal"
Sep 10 14:19:48.685610 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.685558    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/39f59cdebad304ce873337176c339318-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-249.ec2.internal\" (UID: \"39f59cdebad304ce873337176c339318\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-249.ec2.internal"
Sep 10 14:19:48.685728 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.685624    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39f59cdebad304ce873337176c339318-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-249.ec2.internal\" (UID: \"39f59cdebad304ce873337176c339318\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-249.ec2.internal"
Sep 10 14:19:48.685728 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.685635    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f4b9a9d29461bb847438bd99b7b31d4-config\") pod \"kube-apiserver-proxy-ip-10-0-130-249.ec2.internal\" (UID: \"7f4b9a9d29461bb847438bd99b7b31d4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-249.ec2.internal"
Sep 10 14:19:48.685728 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.685631    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/39f59cdebad304ce873337176c339318-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-249.ec2.internal\" (UID: \"39f59cdebad304ce873337176c339318\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-249.ec2.internal"
Sep 10 14:19:48.774001 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:48.773923    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-249.ec2.internal\" not found"
Sep 10 14:19:48.858397 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.858368    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-249.ec2.internal"
Sep 10 14:19:48.862363 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:48.862290    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-249.ec2.internal"
Sep 10 14:19:48.874634 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:48.874612    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-249.ec2.internal\" not found"
Sep 10 14:19:48.975190 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:48.975144    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-249.ec2.internal\" not found"
Sep 10 14:19:49.075759 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:49.075672    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-249.ec2.internal\" not found"
Sep 10 14:19:49.099888 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.099868    2547 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:19:49.123870 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.123847    2547 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:19:49.182240 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.182204    2547 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-249.ec2.internal"
Sep 10 14:19:49.197384 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.197362    2547 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 14:19:49.199404 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.199390    2547 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-249.ec2.internal"
Sep 10 14:19:49.209728 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.209711    2547 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 14:19:49.218014 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.217995    2547 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 14:19:49.218132 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.218114    2547 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 14:19:49.218177 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.218163    2547 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 14:19:49.262811 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.262788    2547 apiserver.go:52] "Watching apiserver"
Sep 10 14:19:49.270022 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.270004    2547 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 14:19:49.270389 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.270371    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-m2v46","kube-system/kube-apiserver-proxy-ip-10-0-130-249.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj","openshift-image-registry/node-ca-46jxd","openshift-multus/multus-additional-cni-plugins-czbln","openshift-multus/multus-gfz6l","openshift-multus/network-metrics-daemon-756dn","openshift-network-diagnostics/network-check-target-d66xs","kube-system/konnectivity-agent-vp8vn","openshift-cluster-node-tuning-operator/tuned-lqlzs","openshift-dns/node-resolver-vz9vp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-249.ec2.internal","openshift-network-operator/iptables-alerter-7xx9q","openshift-ovn-kubernetes/ovnkube-node-4ww4p"]
Sep 10 14:19:49.271873 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.271856    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:19:49.271951 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:49.271934    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-756dn" podUID="c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0"
Sep 10 14:19:49.272842 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.272827    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7xx9q"
Sep 10 14:19:49.274072 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.274051    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj"
Sep 10 14:19:49.275133 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.275117    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-46jxd"
Sep 10 14:19:49.275632 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.275616    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 14:19:49.275729 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.275628    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wksrg\""
Sep 10 14:19:49.275899 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.275880    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 14:19:49.276356 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.276336    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-czbln"
Sep 10 14:19:49.276439 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.276418    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 14:19:49.276439 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.276433    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lh5rx\""
Sep 10 14:19:49.276914 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.276898    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 14:19:49.277269 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.277256    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 14:19:49.277423 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.277408    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.278543 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.278517    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 14:19:49.278642 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.278582    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vp8vn"
Sep 10 14:19:49.282283 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.280395    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t8v7h\""
Sep 10 14:19:49.282283 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.280723    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:19:49.282283 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.280785    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 14:19:49.282283 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.280786    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 14:19:49.282283 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.280919    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 14:19:49.282283 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.281015    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 14:19:49.282283 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.281025    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 14:19:49.282283 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.281485    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 14:19:49.282283 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.281534    2547 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 14:19:49.282283 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.281602    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 14:19:49.282283 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.281791    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2hcfz\""
Sep 10 14:19:49.282283 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.282199    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-shvcc\""
Sep 10 14:19:49.283406 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.283383    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:19:49.283522 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:49.283491    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m2v46" podUID="4e33264c-464d-41c9-8d74-b98e7b92b510"
Sep 10 14:19:49.283568 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.283528    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 14:19:49.283618 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.283569    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 14:19:49.283654 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.283641    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m7jbv\""
Sep 10 14:19:49.283708 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.283691    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 14:19:49.284551 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.284533    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.285743 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.285725    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.286918 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.286903    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vz9vp"
Sep 10 14:19:49.288152 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288077    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 14:19:49.288152 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288102    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:19:49.288152 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288123    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 14:19:49.288152 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288131    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 14:19:49.288442 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:49.288160    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d66xs" podUID="661c9771-a727-49a4-8022-0a265a524e72"
Sep 10 14:19:49.288442 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288254    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-multus-socket-dir-parent\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.288442 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288268    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 14:19:49.288442 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288275    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-host-run-netns\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.288442 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288299    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-host-var-lib-cni-multus\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.288442 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288326    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-host-run-ovn-kubernetes\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.288442 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288354    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-run-openvswitch\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.288442 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288377    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fe41a3f3-0ada-4163-9f76-74d8d30809ed-serviceca\") pod \"node-ca-46jxd\" (UID: \"fe41a3f3-0ada-4163-9f76-74d8d30809ed\") " pod="openshift-image-registry/node-ca-46jxd"
Sep 10 14:19:49.288442 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288382    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 14:19:49.288442 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288332    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7wlvw\""
Sep 10 14:19:49.288442 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288400    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:19:49.288442 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288420    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49wx2\" (UniqueName: \"kubernetes.io/projected/3277b8b3-dc02-440f-95ca-3384cedb4fa6-kube-api-access-49wx2\") pod \"iptables-alerter-7xx9q\" (UID: \"3277b8b3-dc02-440f-95ca-3384cedb4fa6\") " pod="openshift-network-operator/iptables-alerter-7xx9q"
Sep 10 14:19:49.288442 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288446    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17ba33c7-43ee-458d-a33d-2aad75a7283e-cni-binary-copy\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.288442 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288449    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 14:19:49.289111 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288469    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-node-log\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.289111 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288523    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.289111 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288555    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe41a3f3-0ada-4163-9f76-74d8d30809ed-host\") pod \"node-ca-46jxd\" (UID: \"fe41a3f3-0ada-4163-9f76-74d8d30809ed\") " pod="openshift-image-registry/node-ca-46jxd"
Sep 10 14:19:49.289111 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288579    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6ec0ecd9-da59-44ba-8ed3-7e638c76358f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zksgj\" (UID: \"6ec0ecd9-da59-44ba-8ed3-7e638c76358f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj"
Sep 10 14:19:49.289111 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288587    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 14:19:49.289111 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288604    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6ec0ecd9-da59-44ba-8ed3-7e638c76358f-etc-selinux\") pod \"aws-ebs-csi-driver-node-zksgj\" (UID: \"6ec0ecd9-da59-44ba-8ed3-7e638c76358f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj"
Sep 10 14:19:49.289111 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288635    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jq5qb\""
Sep 10 14:19:49.289111 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288629    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-host-var-lib-cni-bin\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.289111 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288668    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl56g\" (UniqueName: \"kubernetes.io/projected/17ba33c7-43ee-458d-a33d-2aad75a7283e-kube-api-access-dl56g\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.289111 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288696    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-host-run-netns\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.289111 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288742    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-host-cni-netd\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.289111 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288765    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1fd944c4-2fe0-4066-bc76-5aaac1353833-env-overrides\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.289111 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288783    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50775e4e-6b60-4dec-91ec-b006bef3d00e-system-cni-dir\") pod \"multus-additional-cni-plugins-czbln\" (UID: \"50775e4e-6b60-4dec-91ec-b006bef3d00e\") " pod="openshift-multus/multus-additional-cni-plugins-czbln"
Sep 10 14:19:49.289111 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288818    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50775e4e-6b60-4dec-91ec-b006bef3d00e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-czbln\" (UID: \"50775e4e-6b60-4dec-91ec-b006bef3d00e\") " pod="openshift-multus/multus-additional-cni-plugins-czbln"
Sep 10 14:19:49.289111 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288842    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54qph\" (UniqueName: \"kubernetes.io/projected/50775e4e-6b60-4dec-91ec-b006bef3d00e-kube-api-access-54qph\") pod \"multus-additional-cni-plugins-czbln\" (UID: \"50775e4e-6b60-4dec-91ec-b006bef3d00e\") " pod="openshift-multus/multus-additional-cni-plugins-czbln"
Sep 10 14:19:49.289111 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288867    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3277b8b3-dc02-440f-95ca-3384cedb4fa6-host-slash\") pod \"iptables-alerter-7xx9q\" (UID: \"3277b8b3-dc02-440f-95ca-3384cedb4fa6\") " pod="openshift-network-operator/iptables-alerter-7xx9q"
Sep 10 14:19:49.289111 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288884    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-cnibin\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.289696 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288929    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-host-kubelet\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.289696 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288958    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvg2q\" (UniqueName: \"kubernetes.io/projected/6ec0ecd9-da59-44ba-8ed3-7e638c76358f-kube-api-access-xvg2q\") pod \"aws-ebs-csi-driver-node-zksgj\" (UID: \"6ec0ecd9-da59-44ba-8ed3-7e638c76358f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj"
Sep 10 14:19:49.289696 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.288975    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-host-cni-bin\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.289696 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289003    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4s2k\" (UniqueName: \"kubernetes.io/projected/1fd944c4-2fe0-4066-bc76-5aaac1353833-kube-api-access-c4s2k\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.289696 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289036    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3277b8b3-dc02-440f-95ca-3384cedb4fa6-iptables-alerter-script\") pod \"iptables-alerter-7xx9q\" (UID: \"3277b8b3-dc02-440f-95ca-3384cedb4fa6\") " pod="openshift-network-operator/iptables-alerter-7xx9q"
Sep 10 14:19:49.289696 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289060    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/17ba33c7-43ee-458d-a33d-2aad75a7283e-multus-daemon-config\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.289696 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289077    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-multus-cni-dir\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.289696 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289108    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4e33264c-464d-41c9-8d74-b98e7b92b510-dbus\") pod \"global-pull-secret-syncer-m2v46\" (UID: \"4e33264c-464d-41c9-8d74-b98e7b92b510\") " pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:19:49.289696 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289123    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-var-lib-openvswitch\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.289696 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289138    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6ec0ecd9-da59-44ba-8ed3-7e638c76358f-device-dir\") pod \"aws-ebs-csi-driver-node-zksgj\" (UID: \"6ec0ecd9-da59-44ba-8ed3-7e638c76358f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj"
Sep 10 14:19:49.289696 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289153    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-host-var-lib-kubelet\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.289696 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289176    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50775e4e-6b60-4dec-91ec-b006bef3d00e-cnibin\") pod \"multus-additional-cni-plugins-czbln\" (UID: \"50775e4e-6b60-4dec-91ec-b006bef3d00e\") " pod="openshift-multus/multus-additional-cni-plugins-czbln"
Sep 10 14:19:49.289696 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289197    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-host-run-k8s-cni-cncf-io\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.289696 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289210    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1fd944c4-2fe0-4066-bc76-5aaac1353833-ovnkube-script-lib\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.289696 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289225    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwhpb\" (UniqueName: \"kubernetes.io/projected/fe41a3f3-0ada-4163-9f76-74d8d30809ed-kube-api-access-qwhpb\") pod \"node-ca-46jxd\" (UID: \"fe41a3f3-0ada-4163-9f76-74d8d30809ed\") " pod="openshift-image-registry/node-ca-46jxd"
Sep 10 14:19:49.289696 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289246    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-multus-conf-dir\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.290288 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289274    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4e33264c-464d-41c9-8d74-b98e7b92b510-kubelet-config\") pod \"global-pull-secret-syncer-m2v46\" (UID: \"4e33264c-464d-41c9-8d74-b98e7b92b510\") " pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:19:49.290288 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289291    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-host-slash\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.290288 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289322    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-run-systemd\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.290288 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289338    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-etc-openvswitch\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.290288 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289352    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-system-cni-dir\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.290288 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289366    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/922dcf42-4b45-45fd-a011-f8a7eede8400-konnectivity-ca\") pod \"konnectivity-agent-vp8vn\" (UID: \"922dcf42-4b45-45fd-a011-f8a7eede8400\") " pod="kube-system/konnectivity-agent-vp8vn"
Sep 10 14:19:49.290288 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289389    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6ec0ecd9-da59-44ba-8ed3-7e638c76358f-registration-dir\") pod \"aws-ebs-csi-driver-node-zksgj\" (UID: \"6ec0ecd9-da59-44ba-8ed3-7e638c76358f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj"
Sep 10 14:19:49.290288 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289421    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6ec0ecd9-da59-44ba-8ed3-7e638c76358f-sys-fs\") pod \"aws-ebs-csi-driver-node-zksgj\" (UID: \"6ec0ecd9-da59-44ba-8ed3-7e638c76358f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj"
Sep 10 14:19:49.290288 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289440    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-hostroot\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.290288 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289460    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/50775e4e-6b60-4dec-91ec-b006bef3d00e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-czbln\" (UID: \"50775e4e-6b60-4dec-91ec-b006bef3d00e\") " pod="openshift-multus/multus-additional-cni-plugins-czbln"
Sep 10 14:19:49.290288 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289492    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1fd944c4-2fe0-4066-bc76-5aaac1353833-ovn-node-metrics-cert\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.290288 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289507    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs\") pod \"network-metrics-daemon-756dn\" (UID: \"c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0\") " pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:19:49.290288 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289522    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-host-run-multus-certs\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.290288 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289537    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-systemd-units\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.290288 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289559    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-run-ovn\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.290288 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289589    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg8pp\" (UniqueName: \"kubernetes.io/projected/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-kube-api-access-mg8pp\") pod \"network-metrics-daemon-756dn\" (UID: \"c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0\") " pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:19:49.290842 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289604    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/50775e4e-6b60-4dec-91ec-b006bef3d00e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-czbln\" (UID: \"50775e4e-6b60-4dec-91ec-b006bef3d00e\") " pod="openshift-multus/multus-additional-cni-plugins-czbln"
Sep 10 14:19:49.290842 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289619    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-etc-kubernetes\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.290842 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289656    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e33264c-464d-41c9-8d74-b98e7b92b510-original-pull-secret\") pod \"global-pull-secret-syncer-m2v46\" (UID: \"4e33264c-464d-41c9-8d74-b98e7b92b510\") " pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:19:49.290842 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289681    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-log-socket\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.290842 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289697    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1fd944c4-2fe0-4066-bc76-5aaac1353833-ovnkube-config\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.290842 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289769    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50775e4e-6b60-4dec-91ec-b006bef3d00e-cni-binary-copy\") pod \"multus-additional-cni-plugins-czbln\" (UID: \"50775e4e-6b60-4dec-91ec-b006bef3d00e\") " pod="openshift-multus/multus-additional-cni-plugins-czbln"
Sep 10 14:19:49.290842 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289786    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/922dcf42-4b45-45fd-a011-f8a7eede8400-agent-certs\") pod \"konnectivity-agent-vp8vn\" (UID: \"922dcf42-4b45-45fd-a011-f8a7eede8400\") " pod="kube-system/konnectivity-agent-vp8vn"
Sep 10 14:19:49.290842 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.289816    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6ec0ecd9-da59-44ba-8ed3-7e638c76358f-socket-dir\") pod \"aws-ebs-csi-driver-node-zksgj\" (UID: \"6ec0ecd9-da59-44ba-8ed3-7e638c76358f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj"
Sep 10 14:19:49.290842 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.290148    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6gnkr\""
Sep 10 14:19:49.290842 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.290223    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 14:19:49.292242 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.292219    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 14:19:49.305762 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.305715    2547 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 14:14:49 +0000 UTC" deadline="2028-05-28 16:20:32.971096568 +0000 UTC"
Sep 10 14:19:49.305762 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.305754    2547 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15026h0m43.665344666s"
Sep 10 14:19:49.306850 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.306833    2547 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 14:19:49.311045 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:49.311026    2547 helpers.go:245] readString: Failed to read "/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec0ecd9_da59_44ba_8ed3_7e638c76358f.slice/cpuset.cpus.effective": read /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec0ecd9_da59_44ba_8ed3_7e638c76358f.slice/cpuset.cpus.effective: no such device
Sep 10 14:19:49.329950 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.329902    2547 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pcxfz"
Sep 10 14:19:49.339238 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.339209    2547 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pcxfz"
Sep 10 14:19:49.383297 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.383275    2547 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 14:19:49.389978 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.389943    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50775e4e-6b60-4dec-91ec-b006bef3d00e-cnibin\") pod \"multus-additional-cni-plugins-czbln\" (UID: \"50775e4e-6b60-4dec-91ec-b006bef3d00e\") " pod="openshift-multus/multus-additional-cni-plugins-czbln"
Sep 10 14:19:49.389978 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.389976    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-etc-modprobe-d\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.390098 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.389998    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-run\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.390098 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390022    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-host\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.390098 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390048    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-host-run-k8s-cni-cncf-io\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.390098 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390078    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50775e4e-6b60-4dec-91ec-b006bef3d00e-cnibin\") pod \"multus-additional-cni-plugins-czbln\" (UID: \"50775e4e-6b60-4dec-91ec-b006bef3d00e\") " pod="openshift-multus/multus-additional-cni-plugins-czbln"
Sep 10 14:19:49.390350 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390105    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-host-run-k8s-cni-cncf-io\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.390350 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390129    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1fd944c4-2fe0-4066-bc76-5aaac1353833-ovnkube-script-lib\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.390350 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390163    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/017bc08f-f2d2-4797-809a-939231ec10a1-tmp\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.390350 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390188    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c952s\" (UniqueName: \"kubernetes.io/projected/017bc08f-f2d2-4797-809a-939231ec10a1-kube-api-access-c952s\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.390350 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390217    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwhpb\" (UniqueName: \"kubernetes.io/projected/fe41a3f3-0ada-4163-9f76-74d8d30809ed-kube-api-access-qwhpb\") pod \"node-ca-46jxd\" (UID: \"fe41a3f3-0ada-4163-9f76-74d8d30809ed\") " pod="openshift-image-registry/node-ca-46jxd"
Sep 10 14:19:49.390350 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390243    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-multus-conf-dir\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.390350 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390268    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4e33264c-464d-41c9-8d74-b98e7b92b510-kubelet-config\") pod \"global-pull-secret-syncer-m2v46\" (UID: \"4e33264c-464d-41c9-8d74-b98e7b92b510\") " pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:19:49.390350 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390292    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-host-slash\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.390350 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390315    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-run-systemd\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.390350 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390323    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-multus-conf-dir\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.390350 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390340    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-etc-openvswitch\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.390932 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390367    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-etc-sysctl-d\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.390932 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390402    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4e33264c-464d-41c9-8d74-b98e7b92b510-kubelet-config\") pod \"global-pull-secret-syncer-m2v46\" (UID: \"4e33264c-464d-41c9-8d74-b98e7b92b510\") " pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:19:49.390932 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390444    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-system-cni-dir\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.390932 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390466    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-run-systemd\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.390932 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390485    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/922dcf42-4b45-45fd-a011-f8a7eede8400-konnectivity-ca\") pod \"konnectivity-agent-vp8vn\" (UID: \"922dcf42-4b45-45fd-a011-f8a7eede8400\") " pod="kube-system/konnectivity-agent-vp8vn"
Sep 10 14:19:49.390932 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390505    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-host-slash\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.390932 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390446    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-etc-openvswitch\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.390932 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390518    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6ec0ecd9-da59-44ba-8ed3-7e638c76358f-registration-dir\") pod \"aws-ebs-csi-driver-node-zksgj\" (UID: \"6ec0ecd9-da59-44ba-8ed3-7e638c76358f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj"
Sep 10 14:19:49.390932 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390547    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6ec0ecd9-da59-44ba-8ed3-7e638c76358f-sys-fs\") pod \"aws-ebs-csi-driver-node-zksgj\" (UID: \"6ec0ecd9-da59-44ba-8ed3-7e638c76358f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj"
Sep 10 14:19:49.390932 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390570    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-hostroot\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.390932 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390595    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-system-cni-dir\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.390932 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390597    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/50775e4e-6b60-4dec-91ec-b006bef3d00e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-czbln\" (UID: \"50775e4e-6b60-4dec-91ec-b006bef3d00e\") " pod="openshift-multus/multus-additional-cni-plugins-czbln"
Sep 10 14:19:49.390932 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390645    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6ec0ecd9-da59-44ba-8ed3-7e638c76358f-registration-dir\") pod \"aws-ebs-csi-driver-node-zksgj\" (UID: \"6ec0ecd9-da59-44ba-8ed3-7e638c76358f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj"
Sep 10 14:19:49.390932 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390662    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1fd944c4-2fe0-4066-bc76-5aaac1353833-ovn-node-metrics-cert\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.390932 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390652    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6ec0ecd9-da59-44ba-8ed3-7e638c76358f-sys-fs\") pod \"aws-ebs-csi-driver-node-zksgj\" (UID: \"6ec0ecd9-da59-44ba-8ed3-7e638c76358f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj"
Sep 10 14:19:49.390932 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390684    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-hostroot\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.390932 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390693    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9x5x\" (UniqueName: \"kubernetes.io/projected/661c9771-a727-49a4-8022-0a265a524e72-kube-api-access-q9x5x\") pod \"network-check-target-d66xs\" (UID: \"661c9771-a727-49a4-8022-0a265a524e72\") " pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:19:49.391809 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390722    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs\") pod \"network-metrics-daemon-756dn\" (UID: \"c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0\") " pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:19:49.391809 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390748    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-host-run-multus-certs\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.391809 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390793    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-systemd-units\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.391809 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390819    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-run-ovn\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.391809 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390846    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-var-lib-kubelet\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.391809 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390871    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5a663e3c-6895-4085-aa41-1efd5d6ab208-hosts-file\") pod \"node-resolver-vz9vp\" (UID: \"5a663e3c-6895-4085-aa41-1efd5d6ab208\") " pod="openshift-dns/node-resolver-vz9vp"
Sep 10 14:19:49.391809 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390899    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg8pp\" (UniqueName: \"kubernetes.io/projected/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-kube-api-access-mg8pp\") pod \"network-metrics-daemon-756dn\" (UID: \"c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0\") " pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:19:49.391809 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390926    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/50775e4e-6b60-4dec-91ec-b006bef3d00e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-czbln\" (UID: \"50775e4e-6b60-4dec-91ec-b006bef3d00e\") " pod="openshift-multus/multus-additional-cni-plugins-czbln"
Sep 10 14:19:49.391809 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390914    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-systemd-units\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.391809 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390965    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-etc-kubernetes\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.391809 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.390992    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e33264c-464d-41c9-8d74-b98e7b92b510-original-pull-secret\") pod \"global-pull-secret-syncer-m2v46\" (UID: \"4e33264c-464d-41c9-8d74-b98e7b92b510\") " pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:19:49.391809 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.391017    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-log-socket\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.391809 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:49.391030    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:19:49.391809 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.391042    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1fd944c4-2fe0-4066-bc76-5aaac1353833-ovnkube-config\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.391809 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.391068    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a663e3c-6895-4085-aa41-1efd5d6ab208-tmp-dir\") pod \"node-resolver-vz9vp\" (UID: \"5a663e3c-6895-4085-aa41-1efd5d6ab208\") " pod="openshift-dns/node-resolver-vz9vp"
Sep 10 14:19:49.391809 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:49.391107    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs podName:c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:49.891083137 +0000 UTC m=+1.932957603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs") pod "network-metrics-daemon-756dn" (UID: "c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:19:49.391809 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.391132    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50775e4e-6b60-4dec-91ec-b006bef3d00e-cni-binary-copy\") pod \"multus-additional-cni-plugins-czbln\" (UID: \"50775e4e-6b60-4dec-91ec-b006bef3d00e\") " pod="openshift-multus/multus-additional-cni-plugins-czbln"
Sep 10 14:19:49.392667 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.391146    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-run-ovn\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.392667 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.391162    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/922dcf42-4b45-45fd-a011-f8a7eede8400-agent-certs\") pod \"konnectivity-agent-vp8vn\" (UID: \"922dcf42-4b45-45fd-a011-f8a7eede8400\") " pod="kube-system/konnectivity-agent-vp8vn"
Sep 10 14:19:49.392667 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.391186    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/922dcf42-4b45-45fd-a011-f8a7eede8400-konnectivity-ca\") pod \"konnectivity-agent-vp8vn\" (UID: \"922dcf42-4b45-45fd-a011-f8a7eede8400\") " pod="kube-system/konnectivity-agent-vp8vn"
Sep 10 14:19:49.392667 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.391191    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6ec0ecd9-da59-44ba-8ed3-7e638c76358f-socket-dir\") pod \"aws-ebs-csi-driver-node-zksgj\" (UID: \"6ec0ecd9-da59-44ba-8ed3-7e638c76358f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj"
Sep 10 14:19:49.392667 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.391195    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/50775e4e-6b60-4dec-91ec-b006bef3d00e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-czbln\" (UID: \"50775e4e-6b60-4dec-91ec-b006bef3d00e\") " pod="openshift-multus/multus-additional-cni-plugins-czbln"
Sep 10 14:19:49.392667 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.391216    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-multus-socket-dir-parent\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.392667 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.391284    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-multus-socket-dir-parent\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.392667 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:49.391321    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:19:49.392667 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.391330    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-host-run-multus-certs\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.392667 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.391359    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-log-socket\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.392667 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:49.391404    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e33264c-464d-41c9-8d74-b98e7b92b510-original-pull-secret podName:4e33264c-464d-41c9-8d74-b98e7b92b510 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:49.891378386 +0000 UTC m=+1.933252868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4e33264c-464d-41c9-8d74-b98e7b92b510-original-pull-secret") pod "global-pull-secret-syncer-m2v46" (UID: "4e33264c-464d-41c9-8d74-b98e7b92b510") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:19:49.392667 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.391406    2547 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 14:19:49.392667 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.391552    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1fd944c4-2fe0-4066-bc76-5aaac1353833-ovnkube-script-lib\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.392667 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.391811    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-host-run-netns\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.392667 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.391846    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-host-var-lib-cni-multus\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.392667 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.391873    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-host-run-ovn-kubernetes\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.392667 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.391900    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-run-openvswitch\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.393431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.391926    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-etc-systemd\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.393431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.391930    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/50775e4e-6b60-4dec-91ec-b006bef3d00e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-czbln\" (UID: \"50775e4e-6b60-4dec-91ec-b006bef3d00e\") " pod="openshift-multus/multus-additional-cni-plugins-czbln"
Sep 10 14:19:49.393431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.391936    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1fd944c4-2fe0-4066-bc76-5aaac1353833-ovnkube-config\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.393431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.391950    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fe41a3f3-0ada-4163-9f76-74d8d30809ed-serviceca\") pod \"node-ca-46jxd\" (UID: \"fe41a3f3-0ada-4163-9f76-74d8d30809ed\") " pod="openshift-image-registry/node-ca-46jxd"
Sep 10 14:19:49.393431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.391979    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49wx2\" (UniqueName: \"kubernetes.io/projected/3277b8b3-dc02-440f-95ca-3384cedb4fa6-kube-api-access-49wx2\") pod \"iptables-alerter-7xx9q\" (UID: \"3277b8b3-dc02-440f-95ca-3384cedb4fa6\") " pod="openshift-network-operator/iptables-alerter-7xx9q"
Sep 10 14:19:49.393431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392024    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-host-run-ovn-kubernetes\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.393431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392027    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-run-openvswitch\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.393431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392052    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6ec0ecd9-da59-44ba-8ed3-7e638c76358f-socket-dir\") pod \"aws-ebs-csi-driver-node-zksgj\" (UID: \"6ec0ecd9-da59-44ba-8ed3-7e638c76358f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj"
Sep 10 14:19:49.393431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392090    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-host-var-lib-cni-multus\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.393431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392098    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-host-run-netns\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.393431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392117    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17ba33c7-43ee-458d-a33d-2aad75a7283e-cni-binary-copy\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.393431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392142    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-node-log\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.393431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392169    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.393431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392192    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe41a3f3-0ada-4163-9f76-74d8d30809ed-host\") pod \"node-ca-46jxd\" (UID: \"fe41a3f3-0ada-4163-9f76-74d8d30809ed\") " pod="openshift-image-registry/node-ca-46jxd"
Sep 10 14:19:49.393431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392213    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6ec0ecd9-da59-44ba-8ed3-7e638c76358f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zksgj\" (UID: \"6ec0ecd9-da59-44ba-8ed3-7e638c76358f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj"
Sep 10 14:19:49.393431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392233    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6ec0ecd9-da59-44ba-8ed3-7e638c76358f-etc-selinux\") pod \"aws-ebs-csi-driver-node-zksgj\" (UID: \"6ec0ecd9-da59-44ba-8ed3-7e638c76358f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj"
Sep 10 14:19:49.393431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392254    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-host-var-lib-cni-bin\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.394072 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392292    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl56g\" (UniqueName: \"kubernetes.io/projected/17ba33c7-43ee-458d-a33d-2aad75a7283e-kube-api-access-dl56g\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.394072 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392313    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-host-run-netns\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.394072 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392332    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-host-cni-netd\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.394072 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392374    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-host-cni-netd\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.394072 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392392    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50775e4e-6b60-4dec-91ec-b006bef3d00e-cni-binary-copy\") pod \"multus-additional-cni-plugins-czbln\" (UID: \"50775e4e-6b60-4dec-91ec-b006bef3d00e\") " pod="openshift-multus/multus-additional-cni-plugins-czbln"
Sep 10 14:19:49.394072 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392398    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6ec0ecd9-da59-44ba-8ed3-7e638c76358f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zksgj\" (UID: \"6ec0ecd9-da59-44ba-8ed3-7e638c76358f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj"
Sep 10 14:19:49.394072 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392405    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fe41a3f3-0ada-4163-9f76-74d8d30809ed-serviceca\") pod \"node-ca-46jxd\" (UID: \"fe41a3f3-0ada-4163-9f76-74d8d30809ed\") " pod="openshift-image-registry/node-ca-46jxd"
Sep 10 14:19:49.394072 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392438    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6ec0ecd9-da59-44ba-8ed3-7e638c76358f-etc-selinux\") pod \"aws-ebs-csi-driver-node-zksgj\" (UID: \"6ec0ecd9-da59-44ba-8ed3-7e638c76358f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj"
Sep 10 14:19:49.394072 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392455    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe41a3f3-0ada-4163-9f76-74d8d30809ed-host\") pod \"node-ca-46jxd\" (UID: \"fe41a3f3-0ada-4163-9f76-74d8d30809ed\") " pod="openshift-image-registry/node-ca-46jxd"
Sep 10 14:19:49.394072 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392459    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-host-var-lib-cni-bin\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.394072 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392455    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-node-log\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.394072 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392473    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-host-run-netns\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.394072 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392487    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1fd944c4-2fe0-4066-bc76-5aaac1353833-env-overrides\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.394072 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392506    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-etc-kubernetes\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.394072 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392512    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.394072 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392516    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50775e4e-6b60-4dec-91ec-b006bef3d00e-system-cni-dir\") pod \"multus-additional-cni-plugins-czbln\" (UID: \"50775e4e-6b60-4dec-91ec-b006bef3d00e\") " pod="openshift-multus/multus-additional-cni-plugins-czbln"
Sep 10 14:19:49.394072 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392550    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50775e4e-6b60-4dec-91ec-b006bef3d00e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-czbln\" (UID: \"50775e4e-6b60-4dec-91ec-b006bef3d00e\") " pod="openshift-multus/multus-additional-cni-plugins-czbln"
Sep 10 14:19:49.394594 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392651    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50775e4e-6b60-4dec-91ec-b006bef3d00e-system-cni-dir\") pod \"multus-additional-cni-plugins-czbln\" (UID: \"50775e4e-6b60-4dec-91ec-b006bef3d00e\") " pod="openshift-multus/multus-additional-cni-plugins-czbln"
Sep 10 14:19:49.394594 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392662    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54qph\" (UniqueName: \"kubernetes.io/projected/50775e4e-6b60-4dec-91ec-b006bef3d00e-kube-api-access-54qph\") pod \"multus-additional-cni-plugins-czbln\" (UID: \"50775e4e-6b60-4dec-91ec-b006bef3d00e\") " pod="openshift-multus/multus-additional-cni-plugins-czbln"
Sep 10 14:19:49.394594 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392736    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3277b8b3-dc02-440f-95ca-3384cedb4fa6-host-slash\") pod \"iptables-alerter-7xx9q\" (UID: \"3277b8b3-dc02-440f-95ca-3384cedb4fa6\") " pod="openshift-network-operator/iptables-alerter-7xx9q"
Sep 10 14:19:49.394594 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392766    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-cnibin\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.394594 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392821    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3277b8b3-dc02-440f-95ca-3384cedb4fa6-host-slash\") pod \"iptables-alerter-7xx9q\" (UID: \"3277b8b3-dc02-440f-95ca-3384cedb4fa6\") " pod="openshift-network-operator/iptables-alerter-7xx9q"
Sep 10 14:19:49.394594 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392834    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-host-kubelet\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.394594 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392820    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50775e4e-6b60-4dec-91ec-b006bef3d00e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-czbln\" (UID: \"50775e4e-6b60-4dec-91ec-b006bef3d00e\") " pod="openshift-multus/multus-additional-cni-plugins-czbln"
Sep 10 14:19:49.394594 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392877    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-host-kubelet\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.394594 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392889    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-cnibin\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.394594 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392917    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17ba33c7-43ee-458d-a33d-2aad75a7283e-cni-binary-copy\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.394594 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.392953    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1fd944c4-2fe0-4066-bc76-5aaac1353833-env-overrides\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.394594 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393004    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-etc-sysconfig\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.394594 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393039    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-etc-kubernetes\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.394594 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393071    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvg2q\" (UniqueName: \"kubernetes.io/projected/6ec0ecd9-da59-44ba-8ed3-7e638c76358f-kube-api-access-xvg2q\") pod \"aws-ebs-csi-driver-node-zksgj\" (UID: \"6ec0ecd9-da59-44ba-8ed3-7e638c76358f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj"
Sep 10 14:19:49.394594 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393097    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-host-cni-bin\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.394594 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393122    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4s2k\" (UniqueName: \"kubernetes.io/projected/1fd944c4-2fe0-4066-bc76-5aaac1353833-kube-api-access-c4s2k\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.394594 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393172    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-etc-sysctl-conf\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.395254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393203    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-host-cni-bin\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.395254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393213    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-sys\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.395254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393241    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3277b8b3-dc02-440f-95ca-3384cedb4fa6-iptables-alerter-script\") pod \"iptables-alerter-7xx9q\" (UID: \"3277b8b3-dc02-440f-95ca-3384cedb4fa6\") " pod="openshift-network-operator/iptables-alerter-7xx9q"
Sep 10 14:19:49.395254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393263    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/17ba33c7-43ee-458d-a33d-2aad75a7283e-multus-daemon-config\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.395254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393285    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-lib-modules\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.395254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393320    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd9lz\" (UniqueName: \"kubernetes.io/projected/5a663e3c-6895-4085-aa41-1efd5d6ab208-kube-api-access-qd9lz\") pod \"node-resolver-vz9vp\" (UID: \"5a663e3c-6895-4085-aa41-1efd5d6ab208\") " pod="openshift-dns/node-resolver-vz9vp"
Sep 10 14:19:49.395254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393357    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-multus-cni-dir\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.395254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393421    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4e33264c-464d-41c9-8d74-b98e7b92b510-dbus\") pod \"global-pull-secret-syncer-m2v46\" (UID: \"4e33264c-464d-41c9-8d74-b98e7b92b510\") " pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:19:49.395254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393433    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-multus-cni-dir\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.395254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393450    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-var-lib-openvswitch\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.395254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393490    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/017bc08f-f2d2-4797-809a-939231ec10a1-etc-tuned\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.395254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393519    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6ec0ecd9-da59-44ba-8ed3-7e638c76358f-device-dir\") pod \"aws-ebs-csi-driver-node-zksgj\" (UID: \"6ec0ecd9-da59-44ba-8ed3-7e638c76358f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj"
Sep 10 14:19:49.395254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393525    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fd944c4-2fe0-4066-bc76-5aaac1353833-var-lib-openvswitch\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.395254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393491    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4e33264c-464d-41c9-8d74-b98e7b92b510-dbus\") pod \"global-pull-secret-syncer-m2v46\" (UID: \"4e33264c-464d-41c9-8d74-b98e7b92b510\") " pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:19:49.395254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393544    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-host-var-lib-kubelet\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.395254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393632    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6ec0ecd9-da59-44ba-8ed3-7e638c76358f-device-dir\") pod \"aws-ebs-csi-driver-node-zksgj\" (UID: \"6ec0ecd9-da59-44ba-8ed3-7e638c76358f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj"
Sep 10 14:19:49.395254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393679    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17ba33c7-43ee-458d-a33d-2aad75a7283e-host-var-lib-kubelet\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.395764 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393724    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/17ba33c7-43ee-458d-a33d-2aad75a7283e-multus-daemon-config\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.395764 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.393832    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3277b8b3-dc02-440f-95ca-3384cedb4fa6-iptables-alerter-script\") pod \"iptables-alerter-7xx9q\" (UID: \"3277b8b3-dc02-440f-95ca-3384cedb4fa6\") " pod="openshift-network-operator/iptables-alerter-7xx9q"
Sep 10 14:19:49.395764 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.395236    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1fd944c4-2fe0-4066-bc76-5aaac1353833-ovn-node-metrics-cert\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.395940 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.395835    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/922dcf42-4b45-45fd-a011-f8a7eede8400-agent-certs\") pod \"konnectivity-agent-vp8vn\" (UID: \"922dcf42-4b45-45fd-a011-f8a7eede8400\") " pod="kube-system/konnectivity-agent-vp8vn"
Sep 10 14:19:49.398757 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.398741    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwhpb\" (UniqueName: \"kubernetes.io/projected/fe41a3f3-0ada-4163-9f76-74d8d30809ed-kube-api-access-qwhpb\") pod \"node-ca-46jxd\" (UID: \"fe41a3f3-0ada-4163-9f76-74d8d30809ed\") " pod="openshift-image-registry/node-ca-46jxd"
Sep 10 14:19:49.400798 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.400749    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg8pp\" (UniqueName: \"kubernetes.io/projected/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-kube-api-access-mg8pp\") pod \"network-metrics-daemon-756dn\" (UID: \"c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0\") " pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:19:49.402105 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.402077    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49wx2\" (UniqueName: \"kubernetes.io/projected/3277b8b3-dc02-440f-95ca-3384cedb4fa6-kube-api-access-49wx2\") pod \"iptables-alerter-7xx9q\" (UID: \"3277b8b3-dc02-440f-95ca-3384cedb4fa6\") " pod="openshift-network-operator/iptables-alerter-7xx9q"
Sep 10 14:19:49.402868 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.402846    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54qph\" (UniqueName: \"kubernetes.io/projected/50775e4e-6b60-4dec-91ec-b006bef3d00e-kube-api-access-54qph\") pod \"multus-additional-cni-plugins-czbln\" (UID: \"50775e4e-6b60-4dec-91ec-b006bef3d00e\") " pod="openshift-multus/multus-additional-cni-plugins-czbln"
Sep 10 14:19:49.403113 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.403095    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl56g\" (UniqueName: \"kubernetes.io/projected/17ba33c7-43ee-458d-a33d-2aad75a7283e-kube-api-access-dl56g\") pod \"multus-gfz6l\" (UID: \"17ba33c7-43ee-458d-a33d-2aad75a7283e\") " pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.403350 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.403334    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4s2k\" (UniqueName: \"kubernetes.io/projected/1fd944c4-2fe0-4066-bc76-5aaac1353833-kube-api-access-c4s2k\") pod \"ovnkube-node-4ww4p\" (UID: \"1fd944c4-2fe0-4066-bc76-5aaac1353833\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.403722 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.403702    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvg2q\" (UniqueName: \"kubernetes.io/projected/6ec0ecd9-da59-44ba-8ed3-7e638c76358f-kube-api-access-xvg2q\") pod \"aws-ebs-csi-driver-node-zksgj\" (UID: \"6ec0ecd9-da59-44ba-8ed3-7e638c76358f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj"
Sep 10 14:19:49.411392 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.411371    2547 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:19:49.414443 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.414427    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:19:49.494297 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494268    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-etc-sysconfig\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.494297 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494298    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-etc-kubernetes\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.494575 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494328    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-etc-sysctl-conf\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.494575 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494398    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-etc-sysconfig\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.494575 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494413    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-etc-kubernetes\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.494575 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494396    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-sys\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.494575 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494444    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-sys\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.494575 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494459    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-lib-modules\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.494575 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494479    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd9lz\" (UniqueName: \"kubernetes.io/projected/5a663e3c-6895-4085-aa41-1efd5d6ab208-kube-api-access-qd9lz\") pod \"node-resolver-vz9vp\" (UID: \"5a663e3c-6895-4085-aa41-1efd5d6ab208\") " pod="openshift-dns/node-resolver-vz9vp"
Sep 10 14:19:49.494575 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494497    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/017bc08f-f2d2-4797-809a-939231ec10a1-etc-tuned\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.494575 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494506    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-etc-sysctl-conf\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.494575 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494542    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-lib-modules\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.495191 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494584    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-etc-modprobe-d\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.495191 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494617    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-run\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.495191 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494642    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-host\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.495191 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494682    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/017bc08f-f2d2-4797-809a-939231ec10a1-tmp\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.495191 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494721    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c952s\" (UniqueName: \"kubernetes.io/projected/017bc08f-f2d2-4797-809a-939231ec10a1-kube-api-access-c952s\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.495191 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494725    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-run\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.495191 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494735    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-etc-modprobe-d\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.495191 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494749    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-etc-sysctl-d\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.495191 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494761    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-host\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.495191 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494803    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9x5x\" (UniqueName: \"kubernetes.io/projected/661c9771-a727-49a4-8022-0a265a524e72-kube-api-access-q9x5x\") pod \"network-check-target-d66xs\" (UID: \"661c9771-a727-49a4-8022-0a265a524e72\") " pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:19:49.495191 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494861    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-var-lib-kubelet\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.495191 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494867    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-etc-sysctl-d\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.495191 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494904    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5a663e3c-6895-4085-aa41-1efd5d6ab208-hosts-file\") pod \"node-resolver-vz9vp\" (UID: \"5a663e3c-6895-4085-aa41-1efd5d6ab208\") " pod="openshift-dns/node-resolver-vz9vp"
Sep 10 14:19:49.495191 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.494996    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5a663e3c-6895-4085-aa41-1efd5d6ab208-hosts-file\") pod \"node-resolver-vz9vp\" (UID: \"5a663e3c-6895-4085-aa41-1efd5d6ab208\") " pod="openshift-dns/node-resolver-vz9vp"
Sep 10 14:19:49.495858 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.495286    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a663e3c-6895-4085-aa41-1efd5d6ab208-tmp-dir\") pod \"node-resolver-vz9vp\" (UID: \"5a663e3c-6895-4085-aa41-1efd5d6ab208\") " pod="openshift-dns/node-resolver-vz9vp"
Sep 10 14:19:49.495858 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.495295    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-var-lib-kubelet\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.495858 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.495356    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-etc-systemd\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.495858 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.495523    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/017bc08f-f2d2-4797-809a-939231ec10a1-etc-systemd\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.495858 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.495572    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a663e3c-6895-4085-aa41-1efd5d6ab208-tmp-dir\") pod \"node-resolver-vz9vp\" (UID: \"5a663e3c-6895-4085-aa41-1efd5d6ab208\") " pod="openshift-dns/node-resolver-vz9vp"
Sep 10 14:19:49.497917 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.497896    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/017bc08f-f2d2-4797-809a-939231ec10a1-etc-tuned\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.498020 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.497938    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/017bc08f-f2d2-4797-809a-939231ec10a1-tmp\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.500946 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:49.500929    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:19:49.501027 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:49.500955    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:19:49.501027 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:49.500968    2547 projected.go:196] Error preparing data for projected volume kube-api-access-q9x5x for pod openshift-network-diagnostics/network-check-target-d66xs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:19:49.501132 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:49.501041    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/661c9771-a727-49a4-8022-0a265a524e72-kube-api-access-q9x5x podName:661c9771-a727-49a4-8022-0a265a524e72 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:50.001021206 +0000 UTC m=+2.042895672 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-q9x5x" (UniqueName: "kubernetes.io/projected/661c9771-a727-49a4-8022-0a265a524e72-kube-api-access-q9x5x") pod "network-check-target-d66xs" (UID: "661c9771-a727-49a4-8022-0a265a524e72") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:19:49.503117 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.503088    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c952s\" (UniqueName: \"kubernetes.io/projected/017bc08f-f2d2-4797-809a-939231ec10a1-kube-api-access-c952s\") pod \"tuned-lqlzs\" (UID: \"017bc08f-f2d2-4797-809a-939231ec10a1\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.503399 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.503383    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd9lz\" (UniqueName: \"kubernetes.io/projected/5a663e3c-6895-4085-aa41-1efd5d6ab208-kube-api-access-qd9lz\") pod \"node-resolver-vz9vp\" (UID: \"5a663e3c-6895-4085-aa41-1efd5d6ab208\") " pod="openshift-dns/node-resolver-vz9vp"
Sep 10 14:19:49.593143 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:49.592992    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f59cdebad304ce873337176c339318.slice/crio-b6d908b3ace90b18163b02ee6c284a8406f43bdfb52851f6f446666047146f3f WatchSource:0}: Error finding container b6d908b3ace90b18163b02ee6c284a8406f43bdfb52851f6f446666047146f3f: Status 404 returned error can't find the container with id b6d908b3ace90b18163b02ee6c284a8406f43bdfb52851f6f446666047146f3f
Sep 10 14:19:49.593570 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:49.593547    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f4b9a9d29461bb847438bd99b7b31d4.slice/crio-ece72f3fccecfe25d3a0d77f051b7aad44ac356735a3f7ed1b796161beb5b5a9 WatchSource:0}: Error finding container ece72f3fccecfe25d3a0d77f051b7aad44ac356735a3f7ed1b796161beb5b5a9: Status 404 returned error can't find the container with id ece72f3fccecfe25d3a0d77f051b7aad44ac356735a3f7ed1b796161beb5b5a9
Sep 10 14:19:49.599545 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.599524    2547 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:19:49.607918 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.607901    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7xx9q"
Sep 10 14:19:49.616359 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:49.616336    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3277b8b3_dc02_440f_95ca_3384cedb4fa6.slice/crio-14461ba8374e26c28714179680a2ac79dbfa40a37b714df71808c2d761a17525 WatchSource:0}: Error finding container 14461ba8374e26c28714179680a2ac79dbfa40a37b714df71808c2d761a17525: Status 404 returned error can't find the container with id 14461ba8374e26c28714179680a2ac79dbfa40a37b714df71808c2d761a17525
Sep 10 14:19:49.623103 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.623079    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj"
Sep 10 14:19:49.630532 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:49.630511    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec0ecd9_da59_44ba_8ed3_7e638c76358f.slice/crio-6fdb51669ac03ed3ca1d7670e705c5a53a4aec153eed7697ea4f3339103a0c84 WatchSource:0}: Error finding container 6fdb51669ac03ed3ca1d7670e705c5a53a4aec153eed7697ea4f3339103a0c84: Status 404 returned error can't find the container with id 6fdb51669ac03ed3ca1d7670e705c5a53a4aec153eed7697ea4f3339103a0c84
Sep 10 14:19:49.648853 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.648825    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-46jxd"
Sep 10 14:19:49.656249 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:49.656225    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe41a3f3_0ada_4163_9f76_74d8d30809ed.slice/crio-f51880ec59f635dd76a4fbde893fac3d8bb21396ae198a7b91e7dcb71f340678 WatchSource:0}: Error finding container f51880ec59f635dd76a4fbde893fac3d8bb21396ae198a7b91e7dcb71f340678: Status 404 returned error can't find the container with id f51880ec59f635dd76a4fbde893fac3d8bb21396ae198a7b91e7dcb71f340678
Sep 10 14:19:49.661644 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.661626    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-czbln"
Sep 10 14:19:49.668968 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:49.668946    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50775e4e_6b60_4dec_91ec_b006bef3d00e.slice/crio-e0f03b862a5388163395475c89f02436308bb5288292f8cf0c585240af964c18 WatchSource:0}: Error finding container e0f03b862a5388163395475c89f02436308bb5288292f8cf0c585240af964c18: Status 404 returned error can't find the container with id e0f03b862a5388163395475c89f02436308bb5288292f8cf0c585240af964c18
Sep 10 14:19:49.678342 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.678324    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gfz6l"
Sep 10 14:19:49.686094 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:49.686066    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ba33c7_43ee_458d_a33d_2aad75a7283e.slice/crio-7e21f334a02815cbb6cb154d092fc084a0f751e39540939cb82e9092ff29a990 WatchSource:0}: Error finding container 7e21f334a02815cbb6cb154d092fc084a0f751e39540939cb82e9092ff29a990: Status 404 returned error can't find the container with id 7e21f334a02815cbb6cb154d092fc084a0f751e39540939cb82e9092ff29a990
Sep 10 14:19:49.687703 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.687679    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vp8vn"
Sep 10 14:19:49.695579 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:49.695559    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922dcf42_4b45_45fd_a011_f8a7eede8400.slice/crio-616d016b9db08a5c2dccbbd854d10e29a57e2bcdcfe7d34469909b7dfae976ab WatchSource:0}: Error finding container 616d016b9db08a5c2dccbbd854d10e29a57e2bcdcfe7d34469909b7dfae976ab: Status 404 returned error can't find the container with id 616d016b9db08a5c2dccbbd854d10e29a57e2bcdcfe7d34469909b7dfae976ab
Sep 10 14:19:49.733397 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.733371    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lqlzs"
Sep 10 14:19:49.738586 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.738561    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vz9vp"
Sep 10 14:19:49.744219 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:49.744194    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017bc08f_f2d2_4797_809a_939231ec10a1.slice/crio-e8c645d7eddc53f841053fe0798b0682c830eefaa4d047e1cd68dc2f2178ca84 WatchSource:0}: Error finding container e8c645d7eddc53f841053fe0798b0682c830eefaa4d047e1cd68dc2f2178ca84: Status 404 returned error can't find the container with id e8c645d7eddc53f841053fe0798b0682c830eefaa4d047e1cd68dc2f2178ca84
Sep 10 14:19:49.747235 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:49.747215    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a663e3c_6895_4085_aa41_1efd5d6ab208.slice/crio-c3bf5dccd13dcb72066daeaa830831958585849976f07f7874438b92daf8ea44 WatchSource:0}: Error finding container c3bf5dccd13dcb72066daeaa830831958585849976f07f7874438b92daf8ea44: Status 404 returned error can't find the container with id c3bf5dccd13dcb72066daeaa830831958585849976f07f7874438b92daf8ea44
Sep 10 14:19:49.800012 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:19:49.799985    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd944c4_2fe0_4066_bc76_5aaac1353833.slice/crio-41a7652ea519c07a79f267f3e743d8f44bce351636b3074662c70bd0a2b4a9c5 WatchSource:0}: Error finding container 41a7652ea519c07a79f267f3e743d8f44bce351636b3074662c70bd0a2b4a9c5: Status 404 returned error can't find the container with id 41a7652ea519c07a79f267f3e743d8f44bce351636b3074662c70bd0a2b4a9c5
Sep 10 14:19:49.897622 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.897529    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs\") pod \"network-metrics-daemon-756dn\" (UID: \"c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0\") " pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:19:49.897622 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:49.897564    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e33264c-464d-41c9-8d74-b98e7b92b510-original-pull-secret\") pod \"global-pull-secret-syncer-m2v46\" (UID: \"4e33264c-464d-41c9-8d74-b98e7b92b510\") " pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:19:49.897855 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:49.897658    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:19:49.897855 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:49.897709    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e33264c-464d-41c9-8d74-b98e7b92b510-original-pull-secret podName:4e33264c-464d-41c9-8d74-b98e7b92b510 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:50.8976938 +0000 UTC m=+2.939568275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4e33264c-464d-41c9-8d74-b98e7b92b510-original-pull-secret") pod "global-pull-secret-syncer-m2v46" (UID: "4e33264c-464d-41c9-8d74-b98e7b92b510") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:19:49.897855 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:49.897697    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:19:49.897855 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:49.897771    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs podName:c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:50.897753642 +0000 UTC m=+2.939628117 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs") pod "network-metrics-daemon-756dn" (UID: "c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:19:50.100840 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:50.100690    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9x5x\" (UniqueName: \"kubernetes.io/projected/661c9771-a727-49a4-8022-0a265a524e72-kube-api-access-q9x5x\") pod \"network-check-target-d66xs\" (UID: \"661c9771-a727-49a4-8022-0a265a524e72\") " pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:19:50.101026 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:50.100913    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:19:50.101026 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:50.100940    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:19:50.101026 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:50.100957    2547 projected.go:196] Error preparing data for projected volume kube-api-access-q9x5x for pod openshift-network-diagnostics/network-check-target-d66xs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:19:50.101026 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:50.101031    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/661c9771-a727-49a4-8022-0a265a524e72-kube-api-access-q9x5x podName:661c9771-a727-49a4-8022-0a265a524e72 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:51.101009179 +0000 UTC m=+3.142884141 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-q9x5x" (UniqueName: "kubernetes.io/projected/661c9771-a727-49a4-8022-0a265a524e72-kube-api-access-q9x5x") pod "network-check-target-d66xs" (UID: "661c9771-a727-49a4-8022-0a265a524e72") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:19:50.347285 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:50.347092    2547 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 14:14:50 +0000 UTC" deadline="2028-04-26 02:33:53.590600597 +0000 UTC"
Sep 10 14:19:50.347285 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:50.347168    2547 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14244h14m3.243436847s"
Sep 10 14:19:50.446281 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:50.446210    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-46jxd" event={"ID":"fe41a3f3-0ada-4163-9f76-74d8d30809ed","Type":"ContainerStarted","Data":"f51880ec59f635dd76a4fbde893fac3d8bb21396ae198a7b91e7dcb71f340678"}
Sep 10 14:19:50.474053 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:50.474015    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-249.ec2.internal" event={"ID":"7f4b9a9d29461bb847438bd99b7b31d4","Type":"ContainerStarted","Data":"ece72f3fccecfe25d3a0d77f051b7aad44ac356735a3f7ed1b796161beb5b5a9"}
Sep 10 14:19:50.490579 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:50.490551    2547 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:19:50.499908 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:50.499754    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p" event={"ID":"1fd944c4-2fe0-4066-bc76-5aaac1353833","Type":"ContainerStarted","Data":"41a7652ea519c07a79f267f3e743d8f44bce351636b3074662c70bd0a2b4a9c5"}
Sep 10 14:19:50.519999 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:50.519967    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vz9vp" event={"ID":"5a663e3c-6895-4085-aa41-1efd5d6ab208","Type":"ContainerStarted","Data":"c3bf5dccd13dcb72066daeaa830831958585849976f07f7874438b92daf8ea44"}
Sep 10 14:19:50.545176 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:50.545135    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lqlzs" event={"ID":"017bc08f-f2d2-4797-809a-939231ec10a1","Type":"ContainerStarted","Data":"e8c645d7eddc53f841053fe0798b0682c830eefaa4d047e1cd68dc2f2178ca84"}
Sep 10 14:19:50.575808 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:50.574810    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vp8vn" event={"ID":"922dcf42-4b45-45fd-a011-f8a7eede8400","Type":"ContainerStarted","Data":"616d016b9db08a5c2dccbbd854d10e29a57e2bcdcfe7d34469909b7dfae976ab"}
Sep 10 14:19:50.592922 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:50.592887    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj" event={"ID":"6ec0ecd9-da59-44ba-8ed3-7e638c76358f","Type":"ContainerStarted","Data":"6fdb51669ac03ed3ca1d7670e705c5a53a4aec153eed7697ea4f3339103a0c84"}
Sep 10 14:19:50.623041 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:50.622923    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7xx9q" event={"ID":"3277b8b3-dc02-440f-95ca-3384cedb4fa6","Type":"ContainerStarted","Data":"14461ba8374e26c28714179680a2ac79dbfa40a37b714df71808c2d761a17525"}
Sep 10 14:19:50.652962 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:50.652872    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-249.ec2.internal" event={"ID":"39f59cdebad304ce873337176c339318","Type":"ContainerStarted","Data":"b6d908b3ace90b18163b02ee6c284a8406f43bdfb52851f6f446666047146f3f"}
Sep 10 14:19:50.660482 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:50.660447    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gfz6l" event={"ID":"17ba33c7-43ee-458d-a33d-2aad75a7283e","Type":"ContainerStarted","Data":"7e21f334a02815cbb6cb154d092fc084a0f751e39540939cb82e9092ff29a990"}
Sep 10 14:19:50.681434 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:50.681403    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-czbln" event={"ID":"50775e4e-6b60-4dec-91ec-b006bef3d00e","Type":"ContainerStarted","Data":"e0f03b862a5388163395475c89f02436308bb5288292f8cf0c585240af964c18"}
Sep 10 14:19:50.908225 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:50.908137    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs\") pod \"network-metrics-daemon-756dn\" (UID: \"c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0\") " pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:19:50.908225 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:50.908196    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e33264c-464d-41c9-8d74-b98e7b92b510-original-pull-secret\") pod \"global-pull-secret-syncer-m2v46\" (UID: \"4e33264c-464d-41c9-8d74-b98e7b92b510\") " pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:19:50.908461 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:50.908348    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:19:50.908461 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:50.908415    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e33264c-464d-41c9-8d74-b98e7b92b510-original-pull-secret podName:4e33264c-464d-41c9-8d74-b98e7b92b510 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:52.908396658 +0000 UTC m=+4.950271131 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4e33264c-464d-41c9-8d74-b98e7b92b510-original-pull-secret") pod "global-pull-secret-syncer-m2v46" (UID: "4e33264c-464d-41c9-8d74-b98e7b92b510") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:19:50.908854 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:50.908832    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:19:50.908953 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:50.908893    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs podName:c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:52.90887526 +0000 UTC m=+4.950749736 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs") pod "network-metrics-daemon-756dn" (UID: "c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:19:51.111095 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:51.110747    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9x5x\" (UniqueName: \"kubernetes.io/projected/661c9771-a727-49a4-8022-0a265a524e72-kube-api-access-q9x5x\") pod \"network-check-target-d66xs\" (UID: \"661c9771-a727-49a4-8022-0a265a524e72\") " pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:19:51.111095 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:51.110963    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:19:51.111095 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:51.110981    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:19:51.111095 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:51.110994    2547 projected.go:196] Error preparing data for projected volume kube-api-access-q9x5x for pod openshift-network-diagnostics/network-check-target-d66xs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:19:51.111095 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:51.111055    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/661c9771-a727-49a4-8022-0a265a524e72-kube-api-access-q9x5x podName:661c9771-a727-49a4-8022-0a265a524e72 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:53.111036867 +0000 UTC m=+5.152911352 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-q9x5x" (UniqueName: "kubernetes.io/projected/661c9771-a727-49a4-8022-0a265a524e72-kube-api-access-q9x5x") pod "network-check-target-d66xs" (UID: "661c9771-a727-49a4-8022-0a265a524e72") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:19:51.347851 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:51.347754    2547 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 14:14:50 +0000 UTC" deadline="2028-04-20 07:23:09.990054268 +0000 UTC"
Sep 10 14:19:51.347851 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:51.347800    2547 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14105h3m18.64225923s"
Sep 10 14:19:51.415591 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:51.415561    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:19:51.415888 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:51.415701    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m2v46" podUID="4e33264c-464d-41c9-8d74-b98e7b92b510"
Sep 10 14:19:51.415888 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:51.415866    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:19:51.416046 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:51.415973    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-756dn" podUID="c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0"
Sep 10 14:19:51.416046 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:51.416037    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:19:51.416158 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:51.416110    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d66xs" podUID="661c9771-a727-49a4-8022-0a265a524e72"
Sep 10 14:19:51.667406 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:51.667096    2547 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:19:52.928016 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:52.927264    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs\") pod \"network-metrics-daemon-756dn\" (UID: \"c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0\") " pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:19:52.928016 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:52.927320    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e33264c-464d-41c9-8d74-b98e7b92b510-original-pull-secret\") pod \"global-pull-secret-syncer-m2v46\" (UID: \"4e33264c-464d-41c9-8d74-b98e7b92b510\") " pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:19:52.928016 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:52.927474    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:19:52.928016 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:52.927475    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:19:52.928016 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:52.927548    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e33264c-464d-41c9-8d74-b98e7b92b510-original-pull-secret podName:4e33264c-464d-41c9-8d74-b98e7b92b510 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:56.927525728 +0000 UTC m=+8.969400205 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4e33264c-464d-41c9-8d74-b98e7b92b510-original-pull-secret") pod "global-pull-secret-syncer-m2v46" (UID: "4e33264c-464d-41c9-8d74-b98e7b92b510") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:19:52.928016 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:52.927568    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs podName:c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:56.927557862 +0000 UTC m=+8.969432328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs") pod "network-metrics-daemon-756dn" (UID: "c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:19:53.129395 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:53.128675    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9x5x\" (UniqueName: \"kubernetes.io/projected/661c9771-a727-49a4-8022-0a265a524e72-kube-api-access-q9x5x\") pod \"network-check-target-d66xs\" (UID: \"661c9771-a727-49a4-8022-0a265a524e72\") " pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:19:53.129395 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:53.128932    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:19:53.129395 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:53.128953    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:19:53.129395 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:53.128967    2547 projected.go:196] Error preparing data for projected volume kube-api-access-q9x5x for pod openshift-network-diagnostics/network-check-target-d66xs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:19:53.129395 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:53.129026    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/661c9771-a727-49a4-8022-0a265a524e72-kube-api-access-q9x5x podName:661c9771-a727-49a4-8022-0a265a524e72 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:57.129008463 +0000 UTC m=+9.170882941 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-q9x5x" (UniqueName: "kubernetes.io/projected/661c9771-a727-49a4-8022-0a265a524e72-kube-api-access-q9x5x") pod "network-check-target-d66xs" (UID: "661c9771-a727-49a4-8022-0a265a524e72") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:19:53.415104 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:53.415023    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:19:53.415268 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:53.415170    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d66xs" podUID="661c9771-a727-49a4-8022-0a265a524e72"
Sep 10 14:19:53.415611 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:53.415583    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:19:53.415740 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:53.415691    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m2v46" podUID="4e33264c-464d-41c9-8d74-b98e7b92b510"
Sep 10 14:19:53.415819 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:53.415743    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:19:53.415877 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:53.415852    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-756dn" podUID="c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0"
Sep 10 14:19:55.416525 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:55.415978    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:19:55.416525 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:55.416118    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m2v46" podUID="4e33264c-464d-41c9-8d74-b98e7b92b510"
Sep 10 14:19:55.416525 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:55.416317    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:19:55.416525 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:55.416410    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d66xs" podUID="661c9771-a727-49a4-8022-0a265a524e72"
Sep 10 14:19:55.416525 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:55.416476    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:19:55.417328 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:55.416568    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-756dn" podUID="c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0"
Sep 10 14:19:56.962380 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:56.962338    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs\") pod \"network-metrics-daemon-756dn\" (UID: \"c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0\") " pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:19:56.962973 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:56.962395    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e33264c-464d-41c9-8d74-b98e7b92b510-original-pull-secret\") pod \"global-pull-secret-syncer-m2v46\" (UID: \"4e33264c-464d-41c9-8d74-b98e7b92b510\") " pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:19:56.962973 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:56.962546    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:19:56.962973 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:56.962612    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e33264c-464d-41c9-8d74-b98e7b92b510-original-pull-secret podName:4e33264c-464d-41c9-8d74-b98e7b92b510 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:04.962592055 +0000 UTC m=+17.004466539 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4e33264c-464d-41c9-8d74-b98e7b92b510-original-pull-secret") pod "global-pull-secret-syncer-m2v46" (UID: "4e33264c-464d-41c9-8d74-b98e7b92b510") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:19:56.962973 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:56.962686    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:19:56.962973 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:56.962726    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs podName:c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:04.962712613 +0000 UTC m=+17.004587095 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs") pod "network-metrics-daemon-756dn" (UID: "c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:19:57.165340 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:57.165251    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9x5x\" (UniqueName: \"kubernetes.io/projected/661c9771-a727-49a4-8022-0a265a524e72-kube-api-access-q9x5x\") pod \"network-check-target-d66xs\" (UID: \"661c9771-a727-49a4-8022-0a265a524e72\") " pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:19:57.165534 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:57.165457    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:19:57.165534 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:57.165486    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:19:57.165534 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:57.165500    2547 projected.go:196] Error preparing data for projected volume kube-api-access-q9x5x for pod openshift-network-diagnostics/network-check-target-d66xs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:19:57.165704 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:57.165562    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/661c9771-a727-49a4-8022-0a265a524e72-kube-api-access-q9x5x podName:661c9771-a727-49a4-8022-0a265a524e72 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:05.165541685 +0000 UTC m=+17.207416163 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-q9x5x" (UniqueName: "kubernetes.io/projected/661c9771-a727-49a4-8022-0a265a524e72-kube-api-access-q9x5x") pod "network-check-target-d66xs" (UID: "661c9771-a727-49a4-8022-0a265a524e72") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:19:57.416099 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:57.416017    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:19:57.416099 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:57.416047    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:19:57.416339 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:57.416163    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d66xs" podUID="661c9771-a727-49a4-8022-0a265a524e72"
Sep 10 14:19:57.416339 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:57.416212    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m2v46" podUID="4e33264c-464d-41c9-8d74-b98e7b92b510"
Sep 10 14:19:57.416339 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:57.416271    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:19:57.416499 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:57.416361    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-756dn" podUID="c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0"
Sep 10 14:19:59.415326 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:59.415284    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:19:59.415894 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:59.415439    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:19:59.415894 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:59.415451    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d66xs" podUID="661c9771-a727-49a4-8022-0a265a524e72"
Sep 10 14:19:59.415894 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:59.415548    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-756dn" podUID="c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0"
Sep 10 14:19:59.415894 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:19:59.415613    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:19:59.415894 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:19:59.415674    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m2v46" podUID="4e33264c-464d-41c9-8d74-b98e7b92b510"
Sep 10 14:20:01.414944 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:01.414908    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:20:01.414944 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:01.414943    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:20:01.415437 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:01.414947    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:20:01.415437 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:01.415034    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d66xs" podUID="661c9771-a727-49a4-8022-0a265a524e72"
Sep 10 14:20:01.415437 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:01.415116    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m2v46" podUID="4e33264c-464d-41c9-8d74-b98e7b92b510"
Sep 10 14:20:01.415437 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:01.415185    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-756dn" podUID="c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0"
Sep 10 14:20:03.415945 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:03.415909    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:20:03.416373 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:03.415909    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:20:03.416373 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:03.416047    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-756dn" podUID="c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0"
Sep 10 14:20:03.416373 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:03.416058    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:20:03.416373 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:03.416155    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m2v46" podUID="4e33264c-464d-41c9-8d74-b98e7b92b510"
Sep 10 14:20:03.416373 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:03.416248    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d66xs" podUID="661c9771-a727-49a4-8022-0a265a524e72"
Sep 10 14:20:05.024709 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:05.024648    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs\") pod \"network-metrics-daemon-756dn\" (UID: \"c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0\") " pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:20:05.025096 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:05.024729    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e33264c-464d-41c9-8d74-b98e7b92b510-original-pull-secret\") pod \"global-pull-secret-syncer-m2v46\" (UID: \"4e33264c-464d-41c9-8d74-b98e7b92b510\") " pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:20:05.025096 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:05.024806    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:20:05.025096 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:05.024867    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:20:05.025096 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:05.024880    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs podName:c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:21.024857167 +0000 UTC m=+33.066731642 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs") pod "network-metrics-daemon-756dn" (UID: "c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:20:05.025096 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:05.024921    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e33264c-464d-41c9-8d74-b98e7b92b510-original-pull-secret podName:4e33264c-464d-41c9-8d74-b98e7b92b510 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:21.024904039 +0000 UTC m=+33.066778512 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4e33264c-464d-41c9-8d74-b98e7b92b510-original-pull-secret") pod "global-pull-secret-syncer-m2v46" (UID: "4e33264c-464d-41c9-8d74-b98e7b92b510") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:20:05.225768 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:05.225732    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9x5x\" (UniqueName: \"kubernetes.io/projected/661c9771-a727-49a4-8022-0a265a524e72-kube-api-access-q9x5x\") pod \"network-check-target-d66xs\" (UID: \"661c9771-a727-49a4-8022-0a265a524e72\") " pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:20:05.225951 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:05.225929    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:20:05.226010 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:05.225957    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:20:05.226010 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:05.225968    2547 projected.go:196] Error preparing data for projected volume kube-api-access-q9x5x for pod openshift-network-diagnostics/network-check-target-d66xs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:20:05.226090 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:05.226023    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/661c9771-a727-49a4-8022-0a265a524e72-kube-api-access-q9x5x podName:661c9771-a727-49a4-8022-0a265a524e72 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:21.226008578 +0000 UTC m=+33.267883062 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-q9x5x" (UniqueName: "kubernetes.io/projected/661c9771-a727-49a4-8022-0a265a524e72-kube-api-access-q9x5x") pod "network-check-target-d66xs" (UID: "661c9771-a727-49a4-8022-0a265a524e72") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:20:05.415369 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:05.415332    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:20:05.415534 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:05.415377    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:20:05.415534 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:05.415341    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:20:05.415534 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:05.415471    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-756dn" podUID="c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0"
Sep 10 14:20:05.415657 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:05.415551    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m2v46" podUID="4e33264c-464d-41c9-8d74-b98e7b92b510"
Sep 10 14:20:05.415711 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:05.415648    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d66xs" podUID="661c9771-a727-49a4-8022-0a265a524e72"
Sep 10 14:20:07.415238 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:07.415198    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:20:07.415238 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:07.415224    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:20:07.415769 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:07.415224    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:20:07.415769 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:07.415325    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d66xs" podUID="661c9771-a727-49a4-8022-0a265a524e72"
Sep 10 14:20:07.415769 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:07.415423    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m2v46" podUID="4e33264c-464d-41c9-8d74-b98e7b92b510"
Sep 10 14:20:07.415769 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:07.415501    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-756dn" podUID="c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0"
Sep 10 14:20:09.415923 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:09.415884    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:20:09.416289 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:09.415955    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:20:09.416289 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:09.415965    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:20:09.416289 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:09.416036    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d66xs" podUID="661c9771-a727-49a4-8022-0a265a524e72"
Sep 10 14:20:09.416289 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:09.416151    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-756dn" podUID="c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0"
Sep 10 14:20:09.416289 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:09.416228    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m2v46" podUID="4e33264c-464d-41c9-8d74-b98e7b92b510"
Sep 10 14:20:10.738527 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:10.738134    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p" event={"ID":"1fd944c4-2fe0-4066-bc76-5aaac1353833","Type":"ContainerStarted","Data":"92aeaa390731357b13b3a771f8c8d4918f553ab14e1054e302949fe4250d5514"}
Sep 10 14:20:10.739317 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:10.738577    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p" event={"ID":"1fd944c4-2fe0-4066-bc76-5aaac1353833","Type":"ContainerStarted","Data":"63a0d0430b62375a2d44f27c86a64a1a09578d789c959ca3cf4b681b82a5d348"}
Sep 10 14:20:10.739317 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:10.738627    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p" event={"ID":"1fd944c4-2fe0-4066-bc76-5aaac1353833","Type":"ContainerStarted","Data":"f84fdefc71902998d809df7636f19d634078e84bbf53d73a53c2f433fe59c98c"}
Sep 10 14:20:10.739317 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:10.738673    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p" event={"ID":"1fd944c4-2fe0-4066-bc76-5aaac1353833","Type":"ContainerStarted","Data":"5969cbe25d200efb12730d151bd3948961c87655f72e4d061ec425aebe06d5c6"}
Sep 10 14:20:10.739317 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:10.738739    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p" event={"ID":"1fd944c4-2fe0-4066-bc76-5aaac1353833","Type":"ContainerStarted","Data":"641428b84c9edd03ccc1ea48fc774201944020df18a01da5729142cb40db65f9"}
Sep 10 14:20:10.739317 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:10.738787    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p" event={"ID":"1fd944c4-2fe0-4066-bc76-5aaac1353833","Type":"ContainerStarted","Data":"069468da4ebf18f19506339e8d7b3bec0251aefc891432f2d679680f5076b0cc"}
Sep 10 14:20:10.740086 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:10.740054    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vz9vp" event={"ID":"5a663e3c-6895-4085-aa41-1efd5d6ab208","Type":"ContainerStarted","Data":"3dd40b3894434d502d02a05c7f6863e573de5c375263966322ed93379b74b57e"}
Sep 10 14:20:10.741747 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:10.741714    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lqlzs" event={"ID":"017bc08f-f2d2-4797-809a-939231ec10a1","Type":"ContainerStarted","Data":"8c92ae9d869bd0ca3f478aebea79a388eefeb1dd78924b01abc248fe24d8b135"}
Sep 10 14:20:10.743440 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:10.743417    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vp8vn" event={"ID":"922dcf42-4b45-45fd-a011-f8a7eede8400","Type":"ContainerStarted","Data":"af26d6677e00d4767c7335567ff8700d98cead561ca35b514ceddef06f01de77"}
Sep 10 14:20:10.745153 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:10.745128    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj" event={"ID":"6ec0ecd9-da59-44ba-8ed3-7e638c76358f","Type":"ContainerStarted","Data":"05c879325456e5ccd0d81338358c510d286befbd698ef876ad3d4217afc044f3"}
Sep 10 14:20:10.746864 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:10.746839    2547 generic.go:356] "Generic (PLEG): container finished" podID="39f59cdebad304ce873337176c339318" containerID="b10ceb0d605f3acec7550afd0d712c224995b85df0099a736e50a4778b9272fa" exitCode=0
Sep 10 14:20:10.746970 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:10.746909    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-249.ec2.internal" event={"ID":"39f59cdebad304ce873337176c339318","Type":"ContainerDied","Data":"b10ceb0d605f3acec7550afd0d712c224995b85df0099a736e50a4778b9272fa"}
Sep 10 14:20:10.748982 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:10.748951    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gfz6l" event={"ID":"17ba33c7-43ee-458d-a33d-2aad75a7283e","Type":"ContainerStarted","Data":"4d362dfd51fab4dca85a1f28b2861690dfd54020649bf363a054f3fb3c763e74"}
Sep 10 14:20:10.750890 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:10.750867    2547 generic.go:356] "Generic (PLEG): container finished" podID="50775e4e-6b60-4dec-91ec-b006bef3d00e" containerID="c2e6718ad3ab865d8c63fb641e46d663ba30cc36bd20c595521b951286670bb7" exitCode=0
Sep 10 14:20:10.750976 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:10.750934    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-czbln" event={"ID":"50775e4e-6b60-4dec-91ec-b006bef3d00e","Type":"ContainerDied","Data":"c2e6718ad3ab865d8c63fb641e46d663ba30cc36bd20c595521b951286670bb7"}
Sep 10 14:20:10.752978 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:10.752938    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-46jxd" event={"ID":"fe41a3f3-0ada-4163-9f76-74d8d30809ed","Type":"ContainerStarted","Data":"d91eff3b2a5a12df49cd87ca988b50b53cbc954151c87c1ab657cdde453fd713"}
Sep 10 14:20:10.754689 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:10.754666    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-249.ec2.internal" event={"ID":"7f4b9a9d29461bb847438bd99b7b31d4","Type":"ContainerStarted","Data":"0c3cfbc74e6fcce552df8ebfdf090ae592b79c274a1945435157101f059b1779"}
Sep 10 14:20:10.761788 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:10.761748    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vz9vp" podStartSLOduration=1.830151633 podStartE2EDuration="21.76173651s" totalImagesPullingTime="19.931584877s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:10.761474553 +0000 UTC m=+22.803349036" watchObservedRunningTime="2026-09-10 14:20:10.76173651 +0000 UTC m=+22.803610989"
Sep 10 14:20:10.800316 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:10.800258    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gfz6l" podStartSLOduration=1.757297383 podStartE2EDuration="21.800238144s" totalImagesPullingTime="20.042940761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:10.799784103 +0000 UTC m=+22.841658585" watchObservedRunningTime="2026-09-10 14:20:10.800238144 +0000 UTC m=+22.842112629"
Sep 10 14:20:10.818906 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:10.818624    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vp8vn" podStartSLOduration=1.835216545 podStartE2EDuration="21.818605483s" totalImagesPullingTime="19.983388938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:10.817548814 +0000 UTC m=+22.859423297" watchObservedRunningTime="2026-09-10 14:20:10.818605483 +0000 UTC m=+22.860479965"
Sep 10 14:20:10.837656 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:10.837613    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-249.ec2.internal" podStartSLOduration=20.837598117 podStartE2EDuration="20.837598117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:10.837395135 +0000 UTC m=+22.879269617" watchObservedRunningTime="2026-09-10 14:20:10.837598117 +0000 UTC m=+22.879472597"
Sep 10 14:20:10.853614 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:10.853557    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-46jxd" podStartSLOduration=1.847770946 podStartE2EDuration="21.85353805s" totalImagesPullingTime="20.005767104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:10.853274997 +0000 UTC m=+22.895149475" watchObservedRunningTime="2026-09-10 14:20:10.85353805 +0000 UTC m=+22.895412533"
Sep 10 14:20:10.958959 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:10.958909    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lqlzs" podStartSLOduration=2.007449469 podStartE2EDuration="21.958891094s" totalImagesPullingTime="19.951441625s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:10.924123251 +0000 UTC m=+22.965997731" watchObservedRunningTime="2026-09-10 14:20:10.958891094 +0000 UTC m=+23.000765583"
Sep 10 14:20:11.415322 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:11.415289    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:20:11.415462 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:11.415304    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:20:11.415462 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:11.415408    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:20:11.415462 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:11.415405    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d66xs" podUID="661c9771-a727-49a4-8022-0a265a524e72"
Sep 10 14:20:11.415581 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:11.415477    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m2v46" podUID="4e33264c-464d-41c9-8d74-b98e7b92b510"
Sep 10 14:20:11.415581 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:11.415541    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-756dn" podUID="c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0"
Sep 10 14:20:11.698282 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:11.698056    2547 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 14:20:11.759360 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:11.759326    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj" event={"ID":"6ec0ecd9-da59-44ba-8ed3-7e638c76358f","Type":"ContainerStarted","Data":"b57c5e5a709740e7756f707ad6eec85b12fc862f4850346f5265b5aa32895fa5"}
Sep 10 14:20:11.761009 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:11.760981    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7xx9q" event={"ID":"3277b8b3-dc02-440f-95ca-3384cedb4fa6","Type":"ContainerStarted","Data":"6f679bec09cc4dd4eb4831698f73c31fff2d059d975f06d42eac81d3b6776287"}
Sep 10 14:20:11.763465 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:11.763417    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-249.ec2.internal" event={"ID":"39f59cdebad304ce873337176c339318","Type":"ContainerStarted","Data":"f619ddc7be81fecd766714454c4e3e7d389ff19b75652cf9f8c50dc00552a409"}
Sep 10 14:20:11.789898 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:11.789844    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-249.ec2.internal" podStartSLOduration=21.789824592 podStartE2EDuration="21.789824592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:11.78946522 +0000 UTC m=+23.831339728" watchObservedRunningTime="2026-09-10 14:20:11.789824592 +0000 UTC m=+23.831699074"
Sep 10 14:20:11.790071 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:11.789944    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7xx9q" podStartSLOduration=2.7119029709999998 podStartE2EDuration="22.789936012s" totalImagesPullingTime="20.078033041s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:11.775073373 +0000 UTC m=+23.816947853" watchObservedRunningTime="2026-09-10 14:20:11.789936012 +0000 UTC m=+23.831810493"
Sep 10 14:20:12.338904 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:12.338785    2547 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T14:20:11.698258764Z","UUID":"cefa01f5-4f5a-44c4-9064-d43b2401bfc7","Handler":null,"Name":"","Endpoint":""}
Sep 10 14:20:12.340662 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:12.340403    2547 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 14:20:12.340662 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:12.340437    2547 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 14:20:12.770387 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:12.770283    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p" event={"ID":"1fd944c4-2fe0-4066-bc76-5aaac1353833","Type":"ContainerStarted","Data":"9adefb073e0d29000e0b7d06e555138c7565d47f0d6df9c5178c324782fdc60e"}
Sep 10 14:20:13.415175 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:13.415142    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:20:13.415356 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:13.415187    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:20:13.415356 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:13.415261    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:20:13.415356 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:13.415272    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m2v46" podUID="4e33264c-464d-41c9-8d74-b98e7b92b510"
Sep 10 14:20:13.415356 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:13.415329    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d66xs" podUID="661c9771-a727-49a4-8022-0a265a524e72"
Sep 10 14:20:13.415531 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:13.415427    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-756dn" podUID="c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0"
Sep 10 14:20:13.774725 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:13.774626    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj" event={"ID":"6ec0ecd9-da59-44ba-8ed3-7e638c76358f","Type":"ContainerStarted","Data":"61b805233ace7696909b8b39798f7c18e2feb93d942efe7225685e096306788d"}
Sep 10 14:20:13.794126 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:13.794064    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zksgj" podStartSLOduration=2.138074776 podStartE2EDuration="24.794043178s" totalImagesPullingTime="22.655968402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:13.792614093 +0000 UTC m=+25.834488573" watchObservedRunningTime="2026-09-10 14:20:13.794043178 +0000 UTC m=+25.835917660"
Sep 10 14:20:15.416052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:15.415875    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:20:15.416474 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:15.415884    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:20:15.416474 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:15.416130    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m2v46" podUID="4e33264c-464d-41c9-8d74-b98e7b92b510"
Sep 10 14:20:15.416474 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:15.415909    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:20:15.416474 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:15.416199    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d66xs" podUID="661c9771-a727-49a4-8022-0a265a524e72"
Sep 10 14:20:15.416474 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:15.416317    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-756dn" podUID="c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0"
Sep 10 14:20:15.733321 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:15.733226    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vp8vn"
Sep 10 14:20:15.733895 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:15.733879    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vp8vn"
Sep 10 14:20:15.782281 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:15.782248    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p" event={"ID":"1fd944c4-2fe0-4066-bc76-5aaac1353833","Type":"ContainerStarted","Data":"0a7dabf930e28eba9528c957be8276e5bff1da84ee696b2a88c8dbb8524cc5c8"}
Sep 10 14:20:15.782510 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:15.782493    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:20:15.782581 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:15.782573    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:20:15.782632 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:15.782623    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:20:15.784360 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:15.784331    2547 generic.go:356] "Generic (PLEG): container finished" podID="50775e4e-6b60-4dec-91ec-b006bef3d00e" containerID="86fbd17f5bebe0cd635e69907ba7e21aa7d1ef52b624bcd59f5b637246092a5f" exitCode=0
Sep 10 14:20:15.784455 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:15.784430    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-czbln" event={"ID":"50775e4e-6b60-4dec-91ec-b006bef3d00e","Type":"ContainerDied","Data":"86fbd17f5bebe0cd635e69907ba7e21aa7d1ef52b624bcd59f5b637246092a5f"}
Sep 10 14:20:15.784623 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:15.784588    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vp8vn"
Sep 10 14:20:15.785276 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:15.785142    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vp8vn"
Sep 10 14:20:15.798602 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:15.798570    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:20:15.804140 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:15.804122    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:20:15.805866 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:15.805830    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p" podStartSLOduration=6.769494996 podStartE2EDuration="26.805817853s" totalImagesPullingTime="20.036322857s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:15.805469105 +0000 UTC m=+27.847343587" watchObservedRunningTime="2026-09-10 14:20:15.805817853 +0000 UTC m=+27.847692334"
Sep 10 14:20:16.788283 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:16.788249    2547 generic.go:356] "Generic (PLEG): container finished" podID="50775e4e-6b60-4dec-91ec-b006bef3d00e" containerID="23bfdb61f0c37a931777d1a6ab5b7dc7bed282d7789618c0712a395d13e67c65" exitCode=0
Sep 10 14:20:16.789133 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:16.788341    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-czbln" event={"ID":"50775e4e-6b60-4dec-91ec-b006bef3d00e","Type":"ContainerDied","Data":"23bfdb61f0c37a931777d1a6ab5b7dc7bed282d7789618c0712a395d13e67c65"}
Sep 10 14:20:17.068744 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:17.068715    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d66xs"]
Sep 10 14:20:17.068904 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:17.068832    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:20:17.068993 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:17.068968    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d66xs" podUID="661c9771-a727-49a4-8022-0a265a524e72"
Sep 10 14:20:17.069673 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:17.069648    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m2v46"]
Sep 10 14:20:17.069780 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:17.069770    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:20:17.069904 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:17.069880    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m2v46" podUID="4e33264c-464d-41c9-8d74-b98e7b92b510"
Sep 10 14:20:17.071577 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:17.071555    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-756dn"]
Sep 10 14:20:17.071678 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:17.071657    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:20:17.071802 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:17.071781    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-756dn" podUID="c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0"
Sep 10 14:20:18.416738 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:18.416538    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:20:18.417156 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:18.416607    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:20:18.417156 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:18.416817    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m2v46" podUID="4e33264c-464d-41c9-8d74-b98e7b92b510"
Sep 10 14:20:18.417156 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:18.416634    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:20:18.417156 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:18.416881    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d66xs" podUID="661c9771-a727-49a4-8022-0a265a524e72"
Sep 10 14:20:18.417156 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:18.416968    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-756dn" podUID="c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0"
Sep 10 14:20:18.794357 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:18.794274    2547 generic.go:356] "Generic (PLEG): container finished" podID="50775e4e-6b60-4dec-91ec-b006bef3d00e" containerID="4eb430643ab7ba94f5b05aa17382cd4321dd176caaf91a893ad43a728290fd58" exitCode=0
Sep 10 14:20:18.794357 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:18.794335    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-czbln" event={"ID":"50775e4e-6b60-4dec-91ec-b006bef3d00e","Type":"ContainerDied","Data":"4eb430643ab7ba94f5b05aa17382cd4321dd176caaf91a893ad43a728290fd58"}
Sep 10 14:20:20.415207 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:20.415175    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:20:20.415207 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:20.415200    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:20:20.415820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:20.415175    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:20:20.415820 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:20.415307    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m2v46" podUID="4e33264c-464d-41c9-8d74-b98e7b92b510"
Sep 10 14:20:20.415820 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:20.415393    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-756dn" podUID="c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0"
Sep 10 14:20:20.415820 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:20.415483    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d66xs" podUID="661c9771-a727-49a4-8022-0a265a524e72"
Sep 10 14:20:21.040854 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:21.040825    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs\") pod \"network-metrics-daemon-756dn\" (UID: \"c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0\") " pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:20:21.040854 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:21.040860    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e33264c-464d-41c9-8d74-b98e7b92b510-original-pull-secret\") pod \"global-pull-secret-syncer-m2v46\" (UID: \"4e33264c-464d-41c9-8d74-b98e7b92b510\") " pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:20:21.041062 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:21.040996    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:20:21.041062 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:21.041028    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:20:21.041147 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:21.041072    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs podName:c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:53.041051148 +0000 UTC m=+65.082925622 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs") pod "network-metrics-daemon-756dn" (UID: "c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:20:21.041147 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:21.041092    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e33264c-464d-41c9-8d74-b98e7b92b510-original-pull-secret podName:4e33264c-464d-41c9-8d74-b98e7b92b510 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:53.041081453 +0000 UTC m=+65.082955918 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4e33264c-464d-41c9-8d74-b98e7b92b510-original-pull-secret") pod "global-pull-secret-syncer-m2v46" (UID: "4e33264c-464d-41c9-8d74-b98e7b92b510") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:20:21.242771 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:21.242739    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9x5x\" (UniqueName: \"kubernetes.io/projected/661c9771-a727-49a4-8022-0a265a524e72-kube-api-access-q9x5x\") pod \"network-check-target-d66xs\" (UID: \"661c9771-a727-49a4-8022-0a265a524e72\") " pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:20:21.242927 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:21.242907    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:20:21.242927 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:21.242926    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:20:21.243023 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:21.242938    2547 projected.go:196] Error preparing data for projected volume kube-api-access-q9x5x for pod openshift-network-diagnostics/network-check-target-d66xs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:20:21.243023 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:21.242990    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/661c9771-a727-49a4-8022-0a265a524e72-kube-api-access-q9x5x podName:661c9771-a727-49a4-8022-0a265a524e72 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:53.242976024 +0000 UTC m=+65.284850496 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-q9x5x" (UniqueName: "kubernetes.io/projected/661c9771-a727-49a4-8022-0a265a524e72-kube-api-access-q9x5x") pod "network-check-target-d66xs" (UID: "661c9771-a727-49a4-8022-0a265a524e72") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:20:22.415743 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:22.415704    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:20:22.416314 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:22.415705    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:20:22.416314 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:22.415852    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-756dn" podUID="c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0"
Sep 10 14:20:22.416314 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:22.415704    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:20:22.416314 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:22.415930    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m2v46" podUID="4e33264c-464d-41c9-8d74-b98e7b92b510"
Sep 10 14:20:22.416314 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:22.415994    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d66xs" podUID="661c9771-a727-49a4-8022-0a265a524e72"
Sep 10 14:20:23.772214 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:23.772173    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-249.ec2.internal" event="NodeReady"
Sep 10 14:20:23.772591 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:23.772302    2547 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 14:20:23.809775 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:23.809745    2547 generic.go:356] "Generic (PLEG): container finished" podID="50775e4e-6b60-4dec-91ec-b006bef3d00e" containerID="c59fd1230540f31dd86cfef4bca31e92995a7efb7b2ae464b70fb08193335ceb" exitCode=0
Sep 10 14:20:23.809913 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:23.809786    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-czbln" event={"ID":"50775e4e-6b60-4dec-91ec-b006bef3d00e","Type":"ContainerDied","Data":"c59fd1230540f31dd86cfef4bca31e92995a7efb7b2ae464b70fb08193335ceb"}
Sep 10 14:20:23.813448 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:23.813424    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-957xl"]
Sep 10 14:20:23.829575 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:23.829542    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2c9kh"]
Sep 10 14:20:23.829796 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:23.829777    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-957xl"
Sep 10 14:20:23.833237 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:23.833214    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 14:20:23.834148 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:23.834040    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-587sj\""
Sep 10 14:20:23.834293 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:23.834253    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 14:20:23.841837 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:23.841813    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-957xl"]
Sep 10 14:20:23.841928 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:23.841860    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2c9kh"]
Sep 10 14:20:23.841989 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:23.841929    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2c9kh"
Sep 10 14:20:23.844487 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:23.844467    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 14:20:23.844568 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:23.844504    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-45wsb\""
Sep 10 14:20:23.844688 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:23.844662    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 14:20:23.844764 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:23.844742    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 14:20:23.960956 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:23.960792    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-config-volume\") pod \"dns-default-957xl\" (UID: \"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798\") " pod="openshift-dns/dns-default-957xl"
Sep 10 14:20:23.961053 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:23.960967    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6415697-570a-4f1f-9317-7e774d60cc04-cert\") pod \"ingress-canary-2c9kh\" (UID: \"d6415697-570a-4f1f-9317-7e774d60cc04\") " pod="openshift-ingress-canary/ingress-canary-2c9kh"
Sep 10 14:20:23.961053 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:23.960986    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8tsx\" (UniqueName: \"kubernetes.io/projected/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-kube-api-access-g8tsx\") pod \"dns-default-957xl\" (UID: \"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798\") " pod="openshift-dns/dns-default-957xl"
Sep 10 14:20:23.961053 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:23.961011    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdzmv\" (UniqueName: \"kubernetes.io/projected/d6415697-570a-4f1f-9317-7e774d60cc04-kube-api-access-kdzmv\") pod \"ingress-canary-2c9kh\" (UID: \"d6415697-570a-4f1f-9317-7e774d60cc04\") " pod="openshift-ingress-canary/ingress-canary-2c9kh"
Sep 10 14:20:23.961053 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:23.961027    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls\") pod \"dns-default-957xl\" (UID: \"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798\") " pod="openshift-dns/dns-default-957xl"
Sep 10 14:20:23.961239 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:23.961140    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-tmp-dir\") pod \"dns-default-957xl\" (UID: \"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798\") " pod="openshift-dns/dns-default-957xl"
Sep 10 14:20:24.062171 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.062094    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-config-volume\") pod \"dns-default-957xl\" (UID: \"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798\") " pod="openshift-dns/dns-default-957xl"
Sep 10 14:20:24.062171 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.062130    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6415697-570a-4f1f-9317-7e774d60cc04-cert\") pod \"ingress-canary-2c9kh\" (UID: \"d6415697-570a-4f1f-9317-7e774d60cc04\") " pod="openshift-ingress-canary/ingress-canary-2c9kh"
Sep 10 14:20:24.062171 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.062157    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8tsx\" (UniqueName: \"kubernetes.io/projected/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-kube-api-access-g8tsx\") pod \"dns-default-957xl\" (UID: \"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798\") " pod="openshift-dns/dns-default-957xl"
Sep 10 14:20:24.062417 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.062196    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdzmv\" (UniqueName: \"kubernetes.io/projected/d6415697-570a-4f1f-9317-7e774d60cc04-kube-api-access-kdzmv\") pod \"ingress-canary-2c9kh\" (UID: \"d6415697-570a-4f1f-9317-7e774d60cc04\") " pod="openshift-ingress-canary/ingress-canary-2c9kh"
Sep 10 14:20:24.062417 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.062219    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls\") pod \"dns-default-957xl\" (UID: \"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798\") " pod="openshift-dns/dns-default-957xl"
Sep 10 14:20:24.062417 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.062272    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-tmp-dir\") pod \"dns-default-957xl\" (UID: \"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798\") " pod="openshift-dns/dns-default-957xl"
Sep 10 14:20:24.062417 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:24.062360    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:20:24.062586 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:24.062442    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls podName:54e8b08a-0bfa-4e3f-87e3-d6a5e886c798 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:24.562420896 +0000 UTC m=+36.604295374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls") pod "dns-default-957xl" (UID: "54e8b08a-0bfa-4e3f-87e3-d6a5e886c798") : secret "dns-default-metrics-tls" not found
Sep 10 14:20:24.062586 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.062553    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-tmp-dir\") pod \"dns-default-957xl\" (UID: \"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798\") " pod="openshift-dns/dns-default-957xl"
Sep 10 14:20:24.062792 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.062774    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-config-volume\") pod \"dns-default-957xl\" (UID: \"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798\") " pod="openshift-dns/dns-default-957xl"
Sep 10 14:20:24.066375 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.066353    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6415697-570a-4f1f-9317-7e774d60cc04-cert\") pod \"ingress-canary-2c9kh\" (UID: \"d6415697-570a-4f1f-9317-7e774d60cc04\") " pod="openshift-ingress-canary/ingress-canary-2c9kh"
Sep 10 14:20:24.070228 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.070206    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8tsx\" (UniqueName: \"kubernetes.io/projected/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-kube-api-access-g8tsx\") pod \"dns-default-957xl\" (UID: \"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798\") " pod="openshift-dns/dns-default-957xl"
Sep 10 14:20:24.070312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.070282    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdzmv\" (UniqueName: \"kubernetes.io/projected/d6415697-570a-4f1f-9317-7e774d60cc04-kube-api-access-kdzmv\") pod \"ingress-canary-2c9kh\" (UID: \"d6415697-570a-4f1f-9317-7e774d60cc04\") " pod="openshift-ingress-canary/ingress-canary-2c9kh"
Sep 10 14:20:24.154034 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.154003    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2c9kh"
Sep 10 14:20:24.418370 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.418340    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:20:24.418534 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.418340    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:20:24.418534 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.418470    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:20:24.421001 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.420982    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 14:20:24.421118 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.421052    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6szzq\""
Sep 10 14:20:24.421118 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.421059    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-flc9p\""
Sep 10 14:20:24.421118 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.421101    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 14:20:24.421269 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.421162    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 14:20:24.421359 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.421346    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 14:20:24.453509 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.453484    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2c9kh"]
Sep 10 14:20:24.457505 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:20:24.457482    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6415697_570a_4f1f_9317_7e774d60cc04.slice/crio-8dec6c27020c718c01191c59b271ec087d7025f75c4fa63a6718d9566bbc1887 WatchSource:0}: Error finding container 8dec6c27020c718c01191c59b271ec087d7025f75c4fa63a6718d9566bbc1887: Status 404 returned error can't find the container with id 8dec6c27020c718c01191c59b271ec087d7025f75c4fa63a6718d9566bbc1887
Sep 10 14:20:24.565634 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.565605    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls\") pod \"dns-default-957xl\" (UID: \"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798\") " pod="openshift-dns/dns-default-957xl"
Sep 10 14:20:24.565796 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:24.565758    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:20:24.565844 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:24.565821    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls podName:54e8b08a-0bfa-4e3f-87e3-d6a5e886c798 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:25.565806129 +0000 UTC m=+37.607680613 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls") pod "dns-default-957xl" (UID: "54e8b08a-0bfa-4e3f-87e3-d6a5e886c798") : secret "dns-default-metrics-tls" not found
Sep 10 14:20:24.813306 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.813223    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2c9kh" event={"ID":"d6415697-570a-4f1f-9317-7e774d60cc04","Type":"ContainerStarted","Data":"8dec6c27020c718c01191c59b271ec087d7025f75c4fa63a6718d9566bbc1887"}
Sep 10 14:20:24.816344 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.816316    2547 generic.go:356] "Generic (PLEG): container finished" podID="50775e4e-6b60-4dec-91ec-b006bef3d00e" containerID="c8f54d82acd7ddec7cab3b2b2df6efbe2e8f136dc21e2e07a890eb00c594151a" exitCode=0
Sep 10 14:20:24.816452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:24.816349    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-czbln" event={"ID":"50775e4e-6b60-4dec-91ec-b006bef3d00e","Type":"ContainerDied","Data":"c8f54d82acd7ddec7cab3b2b2df6efbe2e8f136dc21e2e07a890eb00c594151a"}
Sep 10 14:20:25.575077 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:25.574859    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls\") pod \"dns-default-957xl\" (UID: \"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798\") " pod="openshift-dns/dns-default-957xl"
Sep 10 14:20:25.575255 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:25.575039    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:20:25.575255 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:25.575187    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls podName:54e8b08a-0bfa-4e3f-87e3-d6a5e886c798 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:27.575166829 +0000 UTC m=+39.617041302 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls") pod "dns-default-957xl" (UID: "54e8b08a-0bfa-4e3f-87e3-d6a5e886c798") : secret "dns-default-metrics-tls" not found
Sep 10 14:20:25.823902 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:25.823860    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-czbln" event={"ID":"50775e4e-6b60-4dec-91ec-b006bef3d00e","Type":"ContainerStarted","Data":"0a4a2f693e6b0ec4e950d1e265f13a2e8b126d08025ad7ccd2e58423d7a7ed47"}
Sep 10 14:20:25.848272 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:25.848163    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-czbln" podStartSLOduration=6.347462262 podStartE2EDuration="36.848142549s" totalImagesPullingTime="30.500680287s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:25.847392871 +0000 UTC m=+37.889267353" watchObservedRunningTime="2026-09-10 14:20:25.848142549 +0000 UTC m=+37.890017032"
Sep 10 14:20:27.591712 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:27.591664    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls\") pod \"dns-default-957xl\" (UID: \"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798\") " pod="openshift-dns/dns-default-957xl"
Sep 10 14:20:27.592085 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:27.591826    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:20:27.592085 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:27.591897    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls podName:54e8b08a-0bfa-4e3f-87e3-d6a5e886c798 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:31.591879556 +0000 UTC m=+43.633754035 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls") pod "dns-default-957xl" (UID: "54e8b08a-0bfa-4e3f-87e3-d6a5e886c798") : secret "dns-default-metrics-tls" not found
Sep 10 14:20:27.831190 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:27.831148    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2c9kh" event={"ID":"d6415697-570a-4f1f-9317-7e774d60cc04","Type":"ContainerStarted","Data":"34cb5f0dddde4c5342a57118507d0ee145b84f295fa680f8fabc11d6c468417f"}
Sep 10 14:20:27.850908 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:27.850844    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2c9kh" podStartSLOduration=1.956033429 podStartE2EDuration="4.85082571s" totalImagesPullingTime="2.894792281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:20:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:27.85029488 +0000 UTC m=+39.892169360" watchObservedRunningTime="2026-09-10 14:20:27.85082571 +0000 UTC m=+39.892700188"
Sep 10 14:20:31.620767 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:31.620726    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls\") pod \"dns-default-957xl\" (UID: \"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798\") " pod="openshift-dns/dns-default-957xl"
Sep 10 14:20:31.621150 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:31.620875    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:20:31.621150 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:31.620949    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls podName:54e8b08a-0bfa-4e3f-87e3-d6a5e886c798 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:39.620931757 +0000 UTC m=+51.662806232 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls") pod "dns-default-957xl" (UID: "54e8b08a-0bfa-4e3f-87e3-d6a5e886c798") : secret "dns-default-metrics-tls" not found
Sep 10 14:20:39.676321 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:39.676275    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls\") pod \"dns-default-957xl\" (UID: \"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798\") " pod="openshift-dns/dns-default-957xl"
Sep 10 14:20:39.676911 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:39.676460    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:20:39.676911 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:39.676543    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls podName:54e8b08a-0bfa-4e3f-87e3-d6a5e886c798 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:55.676520989 +0000 UTC m=+67.718395465 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls") pod "dns-default-957xl" (UID: "54e8b08a-0bfa-4e3f-87e3-d6a5e886c798") : secret "dns-default-metrics-tls" not found
Sep 10 14:20:47.801016 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:47.800983    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ww4p"
Sep 10 14:20:53.067183 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:53.067141    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs\") pod \"network-metrics-daemon-756dn\" (UID: \"c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0\") " pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:20:53.067183 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:53.067183    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e33264c-464d-41c9-8d74-b98e7b92b510-original-pull-secret\") pod \"global-pull-secret-syncer-m2v46\" (UID: \"4e33264c-464d-41c9-8d74-b98e7b92b510\") " pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:20:53.069865 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:53.069846    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 14:20:53.069990 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:53.069974    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 14:20:53.077939 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:53.077920    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 10 14:20:53.078023 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:53.077989    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs podName:c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0 nodeName:}" failed. No retries permitted until 2026-09-10 14:21:57.077969486 +0000 UTC m=+129.119843962 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs") pod "network-metrics-daemon-756dn" (UID: "c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0") : secret "metrics-daemon-secret" not found
Sep 10 14:20:53.080870 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:53.080842    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e33264c-464d-41c9-8d74-b98e7b92b510-original-pull-secret\") pod \"global-pull-secret-syncer-m2v46\" (UID: \"4e33264c-464d-41c9-8d74-b98e7b92b510\") " pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:20:53.228056 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:53.228025    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m2v46"
Sep 10 14:20:53.268546 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:53.268518    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9x5x\" (UniqueName: \"kubernetes.io/projected/661c9771-a727-49a4-8022-0a265a524e72-kube-api-access-q9x5x\") pod \"network-check-target-d66xs\" (UID: \"661c9771-a727-49a4-8022-0a265a524e72\") " pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:20:53.271425 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:53.271403    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 14:20:53.282491 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:53.282464    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 14:20:53.292399 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:53.292375    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9x5x\" (UniqueName: \"kubernetes.io/projected/661c9771-a727-49a4-8022-0a265a524e72-kube-api-access-q9x5x\") pod \"network-check-target-d66xs\" (UID: \"661c9771-a727-49a4-8022-0a265a524e72\") " pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:20:53.524083 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:53.524052    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m2v46"]
Sep 10 14:20:53.527535 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:20:53.527507    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e33264c_464d_41c9_8d74_b98e7b92b510.slice/crio-43039d3fb711316f6a916600e71f5540e4b771498413c180fad42c74d16acaac WatchSource:0}: Error finding container 43039d3fb711316f6a916600e71f5540e4b771498413c180fad42c74d16acaac: Status 404 returned error can't find the container with id 43039d3fb711316f6a916600e71f5540e4b771498413c180fad42c74d16acaac
Sep 10 14:20:53.536254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:53.536236    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6szzq\""
Sep 10 14:20:53.543981 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:53.543965    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:20:53.840477 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:53.840405    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d66xs"]
Sep 10 14:20:53.843379 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:20:53.843351    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661c9771_a727_49a4_8022_0a265a524e72.slice/crio-519b1153dd37f3fe04b068628aa22f6b20e0760bfeea095d4bc4e60caf8effc9 WatchSource:0}: Error finding container 519b1153dd37f3fe04b068628aa22f6b20e0760bfeea095d4bc4e60caf8effc9: Status 404 returned error can't find the container with id 519b1153dd37f3fe04b068628aa22f6b20e0760bfeea095d4bc4e60caf8effc9
Sep 10 14:20:53.893888 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:53.893845    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d66xs" event={"ID":"661c9771-a727-49a4-8022-0a265a524e72","Type":"ContainerStarted","Data":"519b1153dd37f3fe04b068628aa22f6b20e0760bfeea095d4bc4e60caf8effc9"}
Sep 10 14:20:53.894862 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:53.894835    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m2v46" event={"ID":"4e33264c-464d-41c9-8d74-b98e7b92b510","Type":"ContainerStarted","Data":"43039d3fb711316f6a916600e71f5540e4b771498413c180fad42c74d16acaac"}
Sep 10 14:20:55.687422 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:55.687376    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls\") pod \"dns-default-957xl\" (UID: \"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798\") " pod="openshift-dns/dns-default-957xl"
Sep 10 14:20:55.688028 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:55.687545    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:20:55.688028 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:20:55.687643    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls podName:54e8b08a-0bfa-4e3f-87e3-d6a5e886c798 nodeName:}" failed. No retries permitted until 2026-09-10 14:21:27.687619932 +0000 UTC m=+99.729494409 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls") pod "dns-default-957xl" (UID: "54e8b08a-0bfa-4e3f-87e3-d6a5e886c798") : secret "dns-default-metrics-tls" not found
Sep 10 14:20:58.910119 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:58.910083    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d66xs" event={"ID":"661c9771-a727-49a4-8022-0a265a524e72","Type":"ContainerStarted","Data":"281dd17e677a6a57421b2142c73c4e5eb112923251646f2366d6bd65d1e8adab"}
Sep 10 14:20:58.910550 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:58.910164    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:20:58.911539 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:58.911514    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m2v46" event={"ID":"4e33264c-464d-41c9-8d74-b98e7b92b510","Type":"ContainerStarted","Data":"c0ddec8497ccc46ffb8f1405d098debc51550dc620a014839e28a0594e656e74"}
Sep 10 14:20:58.931384 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:58.931341    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-d66xs" podStartSLOduration=65.656229587 podStartE2EDuration="1m9.931326725s" totalImagesPullingTime="4.275097138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:58.929006542 +0000 UTC m=+70.970881022" watchObservedRunningTime="2026-09-10 14:20:58.931326725 +0000 UTC m=+70.973201205"
Sep 10 14:20:58.948647 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:20:58.948607    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-m2v46" podStartSLOduration=65.354013596 podStartE2EDuration="1m9.948595169s" totalImagesPullingTime="4.594581573s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:58.94810315 +0000 UTC m=+70.989977631" watchObservedRunningTime="2026-09-10 14:20:58.948595169 +0000 UTC m=+70.990469650"
Sep 10 14:21:17.441938 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:17.441798    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5767c46556-cczp8"]
Sep 10 14:21:17.446196 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:17.446165    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5767c46556-cczp8"
Sep 10 14:21:17.448817 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:17.448797    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 14:21:17.449782 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:17.449763    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 14:21:17.449855 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:17.449775    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 14:21:17.449855 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:17.449763    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 10 14:21:17.460710 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:17.460667    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5767c46556-cczp8"]
Sep 10 14:21:17.527504 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:17.527466    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66j6b\" (UniqueName: \"kubernetes.io/projected/bd413231-2c80-4bfc-b49d-047e1536e635-kube-api-access-66j6b\") pod \"klusterlet-addon-workmgr-5767c46556-cczp8\" (UID: \"bd413231-2c80-4bfc-b49d-047e1536e635\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5767c46556-cczp8"
Sep 10 14:21:17.527681 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:17.527512    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd413231-2c80-4bfc-b49d-047e1536e635-tmp\") pod \"klusterlet-addon-workmgr-5767c46556-cczp8\" (UID: \"bd413231-2c80-4bfc-b49d-047e1536e635\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5767c46556-cczp8"
Sep 10 14:21:17.527681 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:17.527542    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bd413231-2c80-4bfc-b49d-047e1536e635-klusterlet-config\") pod \"klusterlet-addon-workmgr-5767c46556-cczp8\" (UID: \"bd413231-2c80-4bfc-b49d-047e1536e635\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5767c46556-cczp8"
Sep 10 14:21:17.628493 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:17.628449    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66j6b\" (UniqueName: \"kubernetes.io/projected/bd413231-2c80-4bfc-b49d-047e1536e635-kube-api-access-66j6b\") pod \"klusterlet-addon-workmgr-5767c46556-cczp8\" (UID: \"bd413231-2c80-4bfc-b49d-047e1536e635\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5767c46556-cczp8"
Sep 10 14:21:17.628573 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:17.628509    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd413231-2c80-4bfc-b49d-047e1536e635-tmp\") pod \"klusterlet-addon-workmgr-5767c46556-cczp8\" (UID: \"bd413231-2c80-4bfc-b49d-047e1536e635\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5767c46556-cczp8"
Sep 10 14:21:17.628654 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:17.628630    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bd413231-2c80-4bfc-b49d-047e1536e635-klusterlet-config\") pod \"klusterlet-addon-workmgr-5767c46556-cczp8\" (UID: \"bd413231-2c80-4bfc-b49d-047e1536e635\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5767c46556-cczp8"
Sep 10 14:21:17.628898 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:17.628880    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd413231-2c80-4bfc-b49d-047e1536e635-tmp\") pod \"klusterlet-addon-workmgr-5767c46556-cczp8\" (UID: \"bd413231-2c80-4bfc-b49d-047e1536e635\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5767c46556-cczp8"
Sep 10 14:21:17.630992 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:17.630966    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bd413231-2c80-4bfc-b49d-047e1536e635-klusterlet-config\") pod \"klusterlet-addon-workmgr-5767c46556-cczp8\" (UID: \"bd413231-2c80-4bfc-b49d-047e1536e635\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5767c46556-cczp8"
Sep 10 14:21:17.639881 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:17.639857    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66j6b\" (UniqueName: \"kubernetes.io/projected/bd413231-2c80-4bfc-b49d-047e1536e635-kube-api-access-66j6b\") pod \"klusterlet-addon-workmgr-5767c46556-cczp8\" (UID: \"bd413231-2c80-4bfc-b49d-047e1536e635\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5767c46556-cczp8"
Sep 10 14:21:17.755162 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:17.755069    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5767c46556-cczp8"
Sep 10 14:21:18.023883 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:18.023812    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5767c46556-cczp8"]
Sep 10 14:21:18.027291 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:21:18.027250    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd413231_2c80_4bfc_b49d_047e1536e635.slice/crio-f51df53743244789403ec1c5a5296bd8aeab8da55bfff02f32399e05d931e403 WatchSource:0}: Error finding container f51df53743244789403ec1c5a5296bd8aeab8da55bfff02f32399e05d931e403: Status 404 returned error can't find the container with id f51df53743244789403ec1c5a5296bd8aeab8da55bfff02f32399e05d931e403
Sep 10 14:21:18.960426 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:18.960380    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5767c46556-cczp8" event={"ID":"bd413231-2c80-4bfc-b49d-047e1536e635","Type":"ContainerStarted","Data":"f51df53743244789403ec1c5a5296bd8aeab8da55bfff02f32399e05d931e403"}
Sep 10 14:21:21.969712 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:21.969672    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5767c46556-cczp8" event={"ID":"bd413231-2c80-4bfc-b49d-047e1536e635","Type":"ContainerStarted","Data":"d6bb4a14a8a5bb90e3e3d1435fd8ebb517c75bef6f5aa02de23bedce679ff44f"}
Sep 10 14:21:21.970131 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:21.969917    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5767c46556-cczp8"
Sep 10 14:21:21.971559 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:21.971538    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5767c46556-cczp8"
Sep 10 14:21:21.986797 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:21.986746    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5767c46556-cczp8" podStartSLOduration=1.4003729200000001 podStartE2EDuration="4.986732316s" totalImagesPullingTime="3.586359396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:21:21.985944328 +0000 UTC m=+94.027818809" watchObservedRunningTime="2026-09-10 14:21:21.986732316 +0000 UTC m=+94.028606797"
Sep 10 14:21:27.700348 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:27.700307    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls\") pod \"dns-default-957xl\" (UID: \"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798\") " pod="openshift-dns/dns-default-957xl"
Sep 10 14:21:27.700819 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:21:27.700424    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:21:27.700819 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:21:27.700481    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls podName:54e8b08a-0bfa-4e3f-87e3-d6a5e886c798 nodeName:}" failed. No retries permitted until 2026-09-10 14:22:31.700467293 +0000 UTC m=+163.742341765 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls") pod "dns-default-957xl" (UID: "54e8b08a-0bfa-4e3f-87e3-d6a5e886c798") : secret "dns-default-metrics-tls" not found
Sep 10 14:21:29.915590 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:29.915555    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-d66xs"
Sep 10 14:21:57.097173 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:21:57.097138    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs\") pod \"network-metrics-daemon-756dn\" (UID: \"c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0\") " pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:21:57.097637 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:21:57.097261    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 10 14:21:57.097637 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:21:57.097340    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs podName:c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0 nodeName:}" failed. No retries permitted until 2026-09-10 14:23:59.097319292 +0000 UTC m=+251.139193765 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs") pod "network-metrics-daemon-756dn" (UID: "c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0") : secret "metrics-daemon-secret" not found
Sep 10 14:22:26.846497 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:22:26.846440    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-957xl" podUID="54e8b08a-0bfa-4e3f-87e3-d6a5e886c798"
Sep 10 14:22:27.127100 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:22:27.127069    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-957xl"
Sep 10 14:22:27.439407 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:22:27.439372    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-756dn" podUID="c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0"
Sep 10 14:22:31.722589 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:22:31.722553    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls\") pod \"dns-default-957xl\" (UID: \"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798\") " pod="openshift-dns/dns-default-957xl"
Sep 10 14:22:31.722963 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:22:31.722739    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:22:31.722963 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:22:31.722811    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls podName:54e8b08a-0bfa-4e3f-87e3-d6a5e886c798 nodeName:}" failed. No retries permitted until 2026-09-10 14:24:33.722791703 +0000 UTC m=+285.764666180 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls") pod "dns-default-957xl" (UID: "54e8b08a-0bfa-4e3f-87e3-d6a5e886c798") : secret "dns-default-metrics-tls" not found
Sep 10 14:22:38.416752 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:22:38.416712    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:23:21.970573 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:23:21.970471    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5767c46556-cczp8" podUID="bd413231-2c80-4bfc-b49d-047e1536e635" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.8:8000/readyz\": dial tcp 10.132.0.8:8000: connect: connection refused"
Sep 10 14:23:22.257742 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:23:22.257642    2547 generic.go:356] "Generic (PLEG): container finished" podID="bd413231-2c80-4bfc-b49d-047e1536e635" containerID="d6bb4a14a8a5bb90e3e3d1435fd8ebb517c75bef6f5aa02de23bedce679ff44f" exitCode=1
Sep 10 14:23:22.257742 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:23:22.257715    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5767c46556-cczp8" event={"ID":"bd413231-2c80-4bfc-b49d-047e1536e635","Type":"ContainerDied","Data":"d6bb4a14a8a5bb90e3e3d1435fd8ebb517c75bef6f5aa02de23bedce679ff44f"}
Sep 10 14:23:22.258021 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:23:22.258007    2547 scope.go:117] "RemoveContainer" containerID="d6bb4a14a8a5bb90e3e3d1435fd8ebb517c75bef6f5aa02de23bedce679ff44f"
Sep 10 14:23:23.262090 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:23:23.262057    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5767c46556-cczp8" event={"ID":"bd413231-2c80-4bfc-b49d-047e1536e635","Type":"ContainerStarted","Data":"50aa06c2b81b77506c17d0184a85434e3cdd3d2fade470884b68a4371e584cac"}
Sep 10 14:23:23.262522 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:23:23.262333    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5767c46556-cczp8"
Sep 10 14:23:23.262955 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:23:23.262938    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5767c46556-cczp8"
Sep 10 14:23:59.102284 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:23:59.102229    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs\") pod \"network-metrics-daemon-756dn\" (UID: \"c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0\") " pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:23:59.102754 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:23:59.102357    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 10 14:23:59.102754 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:23:59.102421    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs podName:c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0 nodeName:}" failed. No retries permitted until 2026-09-10 14:26:01.10240554 +0000 UTC m=+373.144280012 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs") pod "network-metrics-daemon-756dn" (UID: "c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0") : secret "metrics-daemon-secret" not found
Sep 10 14:24:20.589845 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.589753    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-mg9d6"]
Sep 10 14:24:20.592743 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.592718    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-76f86fcc44-d46qd"]
Sep 10 14:24:20.592861 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.592833    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-mg9d6"
Sep 10 14:24:20.594421 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.594395    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79489b5cbc-zgr4x"]
Sep 10 14:24:20.594536 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.594503    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:20.596032 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.596012    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-7dvk2"]
Sep 10 14:24:20.596128 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.596108    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:20.597660 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.597641    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7dvk2"
Sep 10 14:24:20.597987 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.597970    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 10 14:24:20.606422 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.606264    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-c55ds\""
Sep 10 14:24:20.606422 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.606297    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-62l8z\""
Sep 10 14:24:20.606692 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.606655    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 10 14:24:20.607333 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.606663    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 14:24:20.607333 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.606803    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-fdrtb\""
Sep 10 14:24:20.607475 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.606832    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:24:20.607475 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.606838    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 10 14:24:20.607573 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.607480    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vzc7d\""
Sep 10 14:24:20.607573 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.606839    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 10 14:24:20.607573 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.606883    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 10 14:24:20.607740 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.606949    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 10 14:24:20.607740 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.606960    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 10 14:24:20.607740 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.606961    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:24:20.607740 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.607000    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 14:24:20.607925 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.607006    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 14:24:20.607925 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.607027    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 14:24:20.607925 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.607093    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 10 14:24:20.607925 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.607095    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 14:24:20.607925 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.607128    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 10 14:24:20.608135 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.607953    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-mg9d6"]
Sep 10 14:24:20.608135 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.607055    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 10 14:24:20.610106 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.610087    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 10 14:24:20.616727 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.616708    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 14:24:20.626922 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.626898    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79489b5cbc-zgr4x"]
Sep 10 14:24:20.627668 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.627648    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76f86fcc44-d46qd"]
Sep 10 14:24:20.639485 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.639460    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/359d40e9-f428-4cc3-b4e4-991c081d664a-config\") pod \"console-operator-85b454874b-mg9d6\" (UID: \"359d40e9-f428-4cc3-b4e4-991c081d664a\") " pod="openshift-console-operator/console-operator-85b454874b-mg9d6"
Sep 10 14:24:20.639585 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.639490    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-tls\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:20.639585 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.639509    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfrv8\" (UniqueName: \"kubernetes.io/projected/47ae46ed-3dac-4491-9f8c-098e66424ebc-kube-api-access-pfrv8\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:20.639585 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.639524    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6e2abaa4-6e03-493a-90ef-833dbab4d41e-trusted-ca\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:20.639585 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.639556    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrz7r\" (UniqueName: \"kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-kube-api-access-qrz7r\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:20.639756 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.639648    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6e2abaa4-6e03-493a-90ef-833dbab4d41e-image-registry-private-configuration\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:20.639756 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.639671    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6e2abaa4-6e03-493a-90ef-833dbab4d41e-installation-pull-secrets\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:20.639756 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.639691    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/43b9ae1f-18aa-4310-bdcd-4d9b296d4e42-config\") pod \"service-ca-operator-859f94c4b-7dvk2\" (UID: \"43b9ae1f-18aa-4310-bdcd-4d9b296d4e42\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7dvk2"
Sep 10 14:24:20.639756 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.639749    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-bound-sa-token\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:20.639887 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.639771    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcz4f\" (UniqueName: \"kubernetes.io/projected/359d40e9-f428-4cc3-b4e4-991c081d664a-kube-api-access-pcz4f\") pod \"console-operator-85b454874b-mg9d6\" (UID: \"359d40e9-f428-4cc3-b4e4-991c081d664a\") " pod="openshift-console-operator/console-operator-85b454874b-mg9d6"
Sep 10 14:24:20.639887 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.639785    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6e2abaa4-6e03-493a-90ef-833dbab4d41e-ca-trust-extracted\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:20.639887 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.639835    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/47ae46ed-3dac-4491-9f8c-098e66424ebc-stats-auth\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:20.639887 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.639864    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47ae46ed-3dac-4491-9f8c-098e66424ebc-service-ca-bundle\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:20.640024 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.639886    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjknd\" (UniqueName: \"kubernetes.io/projected/43b9ae1f-18aa-4310-bdcd-4d9b296d4e42-kube-api-access-tjknd\") pod \"service-ca-operator-859f94c4b-7dvk2\" (UID: \"43b9ae1f-18aa-4310-bdcd-4d9b296d4e42\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7dvk2"
Sep 10 14:24:20.640024 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.639907    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ae46ed-3dac-4491-9f8c-098e66424ebc-metrics-certs\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:20.640024 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.639928    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-certificates\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:20.640024 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.639955    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/43b9ae1f-18aa-4310-bdcd-4d9b296d4e42-serving-cert\") pod \"service-ca-operator-859f94c4b-7dvk2\" (UID: \"43b9ae1f-18aa-4310-bdcd-4d9b296d4e42\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7dvk2"
Sep 10 14:24:20.640024 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.639981    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/359d40e9-f428-4cc3-b4e4-991c081d664a-serving-cert\") pod \"console-operator-85b454874b-mg9d6\" (UID: \"359d40e9-f428-4cc3-b4e4-991c081d664a\") " pod="openshift-console-operator/console-operator-85b454874b-mg9d6"
Sep 10 14:24:20.640024 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.639997    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/47ae46ed-3dac-4491-9f8c-098e66424ebc-default-certificate\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:20.640024 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.640011    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/359d40e9-f428-4cc3-b4e4-991c081d664a-trusted-ca\") pod \"console-operator-85b454874b-mg9d6\" (UID: \"359d40e9-f428-4cc3-b4e4-991c081d664a\") " pod="openshift-console-operator/console-operator-85b454874b-mg9d6"
Sep 10 14:24:20.643090 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.643069    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-7dvk2"]
Sep 10 14:24:20.741281 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.741245    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfrv8\" (UniqueName: \"kubernetes.io/projected/47ae46ed-3dac-4491-9f8c-098e66424ebc-kube-api-access-pfrv8\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:20.741281 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.741281    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6e2abaa4-6e03-493a-90ef-833dbab4d41e-trusted-ca\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:20.741533 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.741300    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrz7r\" (UniqueName: \"kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-kube-api-access-qrz7r\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:20.741533 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.741416    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6e2abaa4-6e03-493a-90ef-833dbab4d41e-image-registry-private-configuration\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:20.741533 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.741452    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6e2abaa4-6e03-493a-90ef-833dbab4d41e-installation-pull-secrets\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:20.741533 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.741472    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/43b9ae1f-18aa-4310-bdcd-4d9b296d4e42-config\") pod \"service-ca-operator-859f94c4b-7dvk2\" (UID: \"43b9ae1f-18aa-4310-bdcd-4d9b296d4e42\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7dvk2"
Sep 10 14:24:20.741767 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.741607    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-bound-sa-token\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:20.741767 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.741651    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcz4f\" (UniqueName: \"kubernetes.io/projected/359d40e9-f428-4cc3-b4e4-991c081d664a-kube-api-access-pcz4f\") pod \"console-operator-85b454874b-mg9d6\" (UID: \"359d40e9-f428-4cc3-b4e4-991c081d664a\") " pod="openshift-console-operator/console-operator-85b454874b-mg9d6"
Sep 10 14:24:20.741767 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.741677    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6e2abaa4-6e03-493a-90ef-833dbab4d41e-ca-trust-extracted\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:20.741767 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.741735    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/47ae46ed-3dac-4491-9f8c-098e66424ebc-stats-auth\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:20.741962 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.741782    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47ae46ed-3dac-4491-9f8c-098e66424ebc-service-ca-bundle\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:20.741962 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.741807    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjknd\" (UniqueName: \"kubernetes.io/projected/43b9ae1f-18aa-4310-bdcd-4d9b296d4e42-kube-api-access-tjknd\") pod \"service-ca-operator-859f94c4b-7dvk2\" (UID: \"43b9ae1f-18aa-4310-bdcd-4d9b296d4e42\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7dvk2"
Sep 10 14:24:20.742086 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:20.742064    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/47ae46ed-3dac-4491-9f8c-098e66424ebc-service-ca-bundle podName:47ae46ed-3dac-4491-9f8c-098e66424ebc nodeName:}" failed. No retries permitted until 2026-09-10 14:24:21.24204303 +0000 UTC m=+273.283917511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/47ae46ed-3dac-4491-9f8c-098e66424ebc-service-ca-bundle") pod "router-default-76f86fcc44-d46qd" (UID: "47ae46ed-3dac-4491-9f8c-098e66424ebc") : configmap references non-existent config key: service-ca.crt
Sep 10 14:24:20.742160 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.742116    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/43b9ae1f-18aa-4310-bdcd-4d9b296d4e42-config\") pod \"service-ca-operator-859f94c4b-7dvk2\" (UID: \"43b9ae1f-18aa-4310-bdcd-4d9b296d4e42\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7dvk2"
Sep 10 14:24:20.742278 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.742245    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ae46ed-3dac-4491-9f8c-098e66424ebc-metrics-certs\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:20.742344 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.742296    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-certificates\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:20.742344 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.742297    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6e2abaa4-6e03-493a-90ef-833dbab4d41e-ca-trust-extracted\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:20.742446 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.742361    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/43b9ae1f-18aa-4310-bdcd-4d9b296d4e42-serving-cert\") pod \"service-ca-operator-859f94c4b-7dvk2\" (UID: \"43b9ae1f-18aa-4310-bdcd-4d9b296d4e42\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7dvk2"
Sep 10 14:24:20.742446 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.742386    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/359d40e9-f428-4cc3-b4e4-991c081d664a-serving-cert\") pod \"console-operator-85b454874b-mg9d6\" (UID: \"359d40e9-f428-4cc3-b4e4-991c081d664a\") " pod="openshift-console-operator/console-operator-85b454874b-mg9d6"
Sep 10 14:24:20.742446 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.742412    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/47ae46ed-3dac-4491-9f8c-098e66424ebc-default-certificate\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:20.742600 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.742453    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/359d40e9-f428-4cc3-b4e4-991c081d664a-trusted-ca\") pod \"console-operator-85b454874b-mg9d6\" (UID: \"359d40e9-f428-4cc3-b4e4-991c081d664a\") " pod="openshift-console-operator/console-operator-85b454874b-mg9d6"
Sep 10 14:24:20.742600 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.742478    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6e2abaa4-6e03-493a-90ef-833dbab4d41e-trusted-ca\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:20.742600 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:20.742488    2547 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 14:24:20.742600 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.742479    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/359d40e9-f428-4cc3-b4e4-991c081d664a-config\") pod \"console-operator-85b454874b-mg9d6\" (UID: \"359d40e9-f428-4cc3-b4e4-991c081d664a\") " pod="openshift-console-operator/console-operator-85b454874b-mg9d6"
Sep 10 14:24:20.742600 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.742565    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-tls\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:20.742883 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:20.742678    2547 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 14:24:20.742883 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:20.742692    2547 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79489b5cbc-zgr4x: secret "image-registry-tls" not found
Sep 10 14:24:20.742883 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:20.742756    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-tls podName:6e2abaa4-6e03-493a-90ef-833dbab4d41e nodeName:}" failed. No retries permitted until 2026-09-10 14:24:21.242739046 +0000 UTC m=+273.284613523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-tls") pod "image-registry-79489b5cbc-zgr4x" (UID: "6e2abaa4-6e03-493a-90ef-833dbab4d41e") : secret "image-registry-tls" not found
Sep 10 14:24:20.742883 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:20.742793    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47ae46ed-3dac-4491-9f8c-098e66424ebc-metrics-certs podName:47ae46ed-3dac-4491-9f8c-098e66424ebc nodeName:}" failed. No retries permitted until 2026-09-10 14:24:21.242783219 +0000 UTC m=+273.284657697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47ae46ed-3dac-4491-9f8c-098e66424ebc-metrics-certs") pod "router-default-76f86fcc44-d46qd" (UID: "47ae46ed-3dac-4491-9f8c-098e66424ebc") : secret "router-metrics-certs-default" not found
Sep 10 14:24:20.742883 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.742811    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-certificates\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:20.743254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.743233    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/359d40e9-f428-4cc3-b4e4-991c081d664a-config\") pod \"console-operator-85b454874b-mg9d6\" (UID: \"359d40e9-f428-4cc3-b4e4-991c081d664a\") " pod="openshift-console-operator/console-operator-85b454874b-mg9d6"
Sep 10 14:24:20.743622 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.743595    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/359d40e9-f428-4cc3-b4e4-991c081d664a-trusted-ca\") pod \"console-operator-85b454874b-mg9d6\" (UID: \"359d40e9-f428-4cc3-b4e4-991c081d664a\") " pod="openshift-console-operator/console-operator-85b454874b-mg9d6"
Sep 10 14:24:20.744408 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.744386    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6e2abaa4-6e03-493a-90ef-833dbab4d41e-installation-pull-secrets\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:20.744503 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.744461    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/47ae46ed-3dac-4491-9f8c-098e66424ebc-stats-auth\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:20.744646 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.744625    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6e2abaa4-6e03-493a-90ef-833dbab4d41e-image-registry-private-configuration\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:20.745217 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.745202    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/43b9ae1f-18aa-4310-bdcd-4d9b296d4e42-serving-cert\") pod \"service-ca-operator-859f94c4b-7dvk2\" (UID: \"43b9ae1f-18aa-4310-bdcd-4d9b296d4e42\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7dvk2"
Sep 10 14:24:20.745265 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.745236    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/47ae46ed-3dac-4491-9f8c-098e66424ebc-default-certificate\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:20.745508 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.745494    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/359d40e9-f428-4cc3-b4e4-991c081d664a-serving-cert\") pod \"console-operator-85b454874b-mg9d6\" (UID: \"359d40e9-f428-4cc3-b4e4-991c081d664a\") " pod="openshift-console-operator/console-operator-85b454874b-mg9d6"
Sep 10 14:24:20.747072 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.747049    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zn8bx"]
Sep 10 14:24:20.749443 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.749428    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zn8bx"
Sep 10 14:24:20.751904 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.751882    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 10 14:24:20.752064 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.751976    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-q8sfx\""
Sep 10 14:24:20.752237 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.752217    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:24:20.752331 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.752216    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 10 14:24:20.754974 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.754956    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrz7r\" (UniqueName: \"kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-kube-api-access-qrz7r\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:20.759672 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.759654    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfrv8\" (UniqueName: \"kubernetes.io/projected/47ae46ed-3dac-4491-9f8c-098e66424ebc-kube-api-access-pfrv8\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:20.760175 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.760147    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjknd\" (UniqueName: \"kubernetes.io/projected/43b9ae1f-18aa-4310-bdcd-4d9b296d4e42-kube-api-access-tjknd\") pod \"service-ca-operator-859f94c4b-7dvk2\" (UID: \"43b9ae1f-18aa-4310-bdcd-4d9b296d4e42\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7dvk2"
Sep 10 14:24:20.760281 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.760266    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcz4f\" (UniqueName: \"kubernetes.io/projected/359d40e9-f428-4cc3-b4e4-991c081d664a-kube-api-access-pcz4f\") pod \"console-operator-85b454874b-mg9d6\" (UID: \"359d40e9-f428-4cc3-b4e4-991c081d664a\") " pod="openshift-console-operator/console-operator-85b454874b-mg9d6"
Sep 10 14:24:20.760414 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.760395    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-bound-sa-token\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:20.765396 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.765335    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zn8bx"]
Sep 10 14:24:20.843569 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.843494    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac0fd5c4-8276-4fb3-865c-c76cf73da9da-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-zn8bx\" (UID: \"ac0fd5c4-8276-4fb3-865c-c76cf73da9da\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zn8bx"
Sep 10 14:24:20.843569 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.843553    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdrqs\" (UniqueName: \"kubernetes.io/projected/ac0fd5c4-8276-4fb3-865c-c76cf73da9da-kube-api-access-mdrqs\") pod \"cluster-samples-operator-7cdcfcf76c-zn8bx\" (UID: \"ac0fd5c4-8276-4fb3-865c-c76cf73da9da\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zn8bx"
Sep 10 14:24:20.902591 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.902556    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-mg9d6"
Sep 10 14:24:20.920882 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.920855    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7dvk2"
Sep 10 14:24:20.944524 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.944476    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdrqs\" (UniqueName: \"kubernetes.io/projected/ac0fd5c4-8276-4fb3-865c-c76cf73da9da-kube-api-access-mdrqs\") pod \"cluster-samples-operator-7cdcfcf76c-zn8bx\" (UID: \"ac0fd5c4-8276-4fb3-865c-c76cf73da9da\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zn8bx"
Sep 10 14:24:20.944733 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.944592    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac0fd5c4-8276-4fb3-865c-c76cf73da9da-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-zn8bx\" (UID: \"ac0fd5c4-8276-4fb3-865c-c76cf73da9da\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zn8bx"
Sep 10 14:24:20.944810 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:20.944729    2547 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 14:24:20.944810 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:20.944806    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac0fd5c4-8276-4fb3-865c-c76cf73da9da-samples-operator-tls podName:ac0fd5c4-8276-4fb3-865c-c76cf73da9da nodeName:}" failed. No retries permitted until 2026-09-10 14:24:21.444787413 +0000 UTC m=+273.486661894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ac0fd5c4-8276-4fb3-865c-c76cf73da9da-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-zn8bx" (UID: "ac0fd5c4-8276-4fb3-865c-c76cf73da9da") : secret "samples-operator-tls" not found
Sep 10 14:24:20.956822 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:20.956791    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdrqs\" (UniqueName: \"kubernetes.io/projected/ac0fd5c4-8276-4fb3-865c-c76cf73da9da-kube-api-access-mdrqs\") pod \"cluster-samples-operator-7cdcfcf76c-zn8bx\" (UID: \"ac0fd5c4-8276-4fb3-865c-c76cf73da9da\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zn8bx"
Sep 10 14:24:21.171510 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:21.171378    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-mg9d6"]
Sep 10 14:24:21.173903 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:24:21.173880    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359d40e9_f428_4cc3_b4e4_991c081d664a.slice/crio-1ed3a73667589265c869c0d81f5c6ad54efee8cb308fc2f53b3239da37e434d4 WatchSource:0}: Error finding container 1ed3a73667589265c869c0d81f5c6ad54efee8cb308fc2f53b3239da37e434d4: Status 404 returned error can't find the container with id 1ed3a73667589265c869c0d81f5c6ad54efee8cb308fc2f53b3239da37e434d4
Sep 10 14:24:21.192584 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:21.192558    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-7dvk2"]
Sep 10 14:24:21.195396 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:24:21.195368    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b9ae1f_18aa_4310_bdcd_4d9b296d4e42.slice/crio-09ede2a26f00524f916ccf32870918accc66f39ed060efb928821d1c24382d71 WatchSource:0}: Error finding container 09ede2a26f00524f916ccf32870918accc66f39ed060efb928821d1c24382d71: Status 404 returned error can't find the container with id 09ede2a26f00524f916ccf32870918accc66f39ed060efb928821d1c24382d71
Sep 10 14:24:21.247139 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:21.247107    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47ae46ed-3dac-4491-9f8c-098e66424ebc-service-ca-bundle\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:21.247308 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:21.247145    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ae46ed-3dac-4491-9f8c-098e66424ebc-metrics-certs\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:21.247308 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:21.247185    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-tls\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:21.247308 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:21.247286    2547 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 14:24:21.247308 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:21.247296    2547 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79489b5cbc-zgr4x: secret "image-registry-tls" not found
Sep 10 14:24:21.247487 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:21.247302    2547 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 14:24:21.247487 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:21.247338    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-tls podName:6e2abaa4-6e03-493a-90ef-833dbab4d41e nodeName:}" failed. No retries permitted until 2026-09-10 14:24:22.247325126 +0000 UTC m=+274.289199585 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-tls") pod "image-registry-79489b5cbc-zgr4x" (UID: "6e2abaa4-6e03-493a-90ef-833dbab4d41e") : secret "image-registry-tls" not found
Sep 10 14:24:21.247487 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:21.247379    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47ae46ed-3dac-4491-9f8c-098e66424ebc-metrics-certs podName:47ae46ed-3dac-4491-9f8c-098e66424ebc nodeName:}" failed. No retries permitted until 2026-09-10 14:24:22.247360242 +0000 UTC m=+274.289234724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47ae46ed-3dac-4491-9f8c-098e66424ebc-metrics-certs") pod "router-default-76f86fcc44-d46qd" (UID: "47ae46ed-3dac-4491-9f8c-098e66424ebc") : secret "router-metrics-certs-default" not found
Sep 10 14:24:21.247487 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:21.247399    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/47ae46ed-3dac-4491-9f8c-098e66424ebc-service-ca-bundle podName:47ae46ed-3dac-4491-9f8c-098e66424ebc nodeName:}" failed. No retries permitted until 2026-09-10 14:24:22.247389699 +0000 UTC m=+274.289264157 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/47ae46ed-3dac-4491-9f8c-098e66424ebc-service-ca-bundle") pod "router-default-76f86fcc44-d46qd" (UID: "47ae46ed-3dac-4491-9f8c-098e66424ebc") : configmap references non-existent config key: service-ca.crt
Sep 10 14:24:21.399209 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:21.399170    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7dvk2" event={"ID":"43b9ae1f-18aa-4310-bdcd-4d9b296d4e42","Type":"ContainerStarted","Data":"09ede2a26f00524f916ccf32870918accc66f39ed060efb928821d1c24382d71"}
Sep 10 14:24:21.400160 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:21.400128    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-mg9d6" event={"ID":"359d40e9-f428-4cc3-b4e4-991c081d664a","Type":"ContainerStarted","Data":"1ed3a73667589265c869c0d81f5c6ad54efee8cb308fc2f53b3239da37e434d4"}
Sep 10 14:24:21.448990 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:21.448927    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac0fd5c4-8276-4fb3-865c-c76cf73da9da-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-zn8bx\" (UID: \"ac0fd5c4-8276-4fb3-865c-c76cf73da9da\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zn8bx"
Sep 10 14:24:21.449079 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:21.449069    2547 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 14:24:21.449131 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:21.449122    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac0fd5c4-8276-4fb3-865c-c76cf73da9da-samples-operator-tls podName:ac0fd5c4-8276-4fb3-865c-c76cf73da9da nodeName:}" failed. No retries permitted until 2026-09-10 14:24:22.449107125 +0000 UTC m=+274.490981584 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ac0fd5c4-8276-4fb3-865c-c76cf73da9da-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-zn8bx" (UID: "ac0fd5c4-8276-4fb3-865c-c76cf73da9da") : secret "samples-operator-tls" not found
Sep 10 14:24:22.254438 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:22.254377    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47ae46ed-3dac-4491-9f8c-098e66424ebc-service-ca-bundle\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:22.254438 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:22.254434    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ae46ed-3dac-4491-9f8c-098e66424ebc-metrics-certs\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:22.254975 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:22.254514    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/47ae46ed-3dac-4491-9f8c-098e66424ebc-service-ca-bundle podName:47ae46ed-3dac-4491-9f8c-098e66424ebc nodeName:}" failed. No retries permitted until 2026-09-10 14:24:24.254495741 +0000 UTC m=+276.296370224 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/47ae46ed-3dac-4491-9f8c-098e66424ebc-service-ca-bundle") pod "router-default-76f86fcc44-d46qd" (UID: "47ae46ed-3dac-4491-9f8c-098e66424ebc") : configmap references non-existent config key: service-ca.crt
Sep 10 14:24:22.254975 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:22.254557    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-tls\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:22.254975 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:22.254649    2547 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 14:24:22.254975 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:22.254691    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47ae46ed-3dac-4491-9f8c-098e66424ebc-metrics-certs podName:47ae46ed-3dac-4491-9f8c-098e66424ebc nodeName:}" failed. No retries permitted until 2026-09-10 14:24:24.254680348 +0000 UTC m=+276.296554820 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47ae46ed-3dac-4491-9f8c-098e66424ebc-metrics-certs") pod "router-default-76f86fcc44-d46qd" (UID: "47ae46ed-3dac-4491-9f8c-098e66424ebc") : secret "router-metrics-certs-default" not found
Sep 10 14:24:22.254975 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:22.254653    2547 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 14:24:22.254975 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:22.254728    2547 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79489b5cbc-zgr4x: secret "image-registry-tls" not found
Sep 10 14:24:22.254975 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:22.254758    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-tls podName:6e2abaa4-6e03-493a-90ef-833dbab4d41e nodeName:}" failed. No retries permitted until 2026-09-10 14:24:24.254748596 +0000 UTC m=+276.296623069 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-tls") pod "image-registry-79489b5cbc-zgr4x" (UID: "6e2abaa4-6e03-493a-90ef-833dbab4d41e") : secret "image-registry-tls" not found
Sep 10 14:24:22.456714 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:22.456669    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac0fd5c4-8276-4fb3-865c-c76cf73da9da-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-zn8bx\" (UID: \"ac0fd5c4-8276-4fb3-865c-c76cf73da9da\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zn8bx"
Sep 10 14:24:22.456868 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:22.456828    2547 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 14:24:22.456938 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:22.456893    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac0fd5c4-8276-4fb3-865c-c76cf73da9da-samples-operator-tls podName:ac0fd5c4-8276-4fb3-865c-c76cf73da9da nodeName:}" failed. No retries permitted until 2026-09-10 14:24:24.456875044 +0000 UTC m=+276.498749526 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ac0fd5c4-8276-4fb3-865c-c76cf73da9da-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-zn8bx" (UID: "ac0fd5c4-8276-4fb3-865c-c76cf73da9da") : secret "samples-operator-tls" not found
Sep 10 14:24:24.272116 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:24.272079    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47ae46ed-3dac-4491-9f8c-098e66424ebc-service-ca-bundle\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:24.272116 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:24.272118    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ae46ed-3dac-4491-9f8c-098e66424ebc-metrics-certs\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:24.272642 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:24.272273    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/47ae46ed-3dac-4491-9f8c-098e66424ebc-service-ca-bundle podName:47ae46ed-3dac-4491-9f8c-098e66424ebc nodeName:}" failed. No retries permitted until 2026-09-10 14:24:28.272256045 +0000 UTC m=+280.314130508 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/47ae46ed-3dac-4491-9f8c-098e66424ebc-service-ca-bundle") pod "router-default-76f86fcc44-d46qd" (UID: "47ae46ed-3dac-4491-9f8c-098e66424ebc") : configmap references non-existent config key: service-ca.crt
Sep 10 14:24:24.272642 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:24.272297    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-tls\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:24.272642 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:24.272360    2547 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 14:24:24.272642 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:24.272397    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47ae46ed-3dac-4491-9f8c-098e66424ebc-metrics-certs podName:47ae46ed-3dac-4491-9f8c-098e66424ebc nodeName:}" failed. No retries permitted until 2026-09-10 14:24:28.272386735 +0000 UTC m=+280.314261210 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47ae46ed-3dac-4491-9f8c-098e66424ebc-metrics-certs") pod "router-default-76f86fcc44-d46qd" (UID: "47ae46ed-3dac-4491-9f8c-098e66424ebc") : secret "router-metrics-certs-default" not found
Sep 10 14:24:24.272642 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:24.272410    2547 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 14:24:24.272642 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:24.272420    2547 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79489b5cbc-zgr4x: secret "image-registry-tls" not found
Sep 10 14:24:24.272642 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:24.272452    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-tls podName:6e2abaa4-6e03-493a-90ef-833dbab4d41e nodeName:}" failed. No retries permitted until 2026-09-10 14:24:28.272442369 +0000 UTC m=+280.314316844 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-tls") pod "image-registry-79489b5cbc-zgr4x" (UID: "6e2abaa4-6e03-493a-90ef-833dbab4d41e") : secret "image-registry-tls" not found
Sep 10 14:24:24.410031 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:24.409990    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7dvk2" event={"ID":"43b9ae1f-18aa-4310-bdcd-4d9b296d4e42","Type":"ContainerStarted","Data":"52e693fb872c10560bb27614a85a04291d1c212794a38063193ced41ebcf867b"}
Sep 10 14:24:24.411601 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:24.411575    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mg9d6_359d40e9-f428-4cc3-b4e4-991c081d664a/console-operator/0.log"
Sep 10 14:24:24.411748 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:24.411619    2547 generic.go:356] "Generic (PLEG): container finished" podID="359d40e9-f428-4cc3-b4e4-991c081d664a" containerID="f37b6a5285798c299773fdbd2a22fb9e74822947b137353a8e13d9561cac0777" exitCode=255
Sep 10 14:24:24.411748 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:24.411651    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-mg9d6" event={"ID":"359d40e9-f428-4cc3-b4e4-991c081d664a","Type":"ContainerDied","Data":"f37b6a5285798c299773fdbd2a22fb9e74822947b137353a8e13d9561cac0777"}
Sep 10 14:24:24.411921 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:24.411905    2547 scope.go:117] "RemoveContainer" containerID="f37b6a5285798c299773fdbd2a22fb9e74822947b137353a8e13d9561cac0777"
Sep 10 14:24:24.428980 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:24.428933    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7dvk2" podStartSLOduration=1.979484826 podStartE2EDuration="4.428914529s" totalImagesPullingTime="2.449429703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:24:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:24:24.428444543 +0000 UTC m=+276.470319024" watchObservedRunningTime="2026-09-10 14:24:24.428914529 +0000 UTC m=+276.470789013"
Sep 10 14:24:24.473777 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:24.473752    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac0fd5c4-8276-4fb3-865c-c76cf73da9da-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-zn8bx\" (UID: \"ac0fd5c4-8276-4fb3-865c-c76cf73da9da\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zn8bx"
Sep 10 14:24:24.474211 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:24.474193    2547 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 14:24:24.474371 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:24.474354    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac0fd5c4-8276-4fb3-865c-c76cf73da9da-samples-operator-tls podName:ac0fd5c4-8276-4fb3-865c-c76cf73da9da nodeName:}" failed. No retries permitted until 2026-09-10 14:24:28.474333765 +0000 UTC m=+280.516208241 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ac0fd5c4-8276-4fb3-865c-c76cf73da9da-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-zn8bx" (UID: "ac0fd5c4-8276-4fb3-865c-c76cf73da9da") : secret "samples-operator-tls" not found
Sep 10 14:24:25.191920 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.191887    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gxwmg"]
Sep 10 14:24:25.193821 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.193794    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gxwmg"
Sep 10 14:24:25.196403 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.196384    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 10 14:24:25.196504 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.196488    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dsnt6\""
Sep 10 14:24:25.197566 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.197545    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 10 14:24:25.205568 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.205534    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gxwmg"]
Sep 10 14:24:25.279712 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.279660    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsgjw\" (UniqueName: \"kubernetes.io/projected/6e2e1ff5-7d0f-4de4-8b6c-83d8942f9ea6-kube-api-access-qsgjw\") pod \"migrator-fdbb8fdff-gxwmg\" (UID: \"6e2e1ff5-7d0f-4de4-8b6c-83d8942f9ea6\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gxwmg"
Sep 10 14:24:25.380098 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.380064    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsgjw\" (UniqueName: \"kubernetes.io/projected/6e2e1ff5-7d0f-4de4-8b6c-83d8942f9ea6-kube-api-access-qsgjw\") pod \"migrator-fdbb8fdff-gxwmg\" (UID: \"6e2e1ff5-7d0f-4de4-8b6c-83d8942f9ea6\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gxwmg"
Sep 10 14:24:25.388195 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.388163    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsgjw\" (UniqueName: \"kubernetes.io/projected/6e2e1ff5-7d0f-4de4-8b6c-83d8942f9ea6-kube-api-access-qsgjw\") pod \"migrator-fdbb8fdff-gxwmg\" (UID: \"6e2e1ff5-7d0f-4de4-8b6c-83d8942f9ea6\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gxwmg"
Sep 10 14:24:25.421968 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.421942    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mg9d6_359d40e9-f428-4cc3-b4e4-991c081d664a/console-operator/1.log"
Sep 10 14:24:25.422323 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.422309    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mg9d6_359d40e9-f428-4cc3-b4e4-991c081d664a/console-operator/0.log"
Sep 10 14:24:25.422406 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.422340    2547 generic.go:356] "Generic (PLEG): container finished" podID="359d40e9-f428-4cc3-b4e4-991c081d664a" containerID="50f38b2ee5c7bf1c1d4572cbaf7321d1bbb0a1ad71574ede8e3ae7a8b0a2a148" exitCode=255
Sep 10 14:24:25.422465 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.422429    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-mg9d6" event={"ID":"359d40e9-f428-4cc3-b4e4-991c081d664a","Type":"ContainerDied","Data":"50f38b2ee5c7bf1c1d4572cbaf7321d1bbb0a1ad71574ede8e3ae7a8b0a2a148"}
Sep 10 14:24:25.422523 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.422514    2547 scope.go:117] "RemoveContainer" containerID="f37b6a5285798c299773fdbd2a22fb9e74822947b137353a8e13d9561cac0777"
Sep 10 14:24:25.422776 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.422753    2547 scope.go:117] "RemoveContainer" containerID="50f38b2ee5c7bf1c1d4572cbaf7321d1bbb0a1ad71574ede8e3ae7a8b0a2a148"
Sep 10 14:24:25.422969 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:25.422947    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-mg9d6_openshift-console-operator(359d40e9-f428-4cc3-b4e4-991c081d664a)\"" pod="openshift-console-operator/console-operator-85b454874b-mg9d6" podUID="359d40e9-f428-4cc3-b4e4-991c081d664a"
Sep 10 14:24:25.502509 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.502434    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gxwmg"
Sep 10 14:24:25.694546 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.694510    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-c2xkx"]
Sep 10 14:24:25.696708 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.696679    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:25.699356 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.699334    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 14:24:25.699512 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.699497    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mxclh\""
Sep 10 14:24:25.699809 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.699792    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 14:24:25.699893 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.699875    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 14:24:25.699947 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.699928    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 14:24:25.752559 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.752486    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c2xkx"]
Sep 10 14:24:25.769229 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.769199    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gxwmg"]
Sep 10 14:24:25.772186 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:24:25.772162    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e2e1ff5_7d0f_4de4_8b6c_83d8942f9ea6.slice/crio-c305ff1e0b52d702877d17c4e898a1b68b36217e6bddd78b94632eed9e3e7dec WatchSource:0}: Error finding container c305ff1e0b52d702877d17c4e898a1b68b36217e6bddd78b94632eed9e3e7dec: Status 404 returned error can't find the container with id c305ff1e0b52d702877d17c4e898a1b68b36217e6bddd78b94632eed9e3e7dec
Sep 10 14:24:25.783986 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.783962    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/233305e5-df37-4854-bddb-a383fe612b09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:25.784074 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.783996    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/233305e5-df37-4854-bddb-a383fe612b09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:25.784122 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.784093    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgflb\" (UniqueName: \"kubernetes.io/projected/233305e5-df37-4854-bddb-a383fe612b09-kube-api-access-fgflb\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:25.784157 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.784122    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/233305e5-df37-4854-bddb-a383fe612b09-crio-socket\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:25.784157 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.784146    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/233305e5-df37-4854-bddb-a383fe612b09-data-volume\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:25.885320 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.885277    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/233305e5-df37-4854-bddb-a383fe612b09-crio-socket\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:25.885320 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.885327    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/233305e5-df37-4854-bddb-a383fe612b09-data-volume\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:25.885570 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.885379    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/233305e5-df37-4854-bddb-a383fe612b09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:25.885570 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.885401    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/233305e5-df37-4854-bddb-a383fe612b09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:25.885570 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.885413    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/233305e5-df37-4854-bddb-a383fe612b09-crio-socket\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:25.885570 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:25.885491    2547 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 14:24:25.885570 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:25.885506    2547 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:24:25.885570 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.885540    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgflb\" (UniqueName: \"kubernetes.io/projected/233305e5-df37-4854-bddb-a383fe612b09-kube-api-access-fgflb\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:25.885570 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:25.885549    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/233305e5-df37-4854-bddb-a383fe612b09-kube-rbac-proxy-cm podName:233305e5-df37-4854-bddb-a383fe612b09 nodeName:}" failed. No retries permitted until 2026-09-10 14:24:26.385533498 +0000 UTC m=+278.427407983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/233305e5-df37-4854-bddb-a383fe612b09-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2xkx" (UID: "233305e5-df37-4854-bddb-a383fe612b09") : configmap "kube-rbac-proxy" not found
Sep 10 14:24:25.885849 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:25.885625    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/233305e5-df37-4854-bddb-a383fe612b09-insights-runtime-extractor-tls podName:233305e5-df37-4854-bddb-a383fe612b09 nodeName:}" failed. No retries permitted until 2026-09-10 14:24:26.385605918 +0000 UTC m=+278.427480382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/233305e5-df37-4854-bddb-a383fe612b09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2xkx" (UID: "233305e5-df37-4854-bddb-a383fe612b09") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:24:25.885849 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.885767    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/233305e5-df37-4854-bddb-a383fe612b09-data-volume\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:25.893530 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:25.893509    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgflb\" (UniqueName: \"kubernetes.io/projected/233305e5-df37-4854-bddb-a383fe612b09-kube-api-access-fgflb\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:26.390095 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:26.390056    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/233305e5-df37-4854-bddb-a383fe612b09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:26.390095 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:26.390095    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/233305e5-df37-4854-bddb-a383fe612b09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:26.390594 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:26.390191    2547 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 14:24:26.390594 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:26.390255    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/233305e5-df37-4854-bddb-a383fe612b09-kube-rbac-proxy-cm podName:233305e5-df37-4854-bddb-a383fe612b09 nodeName:}" failed. No retries permitted until 2026-09-10 14:24:27.390239695 +0000 UTC m=+279.432114179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/233305e5-df37-4854-bddb-a383fe612b09-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2xkx" (UID: "233305e5-df37-4854-bddb-a383fe612b09") : configmap "kube-rbac-proxy" not found
Sep 10 14:24:26.390594 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:26.390318    2547 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:24:26.390594 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:26.390365    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/233305e5-df37-4854-bddb-a383fe612b09-insights-runtime-extractor-tls podName:233305e5-df37-4854-bddb-a383fe612b09 nodeName:}" failed. No retries permitted until 2026-09-10 14:24:27.390353342 +0000 UTC m=+279.432227820 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/233305e5-df37-4854-bddb-a383fe612b09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2xkx" (UID: "233305e5-df37-4854-bddb-a383fe612b09") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:24:26.429337 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:26.429304    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gxwmg" event={"ID":"6e2e1ff5-7d0f-4de4-8b6c-83d8942f9ea6","Type":"ContainerStarted","Data":"c305ff1e0b52d702877d17c4e898a1b68b36217e6bddd78b94632eed9e3e7dec"}
Sep 10 14:24:26.430761 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:26.430741    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mg9d6_359d40e9-f428-4cc3-b4e4-991c081d664a/console-operator/1.log"
Sep 10 14:24:26.431119 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:26.431094    2547 scope.go:117] "RemoveContainer" containerID="50f38b2ee5c7bf1c1d4572cbaf7321d1bbb0a1ad71574ede8e3ae7a8b0a2a148"
Sep 10 14:24:26.431276 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:26.431258    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-mg9d6_openshift-console-operator(359d40e9-f428-4cc3-b4e4-991c081d664a)\"" pod="openshift-console-operator/console-operator-85b454874b-mg9d6" podUID="359d40e9-f428-4cc3-b4e4-991c081d664a"
Sep 10 14:24:27.398040 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:27.397949    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/233305e5-df37-4854-bddb-a383fe612b09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:27.398040 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:27.397989    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/233305e5-df37-4854-bddb-a383fe612b09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:27.398413 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:27.398073    2547 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 14:24:27.398413 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:27.398128    2547 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:24:27.398413 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:27.398143    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/233305e5-df37-4854-bddb-a383fe612b09-kube-rbac-proxy-cm podName:233305e5-df37-4854-bddb-a383fe612b09 nodeName:}" failed. No retries permitted until 2026-09-10 14:24:29.39812275 +0000 UTC m=+281.439997209 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/233305e5-df37-4854-bddb-a383fe612b09-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2xkx" (UID: "233305e5-df37-4854-bddb-a383fe612b09") : configmap "kube-rbac-proxy" not found
Sep 10 14:24:27.398413 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:27.398168    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/233305e5-df37-4854-bddb-a383fe612b09-insights-runtime-extractor-tls podName:233305e5-df37-4854-bddb-a383fe612b09 nodeName:}" failed. No retries permitted until 2026-09-10 14:24:29.39815695 +0000 UTC m=+281.440031409 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/233305e5-df37-4854-bddb-a383fe612b09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2xkx" (UID: "233305e5-df37-4854-bddb-a383fe612b09") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:24:27.435978 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:27.435946    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gxwmg" event={"ID":"6e2e1ff5-7d0f-4de4-8b6c-83d8942f9ea6","Type":"ContainerStarted","Data":"bbe04c07910085eaac65f5a961cc238a699f80ad25a6e5bb08bad86ad4f30d22"}
Sep 10 14:24:27.436127 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:27.435995    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gxwmg" event={"ID":"6e2e1ff5-7d0f-4de4-8b6c-83d8942f9ea6","Type":"ContainerStarted","Data":"38cb17b7f8462600f09a242b6271648e92fadef7ec24821e3698ecbd4bb19e62"}
Sep 10 14:24:27.460574 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:27.460528    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gxwmg" podStartSLOduration=1.127711944 podStartE2EDuration="2.46051454s" totalImagesPullingTime="1.332802596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:24:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:24:27.456455379 +0000 UTC m=+279.498329860" watchObservedRunningTime="2026-09-10 14:24:27.46051454 +0000 UTC m=+279.502389021"
Sep 10 14:24:28.305683 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:28.305646    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47ae46ed-3dac-4491-9f8c-098e66424ebc-service-ca-bundle\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:28.305884 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:28.305714    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ae46ed-3dac-4491-9f8c-098e66424ebc-metrics-certs\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:28.305884 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:28.305757    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-tls\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:28.305884 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:28.305805    2547 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 14:24:28.305884 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:28.305853    2547 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 14:24:28.305884 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:28.305863    2547 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79489b5cbc-zgr4x: secret "image-registry-tls" not found
Sep 10 14:24:28.305884 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:28.305870    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47ae46ed-3dac-4491-9f8c-098e66424ebc-metrics-certs podName:47ae46ed-3dac-4491-9f8c-098e66424ebc nodeName:}" failed. No retries permitted until 2026-09-10 14:24:36.30585346 +0000 UTC m=+288.347727936 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47ae46ed-3dac-4491-9f8c-098e66424ebc-metrics-certs") pod "router-default-76f86fcc44-d46qd" (UID: "47ae46ed-3dac-4491-9f8c-098e66424ebc") : secret "router-metrics-certs-default" not found
Sep 10 14:24:28.305884 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:28.305886    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/47ae46ed-3dac-4491-9f8c-098e66424ebc-service-ca-bundle podName:47ae46ed-3dac-4491-9f8c-098e66424ebc nodeName:}" failed. No retries permitted until 2026-09-10 14:24:36.305879783 +0000 UTC m=+288.347754281 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/47ae46ed-3dac-4491-9f8c-098e66424ebc-service-ca-bundle") pod "router-default-76f86fcc44-d46qd" (UID: "47ae46ed-3dac-4491-9f8c-098e66424ebc") : configmap references non-existent config key: service-ca.crt
Sep 10 14:24:28.306218 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:28.305898    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-tls podName:6e2abaa4-6e03-493a-90ef-833dbab4d41e nodeName:}" failed. No retries permitted until 2026-09-10 14:24:36.30589309 +0000 UTC m=+288.347767548 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-tls") pod "image-registry-79489b5cbc-zgr4x" (UID: "6e2abaa4-6e03-493a-90ef-833dbab4d41e") : secret "image-registry-tls" not found
Sep 10 14:24:28.507857 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:28.507809    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac0fd5c4-8276-4fb3-865c-c76cf73da9da-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-zn8bx\" (UID: \"ac0fd5c4-8276-4fb3-865c-c76cf73da9da\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zn8bx"
Sep 10 14:24:28.508345 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:28.508010    2547 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 14:24:28.508345 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:28.508086    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac0fd5c4-8276-4fb3-865c-c76cf73da9da-samples-operator-tls podName:ac0fd5c4-8276-4fb3-865c-c76cf73da9da nodeName:}" failed. No retries permitted until 2026-09-10 14:24:36.508066096 +0000 UTC m=+288.549940572 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ac0fd5c4-8276-4fb3-865c-c76cf73da9da-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-zn8bx" (UID: "ac0fd5c4-8276-4fb3-865c-c76cf73da9da") : secret "samples-operator-tls" not found
Sep 10 14:24:28.638515 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:28.638439    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vz9vp_5a663e3c-6895-4085-aa41-1efd5d6ab208/dns-node-resolver/0.log"
Sep 10 14:24:29.225921 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:29.225883    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-46jxd_fe41a3f3-0ada-4163-9f76-74d8d30809ed/node-ca/0.log"
Sep 10 14:24:29.415413 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:29.415378    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/233305e5-df37-4854-bddb-a383fe612b09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:29.415413 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:29.415414    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/233305e5-df37-4854-bddb-a383fe612b09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:29.415609 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:29.415506    2547 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 14:24:29.415609 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:29.415571    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/233305e5-df37-4854-bddb-a383fe612b09-kube-rbac-proxy-cm podName:233305e5-df37-4854-bddb-a383fe612b09 nodeName:}" failed. No retries permitted until 2026-09-10 14:24:33.415555838 +0000 UTC m=+285.457430310 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/233305e5-df37-4854-bddb-a383fe612b09-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2xkx" (UID: "233305e5-df37-4854-bddb-a383fe612b09") : configmap "kube-rbac-proxy" not found
Sep 10 14:24:29.415682 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:29.415616    2547 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:24:29.415682 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:29.415676    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/233305e5-df37-4854-bddb-a383fe612b09-insights-runtime-extractor-tls podName:233305e5-df37-4854-bddb-a383fe612b09 nodeName:}" failed. No retries permitted until 2026-09-10 14:24:33.415662055 +0000 UTC m=+285.457536539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/233305e5-df37-4854-bddb-a383fe612b09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2xkx" (UID: "233305e5-df37-4854-bddb-a383fe612b09") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:24:30.026081 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:30.026044    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2c9kh_d6415697-570a-4f1f-9317-7e774d60cc04/serve-healthcheck-canary/0.log"
Sep 10 14:24:30.127498 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:30.127459    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-957xl" podUID="54e8b08a-0bfa-4e3f-87e3-d6a5e886c798"
Sep 10 14:24:30.444590 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:30.444559    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-957xl"
Sep 10 14:24:30.626288 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:30.626254    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-gxwmg_6e2e1ff5-7d0f-4de4-8b6c-83d8942f9ea6/migrator/0.log"
Sep 10 14:24:30.826074 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:30.826000    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-gxwmg_6e2e1ff5-7d0f-4de4-8b6c-83d8942f9ea6/graceful-termination/0.log"
Sep 10 14:24:30.903267 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:30.903238    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-mg9d6"
Sep 10 14:24:30.903389 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:30.903291    2547 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-mg9d6"
Sep 10 14:24:30.903667 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:30.903650    2547 scope.go:117] "RemoveContainer" containerID="50f38b2ee5c7bf1c1d4572cbaf7321d1bbb0a1ad71574ede8e3ae7a8b0a2a148"
Sep 10 14:24:30.903840 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:30.903821    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-mg9d6_openshift-console-operator(359d40e9-f428-4cc3-b4e4-991c081d664a)\"" pod="openshift-console-operator/console-operator-85b454874b-mg9d6" podUID="359d40e9-f428-4cc3-b4e4-991c081d664a"
Sep 10 14:24:33.447672 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:33.447632    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/233305e5-df37-4854-bddb-a383fe612b09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:33.448125 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:33.447678    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/233305e5-df37-4854-bddb-a383fe612b09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:33.448125 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:33.447763    2547 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 14:24:33.448125 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:33.447833    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/233305e5-df37-4854-bddb-a383fe612b09-kube-rbac-proxy-cm podName:233305e5-df37-4854-bddb-a383fe612b09 nodeName:}" failed. No retries permitted until 2026-09-10 14:24:41.447817046 +0000 UTC m=+293.489691525 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/233305e5-df37-4854-bddb-a383fe612b09-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2xkx" (UID: "233305e5-df37-4854-bddb-a383fe612b09") : configmap "kube-rbac-proxy" not found
Sep 10 14:24:33.448125 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:33.447838    2547 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:24:33.448125 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:33.447890    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/233305e5-df37-4854-bddb-a383fe612b09-insights-runtime-extractor-tls podName:233305e5-df37-4854-bddb-a383fe612b09 nodeName:}" failed. No retries permitted until 2026-09-10 14:24:41.447875886 +0000 UTC m=+293.489750375 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/233305e5-df37-4854-bddb-a383fe612b09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2xkx" (UID: "233305e5-df37-4854-bddb-a383fe612b09") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:24:33.750312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:33.750214    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls\") pod \"dns-default-957xl\" (UID: \"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798\") " pod="openshift-dns/dns-default-957xl"
Sep 10 14:24:33.750462 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:33.750374    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:24:33.750462 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:33.750441    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls podName:54e8b08a-0bfa-4e3f-87e3-d6a5e886c798 nodeName:}" failed. No retries permitted until 2026-09-10 14:26:35.7504242 +0000 UTC m=+407.792298675 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls") pod "dns-default-957xl" (UID: "54e8b08a-0bfa-4e3f-87e3-d6a5e886c798") : secret "dns-default-metrics-tls" not found
Sep 10 14:24:36.375250 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:36.375217    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47ae46ed-3dac-4491-9f8c-098e66424ebc-service-ca-bundle\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:36.375603 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:36.375269    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ae46ed-3dac-4491-9f8c-098e66424ebc-metrics-certs\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:36.375603 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:36.375319    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-tls\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:36.375603 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:36.375420    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/47ae46ed-3dac-4491-9f8c-098e66424ebc-service-ca-bundle podName:47ae46ed-3dac-4491-9f8c-098e66424ebc nodeName:}" failed. No retries permitted until 2026-09-10 14:24:52.375402426 +0000 UTC m=+304.417276902 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/47ae46ed-3dac-4491-9f8c-098e66424ebc-service-ca-bundle") pod "router-default-76f86fcc44-d46qd" (UID: "47ae46ed-3dac-4491-9f8c-098e66424ebc") : configmap references non-existent config key: service-ca.crt
Sep 10 14:24:36.377795 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:36.377772    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ae46ed-3dac-4491-9f8c-098e66424ebc-metrics-certs\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:36.377894 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:36.377827    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-tls\") pod \"image-registry-79489b5cbc-zgr4x\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") " pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:36.515928 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:36.515887    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:36.577584 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:36.577530    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac0fd5c4-8276-4fb3-865c-c76cf73da9da-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-zn8bx\" (UID: \"ac0fd5c4-8276-4fb3-865c-c76cf73da9da\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zn8bx"
Sep 10 14:24:36.580163 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:36.580133    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac0fd5c4-8276-4fb3-865c-c76cf73da9da-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-zn8bx\" (UID: \"ac0fd5c4-8276-4fb3-865c-c76cf73da9da\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zn8bx"
Sep 10 14:24:36.677155 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:36.677113    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zn8bx"
Sep 10 14:24:36.787483 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:36.787284    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79489b5cbc-zgr4x"]
Sep 10 14:24:36.789995 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:24:36.789965    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e2abaa4_6e03_493a_90ef_833dbab4d41e.slice/crio-762b9bb5b45403f8bfbd46daf6a4d0ce16944240dd9e77a0a77da18519a07ff3 WatchSource:0}: Error finding container 762b9bb5b45403f8bfbd46daf6a4d0ce16944240dd9e77a0a77da18519a07ff3: Status 404 returned error can't find the container with id 762b9bb5b45403f8bfbd46daf6a4d0ce16944240dd9e77a0a77da18519a07ff3
Sep 10 14:24:36.943868 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:36.943837    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zn8bx"]
Sep 10 14:24:37.466805 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:37.466768    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x" event={"ID":"6e2abaa4-6e03-493a-90ef-833dbab4d41e","Type":"ContainerStarted","Data":"0d46e6a7d3934d9ac069ba1db1fb045d53e22364604d5ba3b759fa381d912b71"}
Sep 10 14:24:37.467263 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:37.466828    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x" event={"ID":"6e2abaa4-6e03-493a-90ef-833dbab4d41e","Type":"ContainerStarted","Data":"762b9bb5b45403f8bfbd46daf6a4d0ce16944240dd9e77a0a77da18519a07ff3"}
Sep 10 14:24:37.467263 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:37.466957    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:24:37.467869 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:37.467852    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zn8bx" event={"ID":"ac0fd5c4-8276-4fb3-865c-c76cf73da9da","Type":"ContainerStarted","Data":"77d12ae2c5e61082d82bbe145cc5158f89c40fc58fd470b66d4ac83a1279aa7c"}
Sep 10 14:24:37.489284 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:37.489238    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x" podStartSLOduration=17.48922451 podStartE2EDuration="17.48922451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:24:37.488110845 +0000 UTC m=+289.529985327" watchObservedRunningTime="2026-09-10 14:24:37.48922451 +0000 UTC m=+289.531099043"
Sep 10 14:24:39.475631 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:39.475585    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zn8bx" event={"ID":"ac0fd5c4-8276-4fb3-865c-c76cf73da9da","Type":"ContainerStarted","Data":"05a56520f03759ec94d92f797c92af450e013cdd6dd77503dd0d18cbd89d838a"}
Sep 10 14:24:39.475631 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:39.475636    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zn8bx" event={"ID":"ac0fd5c4-8276-4fb3-865c-c76cf73da9da","Type":"ContainerStarted","Data":"2033ddd5d3c52b752e70985a539b3e72bfb4a6422dd09d272647039dea7584c3"}
Sep 10 14:24:39.491317 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:39.491256    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zn8bx" podStartSLOduration=17.748122593 podStartE2EDuration="19.491238752s" totalImagesPullingTime="1.743116159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:24:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:24:39.491007261 +0000 UTC m=+291.532881742" watchObservedRunningTime="2026-09-10 14:24:39.491238752 +0000 UTC m=+291.533113233"
Sep 10 14:24:41.417410 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:41.417364    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-756dn" podUID="c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0"
Sep 10 14:24:41.516162 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:41.516127    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/233305e5-df37-4854-bddb-a383fe612b09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:41.516322 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:41.516169    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/233305e5-df37-4854-bddb-a383fe612b09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:41.516322 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:41.516308    2547 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:24:41.516399 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:24:41.516374    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/233305e5-df37-4854-bddb-a383fe612b09-insights-runtime-extractor-tls podName:233305e5-df37-4854-bddb-a383fe612b09 nodeName:}" failed. No retries permitted until 2026-09-10 14:24:57.516356163 +0000 UTC m=+309.558230636 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/233305e5-df37-4854-bddb-a383fe612b09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2xkx" (UID: "233305e5-df37-4854-bddb-a383fe612b09") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:24:41.516817 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:41.516793    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/233305e5-df37-4854-bddb-a383fe612b09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:42.415541 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:42.415509    2547 scope.go:117] "RemoveContainer" containerID="50f38b2ee5c7bf1c1d4572cbaf7321d1bbb0a1ad71574ede8e3ae7a8b0a2a148"
Sep 10 14:24:43.489320 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:43.489290    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mg9d6_359d40e9-f428-4cc3-b4e4-991c081d664a/console-operator/1.log"
Sep 10 14:24:43.489692 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:43.489356    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-mg9d6" event={"ID":"359d40e9-f428-4cc3-b4e4-991c081d664a","Type":"ContainerStarted","Data":"d383f70433444ff29b98b27f93494e3a351cd6f16eb86462dd93e99913cf5022"}
Sep 10 14:24:43.489692 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:43.489606    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-mg9d6"
Sep 10 14:24:43.494212 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:43.494187    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-mg9d6"
Sep 10 14:24:43.524581 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:43.524530    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-mg9d6" podStartSLOduration=21.053439448 podStartE2EDuration="23.524513771s" totalImagesPullingTime="2.471074323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:24:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:24:43.523722657 +0000 UTC m=+295.565597129" watchObservedRunningTime="2026-09-10 14:24:43.524513771 +0000 UTC m=+295.566388253"
Sep 10 14:24:48.292973 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:48.292941    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mg9d6_359d40e9-f428-4cc3-b4e4-991c081d664a/console-operator/1.log"
Sep 10 14:24:48.293438 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:48.293086    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mg9d6_359d40e9-f428-4cc3-b4e4-991c081d664a/console-operator/1.log"
Sep 10 14:24:48.303646 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:48.303622    2547 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 14:24:50.142979 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:50.142944    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-dsgb8"]
Sep 10 14:24:50.146104 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:50.146077    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-dsgb8"
Sep 10 14:24:50.150192 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:50.150168    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 14:24:50.150300 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:50.150231    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-9zgq8\""
Sep 10 14:24:50.150504 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:50.150490    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 14:24:50.157414 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:50.157395    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79489b5cbc-zgr4x"]
Sep 10 14:24:50.158450 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:50.158422    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-dsgb8"]
Sep 10 14:24:50.293073 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:50.293036    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbgwf\" (UniqueName: \"kubernetes.io/projected/af9ef9d2-bc7a-4d77-b0ac-ed3531249fcc-kube-api-access-zbgwf\") pod \"downloads-85d98c998f-dsgb8\" (UID: \"af9ef9d2-bc7a-4d77-b0ac-ed3531249fcc\") " pod="openshift-console/downloads-85d98c998f-dsgb8"
Sep 10 14:24:50.394047 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:50.393969    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbgwf\" (UniqueName: \"kubernetes.io/projected/af9ef9d2-bc7a-4d77-b0ac-ed3531249fcc-kube-api-access-zbgwf\") pod \"downloads-85d98c998f-dsgb8\" (UID: \"af9ef9d2-bc7a-4d77-b0ac-ed3531249fcc\") " pod="openshift-console/downloads-85d98c998f-dsgb8"
Sep 10 14:24:50.401578 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:50.401550    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbgwf\" (UniqueName: \"kubernetes.io/projected/af9ef9d2-bc7a-4d77-b0ac-ed3531249fcc-kube-api-access-zbgwf\") pod \"downloads-85d98c998f-dsgb8\" (UID: \"af9ef9d2-bc7a-4d77-b0ac-ed3531249fcc\") " pod="openshift-console/downloads-85d98c998f-dsgb8"
Sep 10 14:24:50.454636 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:50.454604    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-dsgb8"
Sep 10 14:24:50.725780 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:50.725748    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-dsgb8"]
Sep 10 14:24:50.729034 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:24:50.728998    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9ef9d2_bc7a_4d77_b0ac_ed3531249fcc.slice/crio-70ec387dc81f4a2d15cb8a775ca3eb4549581bfb72c21cea8ed4b3b0c591620a WatchSource:0}: Error finding container 70ec387dc81f4a2d15cb8a775ca3eb4549581bfb72c21cea8ed4b3b0c591620a: Status 404 returned error can't find the container with id 70ec387dc81f4a2d15cb8a775ca3eb4549581bfb72c21cea8ed4b3b0c591620a
Sep 10 14:24:50.730983 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:50.730967    2547 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:24:51.516448 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:51.516406    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-dsgb8" event={"ID":"af9ef9d2-bc7a-4d77-b0ac-ed3531249fcc","Type":"ContainerStarted","Data":"70ec387dc81f4a2d15cb8a775ca3eb4549581bfb72c21cea8ed4b3b0c591620a"}
Sep 10 14:24:52.412134 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:52.412102    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47ae46ed-3dac-4491-9f8c-098e66424ebc-service-ca-bundle\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:52.412754 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:52.412725    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47ae46ed-3dac-4491-9f8c-098e66424ebc-service-ca-bundle\") pod \"router-default-76f86fcc44-d46qd\" (UID: \"47ae46ed-3dac-4491-9f8c-098e66424ebc\") " pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:52.713261 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:52.713187    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-c55ds\""
Sep 10 14:24:52.721515 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:52.721484    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:53.000486 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:53.000435    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76f86fcc44-d46qd"]
Sep 10 14:24:53.003087 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:24:53.003058    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ae46ed_3dac_4491_9f8c_098e66424ebc.slice/crio-b51bac122a3d4b6d13bcdcef5b3811d51a99658d07efa081a43aa9aa2b907142 WatchSource:0}: Error finding container b51bac122a3d4b6d13bcdcef5b3811d51a99658d07efa081a43aa9aa2b907142: Status 404 returned error can't find the container with id b51bac122a3d4b6d13bcdcef5b3811d51a99658d07efa081a43aa9aa2b907142
Sep 10 14:24:53.526139 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:53.526104    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76f86fcc44-d46qd" event={"ID":"47ae46ed-3dac-4491-9f8c-098e66424ebc","Type":"ContainerStarted","Data":"20e7ee3e61e003189be79ebbac908821e2b21cec9afedc9796e0c60a378eeddd"}
Sep 10 14:24:53.526302 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:53.526161    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76f86fcc44-d46qd" event={"ID":"47ae46ed-3dac-4491-9f8c-098e66424ebc","Type":"ContainerStarted","Data":"b51bac122a3d4b6d13bcdcef5b3811d51a99658d07efa081a43aa9aa2b907142"}
Sep 10 14:24:53.545228 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:53.545178    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-76f86fcc44-d46qd" podStartSLOduration=33.545163511 podStartE2EDuration="33.545163511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:24:53.544171903 +0000 UTC m=+305.586046384" watchObservedRunningTime="2026-09-10 14:24:53.545163511 +0000 UTC m=+305.587037992"
Sep 10 14:24:53.722472 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:53.722435    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:53.725233 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:53.725202    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:54.529748 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:54.529713    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:54.530999 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:54.530976    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-76f86fcc44-d46qd"
Sep 10 14:24:56.415433 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:56.415395    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:24:57.557227 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:57.557190    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/233305e5-df37-4854-bddb-a383fe612b09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:57.559775 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:57.559744    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/233305e5-df37-4854-bddb-a383fe612b09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2xkx\" (UID: \"233305e5-df37-4854-bddb-a383fe612b09\") " pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:57.808972 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:57.808890    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mxclh\""
Sep 10 14:24:57.816971 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:57.816946    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c2xkx"
Sep 10 14:24:58.086064 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:58.085977    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c2xkx"]
Sep 10 14:24:58.089668 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:24:58.089642    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233305e5_df37_4854_bddb_a383fe612b09.slice/crio-61b0662ed6e08968606b28ec1a694cf22e63eaa32074ecbf3acef3f8f4f8a847 WatchSource:0}: Error finding container 61b0662ed6e08968606b28ec1a694cf22e63eaa32074ecbf3acef3f8f4f8a847: Status 404 returned error can't find the container with id 61b0662ed6e08968606b28ec1a694cf22e63eaa32074ecbf3acef3f8f4f8a847
Sep 10 14:24:58.547827 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:58.547786    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2xkx" event={"ID":"233305e5-df37-4854-bddb-a383fe612b09","Type":"ContainerStarted","Data":"5de9ecb36f44558e05c275fcdbda66563fa9ac90b199ececfcab9269fdee613e"}
Sep 10 14:24:58.547992 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:58.547850    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2xkx" event={"ID":"233305e5-df37-4854-bddb-a383fe612b09","Type":"ContainerStarted","Data":"61b0662ed6e08968606b28ec1a694cf22e63eaa32074ecbf3acef3f8f4f8a847"}
Sep 10 14:24:59.553589 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:24:59.553538    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2xkx" event={"ID":"233305e5-df37-4854-bddb-a383fe612b09","Type":"ContainerStarted","Data":"dbf00a98eb54e9d3e8520fc0f5236137d209565d7d6afef2ac3bd925c8cf6eea"}
Sep 10 14:25:00.163553 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:00.163521    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:25:05.975810 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:05.975772    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-fxvnj"]
Sep 10 14:25:05.979456 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:05.979433    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fxvnj"
Sep 10 14:25:05.985217 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:05.985192    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wx5bs\""
Sep 10 14:25:05.985350 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:05.985229    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 10 14:25:05.985350 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:05.985240    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 10 14:25:05.985589 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:05.985559    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 14:25:05.985673 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:05.985560    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 14:25:05.985673 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:05.985565    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 14:25:06.019916 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.019881    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-fxvnj"]
Sep 10 14:25:06.028662 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.028624    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12b839b8-01d2-4eb6-ab72-67f26377a6d2-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-fxvnj\" (UID: \"12b839b8-01d2-4eb6-ab72-67f26377a6d2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fxvnj"
Sep 10 14:25:06.028846 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.028668    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12b839b8-01d2-4eb6-ab72-67f26377a6d2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-fxvnj\" (UID: \"12b839b8-01d2-4eb6-ab72-67f26377a6d2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fxvnj"
Sep 10 14:25:06.028846 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.028736    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12b839b8-01d2-4eb6-ab72-67f26377a6d2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-fxvnj\" (UID: \"12b839b8-01d2-4eb6-ab72-67f26377a6d2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fxvnj"
Sep 10 14:25:06.028846 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.028772    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnsnj\" (UniqueName: \"kubernetes.io/projected/12b839b8-01d2-4eb6-ab72-67f26377a6d2-kube-api-access-cnsnj\") pod \"openshift-state-metrics-657fd6c54-fxvnj\" (UID: \"12b839b8-01d2-4eb6-ab72-67f26377a6d2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fxvnj"
Sep 10 14:25:06.129622 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.129578    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnsnj\" (UniqueName: \"kubernetes.io/projected/12b839b8-01d2-4eb6-ab72-67f26377a6d2-kube-api-access-cnsnj\") pod \"openshift-state-metrics-657fd6c54-fxvnj\" (UID: \"12b839b8-01d2-4eb6-ab72-67f26377a6d2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fxvnj"
Sep 10 14:25:06.129834 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.129726    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12b839b8-01d2-4eb6-ab72-67f26377a6d2-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-fxvnj\" (UID: \"12b839b8-01d2-4eb6-ab72-67f26377a6d2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fxvnj"
Sep 10 14:25:06.129834 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.129761    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12b839b8-01d2-4eb6-ab72-67f26377a6d2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-fxvnj\" (UID: \"12b839b8-01d2-4eb6-ab72-67f26377a6d2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fxvnj"
Sep 10 14:25:06.129834 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.129792    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12b839b8-01d2-4eb6-ab72-67f26377a6d2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-fxvnj\" (UID: \"12b839b8-01d2-4eb6-ab72-67f26377a6d2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fxvnj"
Sep 10 14:25:06.129988 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:25:06.129901    2547 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 10 14:25:06.129988 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:25:06.129968    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12b839b8-01d2-4eb6-ab72-67f26377a6d2-openshift-state-metrics-tls podName:12b839b8-01d2-4eb6-ab72-67f26377a6d2 nodeName:}" failed. No retries permitted until 2026-09-10 14:25:06.629950432 +0000 UTC m=+318.671824906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/12b839b8-01d2-4eb6-ab72-67f26377a6d2-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-fxvnj" (UID: "12b839b8-01d2-4eb6-ab72-67f26377a6d2") : secret "openshift-state-metrics-tls" not found
Sep 10 14:25:06.130527 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.130501    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12b839b8-01d2-4eb6-ab72-67f26377a6d2-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-fxvnj\" (UID: \"12b839b8-01d2-4eb6-ab72-67f26377a6d2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fxvnj"
Sep 10 14:25:06.132389 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.132352    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12b839b8-01d2-4eb6-ab72-67f26377a6d2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-fxvnj\" (UID: \"12b839b8-01d2-4eb6-ab72-67f26377a6d2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fxvnj"
Sep 10 14:25:06.154144 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.154116    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnsnj\" (UniqueName: \"kubernetes.io/projected/12b839b8-01d2-4eb6-ab72-67f26377a6d2-kube-api-access-cnsnj\") pod \"openshift-state-metrics-657fd6c54-fxvnj\" (UID: \"12b839b8-01d2-4eb6-ab72-67f26377a6d2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fxvnj"
Sep 10 14:25:06.243884 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.243809    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-btcc8"]
Sep 10 14:25:06.247332 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.247301    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.250304 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.250281    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 14:25:06.250438 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.250383    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 14:25:06.254293 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.254273    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 14:25:06.255359 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.255336    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4vtq9\""
Sep 10 14:25:06.331139 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.331101    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d3d95df4-9fab-4b3c-a604-c856fd795d00-node-exporter-tls\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.331139 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.331137    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zszcf\" (UniqueName: \"kubernetes.io/projected/d3d95df4-9fab-4b3c-a604-c856fd795d00-kube-api-access-zszcf\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.331384 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.331156    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d3d95df4-9fab-4b3c-a604-c856fd795d00-node-exporter-textfile\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.331384 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.331211    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d3d95df4-9fab-4b3c-a604-c856fd795d00-root\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.331384 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.331261    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d3d95df4-9fab-4b3c-a604-c856fd795d00-node-exporter-accelerators-collector-config\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.331384 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.331324    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3d95df4-9fab-4b3c-a604-c856fd795d00-metrics-client-ca\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.331384 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.331348    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3d95df4-9fab-4b3c-a604-c856fd795d00-sys\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.331384 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.331370    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3d95df4-9fab-4b3c-a604-c856fd795d00-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.331616 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.331397    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d3d95df4-9fab-4b3c-a604-c856fd795d00-node-exporter-wtmp\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.432466 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.432430    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d3d95df4-9fab-4b3c-a604-c856fd795d00-node-exporter-accelerators-collector-config\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.432661 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.432528    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3d95df4-9fab-4b3c-a604-c856fd795d00-metrics-client-ca\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.432661 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.432558    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3d95df4-9fab-4b3c-a604-c856fd795d00-sys\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.432661 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.432586    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3d95df4-9fab-4b3c-a604-c856fd795d00-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.432661 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.432612    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d3d95df4-9fab-4b3c-a604-c856fd795d00-node-exporter-wtmp\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.432661 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.432650    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d3d95df4-9fab-4b3c-a604-c856fd795d00-node-exporter-tls\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.432960 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.432674    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zszcf\" (UniqueName: \"kubernetes.io/projected/d3d95df4-9fab-4b3c-a604-c856fd795d00-kube-api-access-zszcf\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.432960 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.432713    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d3d95df4-9fab-4b3c-a604-c856fd795d00-node-exporter-textfile\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.432960 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.432714    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3d95df4-9fab-4b3c-a604-c856fd795d00-sys\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.432960 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.432780    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d3d95df4-9fab-4b3c-a604-c856fd795d00-root\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.432960 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.432854    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d3d95df4-9fab-4b3c-a604-c856fd795d00-node-exporter-wtmp\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.432960 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.432867    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d3d95df4-9fab-4b3c-a604-c856fd795d00-root\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.433249 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.433188    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d3d95df4-9fab-4b3c-a604-c856fd795d00-node-exporter-accelerators-collector-config\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.433249 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.433194    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3d95df4-9fab-4b3c-a604-c856fd795d00-metrics-client-ca\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.433444 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.433423    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d3d95df4-9fab-4b3c-a604-c856fd795d00-node-exporter-textfile\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.435381 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.435356    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3d95df4-9fab-4b3c-a604-c856fd795d00-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.435541 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.435521    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d3d95df4-9fab-4b3c-a604-c856fd795d00-node-exporter-tls\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.535517 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.535449    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zszcf\" (UniqueName: \"kubernetes.io/projected/d3d95df4-9fab-4b3c-a604-c856fd795d00-kube-api-access-zszcf\") pod \"node-exporter-btcc8\" (UID: \"d3d95df4-9fab-4b3c-a604-c856fd795d00\") " pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.558420 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.558378    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-btcc8"
Sep 10 14:25:06.634531 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.634492    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12b839b8-01d2-4eb6-ab72-67f26377a6d2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-fxvnj\" (UID: \"12b839b8-01d2-4eb6-ab72-67f26377a6d2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fxvnj"
Sep 10 14:25:06.637223 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.637195    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12b839b8-01d2-4eb6-ab72-67f26377a6d2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-fxvnj\" (UID: \"12b839b8-01d2-4eb6-ab72-67f26377a6d2\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fxvnj"
Sep 10 14:25:06.890917 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:06.890824    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fxvnj"
Sep 10 14:25:07.422985 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:25:07.422930    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d95df4_9fab_4b3c_a604_c856fd795d00.slice/crio-15746b2ff8e1510d956050bfe0f1475673af6db461cc0719e390be0de4e03038 WatchSource:0}: Error finding container 15746b2ff8e1510d956050bfe0f1475673af6db461cc0719e390be0de4e03038: Status 404 returned error can't find the container with id 15746b2ff8e1510d956050bfe0f1475673af6db461cc0719e390be0de4e03038
Sep 10 14:25:07.584174 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:07.584133    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-btcc8" event={"ID":"d3d95df4-9fab-4b3c-a604-c856fd795d00","Type":"ContainerStarted","Data":"15746b2ff8e1510d956050bfe0f1475673af6db461cc0719e390be0de4e03038"}
Sep 10 14:25:07.585812 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:07.585788    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-dsgb8" event={"ID":"af9ef9d2-bc7a-4d77-b0ac-ed3531249fcc","Type":"ContainerStarted","Data":"16f960d5496851967772dddd845e0c8d7bcd73c527a65f79b17e74ab44c59a70"}
Sep 10 14:25:07.585973 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:07.585952    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-dsgb8"
Sep 10 14:25:07.587445 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:07.587405    2547 patch_prober.go:28] interesting pod/downloads-85d98c998f-dsgb8 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.16:8080/\": dial tcp 10.132.0.16:8080: connect: connection refused" start-of-body=
Sep 10 14:25:07.587531 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:07.587479    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-85d98c998f-dsgb8" podUID="af9ef9d2-bc7a-4d77-b0ac-ed3531249fcc" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.16:8080/\": dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 10 14:25:07.588287 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:07.588257    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2xkx" event={"ID":"233305e5-df37-4854-bddb-a383fe612b09","Type":"ContainerStarted","Data":"16c5da58c9d100c5dad9e64efd4610538d9108110807f637d3f4f043032a782a"}
Sep 10 14:25:07.621600 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:07.621547    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-dsgb8" podStartSLOduration=0.88950955 podStartE2EDuration="17.621531679s" totalImagesPullingTime="16.732022129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:25:07.620832187 +0000 UTC m=+319.662706706" watchObservedRunningTime="2026-09-10 14:25:07.621531679 +0000 UTC m=+319.663406159"
Sep 10 14:25:07.670164 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:07.670056    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-c2xkx" podStartSLOduration=33.482789738 podStartE2EDuration="42.670035856s" totalImagesPullingTime="9.187246118s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:24:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:25:07.669559061 +0000 UTC m=+319.711433543" watchObservedRunningTime="2026-09-10 14:25:07.670035856 +0000 UTC m=+319.711910337"
Sep 10 14:25:07.731666 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:07.731632    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-fxvnj"]
Sep 10 14:25:07.735519 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:25:07.735484    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b839b8_01d2_4eb6_ab72_67f26377a6d2.slice/crio-866dfb14e41ffc76ea656975b54560b1104d385b5ee82385461290ce4a1ff86f WatchSource:0}: Error finding container 866dfb14e41ffc76ea656975b54560b1104d385b5ee82385461290ce4a1ff86f: Status 404 returned error can't find the container with id 866dfb14e41ffc76ea656975b54560b1104d385b5ee82385461290ce4a1ff86f
Sep 10 14:25:08.594426 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:08.594387    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fxvnj" event={"ID":"12b839b8-01d2-4eb6-ab72-67f26377a6d2","Type":"ContainerStarted","Data":"89d40787e285f332a679ab6acde336b6af67fb1bd740debc6752467c2f8b700f"}
Sep 10 14:25:08.594893 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:08.594466    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fxvnj" event={"ID":"12b839b8-01d2-4eb6-ab72-67f26377a6d2","Type":"ContainerStarted","Data":"a636c99fb3bcacb332b34bf0b24547c5a734cf843b186aa9cdf97051069316c8"}
Sep 10 14:25:08.594893 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:08.594499    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fxvnj" event={"ID":"12b839b8-01d2-4eb6-ab72-67f26377a6d2","Type":"ContainerStarted","Data":"866dfb14e41ffc76ea656975b54560b1104d385b5ee82385461290ce4a1ff86f"}
Sep 10 14:25:08.596879 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:08.596853    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-btcc8" event={"ID":"d3d95df4-9fab-4b3c-a604-c856fd795d00","Type":"ContainerStarted","Data":"6615b5cda2eb90ab188dfd84c0e58c08f2ab12ebc0135284be88e24976b24ce1"}
Sep 10 14:25:08.606852 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:08.606819    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-dsgb8"
Sep 10 14:25:09.176303 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.175612    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"]
Sep 10 14:25:09.181264 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.180468    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.184927 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.184543    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vdcsb\""
Sep 10 14:25:09.184927 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.184728    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 10 14:25:09.184927 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.184802    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 10 14:25:09.185175 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.185002    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2a7rcgi0fcv0e\""
Sep 10 14:25:09.186741 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.186245    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 10 14:25:09.186741 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.186268    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 10 14:25:09.186741 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.186526    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 10 14:25:09.197344 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.196765    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"]
Sep 10 14:25:09.261391 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.260986    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hgbh\" (UniqueName: \"kubernetes.io/projected/8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19-kube-api-access-8hgbh\") pod \"thanos-querier-ffd5c7b6b-4csxv\" (UID: \"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19\") " pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.261391 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.261035    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19-secret-grpc-tls\") pod \"thanos-querier-ffd5c7b6b-4csxv\" (UID: \"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19\") " pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.261391 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.261071    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-ffd5c7b6b-4csxv\" (UID: \"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19\") " pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.261391 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.261119    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-ffd5c7b6b-4csxv\" (UID: \"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19\") " pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.261391 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.261184    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19-secret-thanos-querier-tls\") pod \"thanos-querier-ffd5c7b6b-4csxv\" (UID: \"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19\") " pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.261391 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.261218    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-ffd5c7b6b-4csxv\" (UID: \"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19\") " pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.261391 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.261243    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19-metrics-client-ca\") pod \"thanos-querier-ffd5c7b6b-4csxv\" (UID: \"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19\") " pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.261391 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.261268    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-ffd5c7b6b-4csxv\" (UID: \"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19\") " pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.363240 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.362410    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19-secret-thanos-querier-tls\") pod \"thanos-querier-ffd5c7b6b-4csxv\" (UID: \"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19\") " pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.363240 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.362468    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-ffd5c7b6b-4csxv\" (UID: \"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19\") " pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.363240 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.362499    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19-metrics-client-ca\") pod \"thanos-querier-ffd5c7b6b-4csxv\" (UID: \"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19\") " pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.363240 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.362541    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-ffd5c7b6b-4csxv\" (UID: \"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19\") " pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.363240 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.362618    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hgbh\" (UniqueName: \"kubernetes.io/projected/8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19-kube-api-access-8hgbh\") pod \"thanos-querier-ffd5c7b6b-4csxv\" (UID: \"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19\") " pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.363240 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.362654    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19-secret-grpc-tls\") pod \"thanos-querier-ffd5c7b6b-4csxv\" (UID: \"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19\") " pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.363240 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.362691    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-ffd5c7b6b-4csxv\" (UID: \"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19\") " pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.363240 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.363182    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-ffd5c7b6b-4csxv\" (UID: \"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19\") " pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.367556 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.363614    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19-metrics-client-ca\") pod \"thanos-querier-ffd5c7b6b-4csxv\" (UID: \"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19\") " pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.367556 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.367457    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-ffd5c7b6b-4csxv\" (UID: \"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19\") " pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.368001 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.367953    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19-secret-thanos-querier-tls\") pod \"thanos-querier-ffd5c7b6b-4csxv\" (UID: \"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19\") " pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.368803 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.368753    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-ffd5c7b6b-4csxv\" (UID: \"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19\") " pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.368803 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.368767    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19-secret-grpc-tls\") pod \"thanos-querier-ffd5c7b6b-4csxv\" (UID: \"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19\") " pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.369155 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.369127    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-ffd5c7b6b-4csxv\" (UID: \"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19\") " pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.369510 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.369469    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-ffd5c7b6b-4csxv\" (UID: \"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19\") " pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.372947 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.372923    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hgbh\" (UniqueName: \"kubernetes.io/projected/8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19-kube-api-access-8hgbh\") pod \"thanos-querier-ffd5c7b6b-4csxv\" (UID: \"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19\") " pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.497637 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.497607    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:09.605060 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.605017    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fxvnj" event={"ID":"12b839b8-01d2-4eb6-ab72-67f26377a6d2","Type":"ContainerStarted","Data":"e0ecb3596c750f09729786223e95bcf6cee953f5f1813bc7439308726fab3fe6"}
Sep 10 14:25:09.607250 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.607221    2547 generic.go:356] "Generic (PLEG): container finished" podID="d3d95df4-9fab-4b3c-a604-c856fd795d00" containerID="6615b5cda2eb90ab188dfd84c0e58c08f2ab12ebc0135284be88e24976b24ce1" exitCode=0
Sep 10 14:25:09.607366 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.607317    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-btcc8" event={"ID":"d3d95df4-9fab-4b3c-a604-c856fd795d00","Type":"ContainerDied","Data":"6615b5cda2eb90ab188dfd84c0e58c08f2ab12ebc0135284be88e24976b24ce1"}
Sep 10 14:25:09.642312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.642050    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fxvnj" podStartSLOduration=3.016437434 podStartE2EDuration="4.642034667s" totalImagesPullingTime="1.625597233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:25:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:25:09.640636425 +0000 UTC m=+321.682510906" watchObservedRunningTime="2026-09-10 14:25:09.642034667 +0000 UTC m=+321.683909148"
Sep 10 14:25:09.806597 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:09.806564    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"]
Sep 10 14:25:09.811309 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:25:09.811279    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e764f9d_c7bb_4fb1_b293_ecd77ebb4a19.slice/crio-9a354e3719dd1ed0e4e8d1348c228c76cdd60d36792c927d113d045efe101b30 WatchSource:0}: Error finding container 9a354e3719dd1ed0e4e8d1348c228c76cdd60d36792c927d113d045efe101b30: Status 404 returned error can't find the container with id 9a354e3719dd1ed0e4e8d1348c228c76cdd60d36792c927d113d045efe101b30
Sep 10 14:25:10.617692 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:10.617639    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv" event={"ID":"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19","Type":"ContainerStarted","Data":"9a354e3719dd1ed0e4e8d1348c228c76cdd60d36792c927d113d045efe101b30"}
Sep 10 14:25:10.622146 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:10.622118    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-btcc8" event={"ID":"d3d95df4-9fab-4b3c-a604-c856fd795d00","Type":"ContainerStarted","Data":"4855226cf6d65bd99736776275d9fc4049b8c67fff11ed2566b2fb7a849a53c7"}
Sep 10 14:25:10.622367 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:10.622354    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-btcc8" event={"ID":"d3d95df4-9fab-4b3c-a604-c856fd795d00","Type":"ContainerStarted","Data":"173eec82b32a4c4ac9968009746ba035b88a9779cc353151a8a119a7b7b66f95"}
Sep 10 14:25:10.660117 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:10.660056    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-btcc8" podStartSLOduration=3.50468602 podStartE2EDuration="4.660036018s" totalImagesPullingTime="1.075594823s" totalInitContainerRuntime="79.755175ms" isStatefulPod=true podCreationTimestamp="2026-09-10 14:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:25:10.658430246 +0000 UTC m=+322.700304721" watchObservedRunningTime="2026-09-10 14:25:10.660036018 +0000 UTC m=+322.701910500"
Sep 10 14:25:12.632076 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:12.632045    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv" event={"ID":"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19","Type":"ContainerStarted","Data":"37085bf96b04637a0a5fc52c8e2fc3890f0eab39cd6680d0cda3e0a68988bd3e"}
Sep 10 14:25:13.644032 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:13.643989    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv" event={"ID":"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19","Type":"ContainerStarted","Data":"81575454ab0b2ad8f16ed6e9cd2d2c223db0f80806fd18782fd82ad2171179b8"}
Sep 10 14:25:13.644467 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:13.644094    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv" event={"ID":"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19","Type":"ContainerStarted","Data":"1cbeb45b41f0f335ab2057caee6d90c10fce4597f8fcca247045eab2b4da6bdb"}
Sep 10 14:25:15.183354 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.183302    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x" podUID="6e2abaa4-6e03-493a-90ef-833dbab4d41e" containerName="registry" containerID="cri-o://0d46e6a7d3934d9ac069ba1db1fb045d53e22364604d5ba3b759fa381d912b71" gracePeriod=30
Sep 10 14:25:15.467433 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.467398    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:25:15.520551 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.520509    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrz7r\" (UniqueName: \"kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-kube-api-access-qrz7r\") pod \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") "
Sep 10 14:25:15.520751 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.520561    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6e2abaa4-6e03-493a-90ef-833dbab4d41e-installation-pull-secrets\") pod \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") "
Sep 10 14:25:15.520751 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.520588    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-bound-sa-token\") pod \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") "
Sep 10 14:25:15.520751 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.520621    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6e2abaa4-6e03-493a-90ef-833dbab4d41e-image-registry-private-configuration\") pod \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") "
Sep 10 14:25:15.520751 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.520740    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6e2abaa4-6e03-493a-90ef-833dbab4d41e-trusted-ca\") pod \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") "
Sep 10 14:25:15.520974 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.520772    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-tls\") pod \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") "
Sep 10 14:25:15.520974 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.520802    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6e2abaa4-6e03-493a-90ef-833dbab4d41e-ca-trust-extracted\") pod \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") "
Sep 10 14:25:15.520974 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.520847    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-certificates\") pod \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\" (UID: \"6e2abaa4-6e03-493a-90ef-833dbab4d41e\") "
Sep 10 14:25:15.521906 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.521525    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6e2abaa4-6e03-493a-90ef-833dbab4d41e" (UID: "6e2abaa4-6e03-493a-90ef-833dbab4d41e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:25:15.522210 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.522164    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e2abaa4-6e03-493a-90ef-833dbab4d41e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6e2abaa4-6e03-493a-90ef-833dbab4d41e" (UID: "6e2abaa4-6e03-493a-90ef-833dbab4d41e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:25:15.523355 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.523327    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e2abaa4-6e03-493a-90ef-833dbab4d41e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6e2abaa4-6e03-493a-90ef-833dbab4d41e" (UID: "6e2abaa4-6e03-493a-90ef-833dbab4d41e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:25:15.523459 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.523430    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-kube-api-access-qrz7r" (OuterVolumeSpecName: "kube-api-access-qrz7r") pod "6e2abaa4-6e03-493a-90ef-833dbab4d41e" (UID: "6e2abaa4-6e03-493a-90ef-833dbab4d41e"). InnerVolumeSpecName "kube-api-access-qrz7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:25:15.524495 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.524466    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6e2abaa4-6e03-493a-90ef-833dbab4d41e" (UID: "6e2abaa4-6e03-493a-90ef-833dbab4d41e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:25:15.524585 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.524495    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6e2abaa4-6e03-493a-90ef-833dbab4d41e" (UID: "6e2abaa4-6e03-493a-90ef-833dbab4d41e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:25:15.525099 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.525072    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e2abaa4-6e03-493a-90ef-833dbab4d41e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6e2abaa4-6e03-493a-90ef-833dbab4d41e" (UID: "6e2abaa4-6e03-493a-90ef-833dbab4d41e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:25:15.533311 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.533284    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e2abaa4-6e03-493a-90ef-833dbab4d41e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6e2abaa4-6e03-493a-90ef-833dbab4d41e" (UID: "6e2abaa4-6e03-493a-90ef-833dbab4d41e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:25:15.621835 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.621760    2547 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6e2abaa4-6e03-493a-90ef-833dbab4d41e-trusted-ca\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:25:15.621835 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.621791    2547 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-tls\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:25:15.621835 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.621805    2547 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6e2abaa4-6e03-493a-90ef-833dbab4d41e-ca-trust-extracted\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:25:15.621835 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.621820    2547 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6e2abaa4-6e03-493a-90ef-833dbab4d41e-registry-certificates\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:25:15.621835 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.621832    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrz7r\" (UniqueName: \"kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-kube-api-access-qrz7r\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:25:15.621835 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.621840    2547 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6e2abaa4-6e03-493a-90ef-833dbab4d41e-installation-pull-secrets\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:25:15.622172 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.621848    2547 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6e2abaa4-6e03-493a-90ef-833dbab4d41e-bound-sa-token\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:25:15.622172 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.621857    2547 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6e2abaa4-6e03-493a-90ef-833dbab4d41e-image-registry-private-configuration\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:25:15.655899 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.655857    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv" event={"ID":"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19","Type":"ContainerStarted","Data":"5ed04ee925de56cc5316be4d1d49fe85681023c14a8238cd0c19cc9f8da2128b"}
Sep 10 14:25:15.656086 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.655948    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv" event={"ID":"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19","Type":"ContainerStarted","Data":"f4629c47992e86cabb28b07569e517f6e10970620fa59fbf806265eb99c14632"}
Sep 10 14:25:15.656086 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.655993    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv" event={"ID":"8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19","Type":"ContainerStarted","Data":"7c5cd2ddd771bf3bcaba28bf3349a4c4fc9a4eca65265041512844e39b0d5dc1"}
Sep 10 14:25:15.656086 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.656042    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:15.657339 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.657308    2547 generic.go:356] "Generic (PLEG): container finished" podID="6e2abaa4-6e03-493a-90ef-833dbab4d41e" containerID="0d46e6a7d3934d9ac069ba1db1fb045d53e22364604d5ba3b759fa381d912b71" exitCode=0
Sep 10 14:25:15.657457 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.657341    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x"
Sep 10 14:25:15.657457 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.657365    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x" event={"ID":"6e2abaa4-6e03-493a-90ef-833dbab4d41e","Type":"ContainerDied","Data":"0d46e6a7d3934d9ac069ba1db1fb045d53e22364604d5ba3b759fa381d912b71"}
Sep 10 14:25:15.657457 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.657445    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79489b5cbc-zgr4x" event={"ID":"6e2abaa4-6e03-493a-90ef-833dbab4d41e","Type":"ContainerDied","Data":"762b9bb5b45403f8bfbd46daf6a4d0ce16944240dd9e77a0a77da18519a07ff3"}
Sep 10 14:25:15.657611 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.657468    2547 scope.go:117] "RemoveContainer" containerID="0d46e6a7d3934d9ac069ba1db1fb045d53e22364604d5ba3b759fa381d912b71"
Sep 10 14:25:15.667429 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.667402    2547 scope.go:117] "RemoveContainer" containerID="0d46e6a7d3934d9ac069ba1db1fb045d53e22364604d5ba3b759fa381d912b71"
Sep 10 14:25:15.667778 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:25:15.667751    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d46e6a7d3934d9ac069ba1db1fb045d53e22364604d5ba3b759fa381d912b71\": container with ID starting with 0d46e6a7d3934d9ac069ba1db1fb045d53e22364604d5ba3b759fa381d912b71 not found: ID does not exist" containerID="0d46e6a7d3934d9ac069ba1db1fb045d53e22364604d5ba3b759fa381d912b71"
Sep 10 14:25:15.667871 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.667789    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d46e6a7d3934d9ac069ba1db1fb045d53e22364604d5ba3b759fa381d912b71"} err="failed to get container status \"0d46e6a7d3934d9ac069ba1db1fb045d53e22364604d5ba3b759fa381d912b71\": rpc error: code = NotFound desc = could not find container \"0d46e6a7d3934d9ac069ba1db1fb045d53e22364604d5ba3b759fa381d912b71\": container with ID starting with 0d46e6a7d3934d9ac069ba1db1fb045d53e22364604d5ba3b759fa381d912b71 not found: ID does not exist"
Sep 10 14:25:15.700852 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.700796    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv" podStartSLOduration=2.089330171 podStartE2EDuration="6.700779312s" totalImagesPullingTime="4.611449141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:25:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:25:15.699380642 +0000 UTC m=+327.741255122" watchObservedRunningTime="2026-09-10 14:25:15.700779312 +0000 UTC m=+327.742653793"
Sep 10 14:25:15.735914 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.735877    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79489b5cbc-zgr4x"]
Sep 10 14:25:15.738753 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.738729    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-79489b5cbc-zgr4x"]
Sep 10 14:25:15.955095 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.955059    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c677db8b6-w2fld"]
Sep 10 14:25:15.955442 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.955421    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e2abaa4-6e03-493a-90ef-833dbab4d41e" containerName="registry"
Sep 10 14:25:15.955442 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.955440    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e2abaa4-6e03-493a-90ef-833dbab4d41e" containerName="registry"
Sep 10 14:25:15.955623 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.955555    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e2abaa4-6e03-493a-90ef-833dbab4d41e" containerName="registry"
Sep 10 14:25:15.991537 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.991505    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c677db8b6-w2fld"]
Sep 10 14:25:15.991723 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.991643    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:15.996009 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.995979    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 14:25:15.996145 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.995990    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 14:25:15.996213 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.996146    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 14:25:15.996411 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.996391    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 14:25:15.997345 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.997279    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 14:25:15.997584 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:15.997503    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zf5dh\""
Sep 10 14:25:16.001879 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.001856    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 14:25:16.127135 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.127089    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-console-config\") pod \"console-7c677db8b6-w2fld\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") " pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:16.127315 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.127146    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmrvn\" (UniqueName: \"kubernetes.io/projected/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-kube-api-access-kmrvn\") pod \"console-7c677db8b6-w2fld\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") " pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:16.127315 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.127186    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-oauth-serving-cert\") pod \"console-7c677db8b6-w2fld\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") " pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:16.127315 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.127224    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-console-serving-cert\") pod \"console-7c677db8b6-w2fld\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") " pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:16.127315 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.127252    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-service-ca\") pod \"console-7c677db8b6-w2fld\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") " pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:16.127315 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.127279    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-console-oauth-config\") pod \"console-7c677db8b6-w2fld\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") " pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:16.127522 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.127342    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-trusted-ca-bundle\") pod \"console-7c677db8b6-w2fld\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") " pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:16.234067 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.233974    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-trusted-ca-bundle\") pod \"console-7c677db8b6-w2fld\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") " pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:16.234501 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.234116    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-console-config\") pod \"console-7c677db8b6-w2fld\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") " pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:16.234501 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.234160    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmrvn\" (UniqueName: \"kubernetes.io/projected/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-kube-api-access-kmrvn\") pod \"console-7c677db8b6-w2fld\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") " pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:16.234501 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.234204    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-oauth-serving-cert\") pod \"console-7c677db8b6-w2fld\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") " pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:16.234501 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.234244    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-console-serving-cert\") pod \"console-7c677db8b6-w2fld\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") " pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:16.234501 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.234278    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-service-ca\") pod \"console-7c677db8b6-w2fld\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") " pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:16.234501 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.234311    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-console-oauth-config\") pod \"console-7c677db8b6-w2fld\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") " pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:16.235853 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.235821    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-console-config\") pod \"console-7c677db8b6-w2fld\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") " pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:16.235988 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.235960    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-oauth-serving-cert\") pod \"console-7c677db8b6-w2fld\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") " pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:16.236051 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.235994    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-service-ca\") pod \"console-7c677db8b6-w2fld\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") " pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:16.236716 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.236679    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-trusted-ca-bundle\") pod \"console-7c677db8b6-w2fld\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") " pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:16.237334 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.237308    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-console-oauth-config\") pod \"console-7c677db8b6-w2fld\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") " pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:16.237994 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.237972    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-console-serving-cert\") pod \"console-7c677db8b6-w2fld\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") " pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:16.243410 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.243388    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmrvn\" (UniqueName: \"kubernetes.io/projected/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-kube-api-access-kmrvn\") pod \"console-7c677db8b6-w2fld\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") " pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:16.304537 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.304500    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:16.423374 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.423330    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e2abaa4-6e03-493a-90ef-833dbab4d41e" path="/var/lib/kubelet/pods/6e2abaa4-6e03-493a-90ef-833dbab4d41e/volumes"
Sep 10 14:25:16.618354 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.618268    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c677db8b6-w2fld"]
Sep 10 14:25:16.621513 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:25:16.621480    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e3fe6d_e07f_40e0_9beb_0bdb3fe50d1b.slice/crio-d12cb1d8010eff895e05fbb7e1703427f88d999d6af6a50f44e1fbb0ddd49297 WatchSource:0}: Error finding container d12cb1d8010eff895e05fbb7e1703427f88d999d6af6a50f44e1fbb0ddd49297: Status 404 returned error can't find the container with id d12cb1d8010eff895e05fbb7e1703427f88d999d6af6a50f44e1fbb0ddd49297
Sep 10 14:25:16.664872 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:16.664825    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c677db8b6-w2fld" event={"ID":"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b","Type":"ContainerStarted","Data":"d12cb1d8010eff895e05fbb7e1703427f88d999d6af6a50f44e1fbb0ddd49297"}
Sep 10 14:25:20.682232 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:20.682201    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c677db8b6-w2fld" event={"ID":"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b","Type":"ContainerStarted","Data":"7455402e2047bd0447d8f1b16f4c2a44ede8e735de44c5c8f63fc051fe3afdb7"}
Sep 10 14:25:20.701508 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:20.701452    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c677db8b6-w2fld" podStartSLOduration=2.122666137 podStartE2EDuration="5.701431629s" totalImagesPullingTime="3.578765492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:25:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:25:20.700332791 +0000 UTC m=+332.742207273" watchObservedRunningTime="2026-09-10 14:25:20.701431629 +0000 UTC m=+332.743306112"
Sep 10 14:25:21.672155 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:21.672116    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-ffd5c7b6b-4csxv"
Sep 10 14:25:26.304988 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:26.304945    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:26.305373 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:26.305021    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:26.309823 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:26.309798    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:26.706731 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:26.706686    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:25:35.734671 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:35.734635    2547 generic.go:356] "Generic (PLEG): container finished" podID="43b9ae1f-18aa-4310-bdcd-4d9b296d4e42" containerID="52e693fb872c10560bb27614a85a04291d1c212794a38063193ced41ebcf867b" exitCode=0
Sep 10 14:25:35.735075 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:35.734728    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7dvk2" event={"ID":"43b9ae1f-18aa-4310-bdcd-4d9b296d4e42","Type":"ContainerDied","Data":"52e693fb872c10560bb27614a85a04291d1c212794a38063193ced41ebcf867b"}
Sep 10 14:25:35.735075 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:35.735033    2547 scope.go:117] "RemoveContainer" containerID="52e693fb872c10560bb27614a85a04291d1c212794a38063193ced41ebcf867b"
Sep 10 14:25:36.739800 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:36.739757    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-7dvk2" event={"ID":"43b9ae1f-18aa-4310-bdcd-4d9b296d4e42","Type":"ContainerStarted","Data":"c76c835092d84078424d0fa718fd6d189d99c7f0be87725520546b62462643f8"}
Sep 10 14:25:56.243551 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:25:56.243522    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vz9vp_5a663e3c-6895-4085-aa41-1efd5d6ab208/dns-node-resolver/0.log"
Sep 10 14:26:01.188885 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:26:01.188834    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs\") pod \"network-metrics-daemon-756dn\" (UID: \"c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0\") " pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:26:01.191118 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:26:01.191087    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0-metrics-certs\") pod \"network-metrics-daemon-756dn\" (UID: \"c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0\") " pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:26:01.218917 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:26:01.218889    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-flc9p\""
Sep 10 14:26:01.226386 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:26:01.226365    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-756dn"
Sep 10 14:26:01.494563 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:26:01.494462    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-756dn"]
Sep 10 14:26:01.497471 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:26:01.497440    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1994e0b_ad17_4c7b_bdd1_4672e8d1efd0.slice/crio-d82b7a4eea95aa1e9cdb017b25a076928b025ad360bbc185f1e294345ccf9f3b WatchSource:0}: Error finding container d82b7a4eea95aa1e9cdb017b25a076928b025ad360bbc185f1e294345ccf9f3b: Status 404 returned error can't find the container with id d82b7a4eea95aa1e9cdb017b25a076928b025ad360bbc185f1e294345ccf9f3b
Sep 10 14:26:01.829451 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:26:01.829361    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-756dn" event={"ID":"c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0","Type":"ContainerStarted","Data":"d82b7a4eea95aa1e9cdb017b25a076928b025ad360bbc185f1e294345ccf9f3b"}
Sep 10 14:26:02.834930 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:26:02.834840    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-756dn" event={"ID":"c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0","Type":"ContainerStarted","Data":"66d0a99c1a7f3743d44da5688b63b796a7f20adf74fb8fba983614fdd6d8a35d"}
Sep 10 14:26:02.834930 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:26:02.834896    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-756dn" event={"ID":"c1994e0b-ad17-4c7b-bdd1-4672e8d1efd0","Type":"ContainerStarted","Data":"9954a5cf9337f84ebf87010f45a9e60427843490cdbcdec43f553352303ebe9d"}
Sep 10 14:26:02.851641 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:26:02.851586    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-756dn" podStartSLOduration=372.851352355 podStartE2EDuration="6m13.851570772s" totalImagesPullingTime="1.000218417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:26:02.850878438 +0000 UTC m=+374.892752919" watchObservedRunningTime="2026-09-10 14:26:02.851570772 +0000 UTC m=+374.893445252"
Sep 10 14:26:33.445851 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:26:33.445796    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-957xl" podUID="54e8b08a-0bfa-4e3f-87e3-d6a5e886c798"
Sep 10 14:26:33.946678 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:26:33.946646    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-957xl"
Sep 10 14:26:35.772407 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:26:35.772366    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls\") pod \"dns-default-957xl\" (UID: \"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798\") " pod="openshift-dns/dns-default-957xl"
Sep 10 14:26:35.774610 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:26:35.774588    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54e8b08a-0bfa-4e3f-87e3-d6a5e886c798-metrics-tls\") pod \"dns-default-957xl\" (UID: \"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798\") " pod="openshift-dns/dns-default-957xl"
Sep 10 14:26:36.050730 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:26:36.050639    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-587sj\""
Sep 10 14:26:36.059049 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:26:36.059029    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-957xl"
Sep 10 14:26:36.327398 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:26:36.327319    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-957xl"]
Sep 10 14:26:36.330432 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:26:36.330407    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e8b08a_0bfa_4e3f_87e3_d6a5e886c798.slice/crio-f51c9e6a8f80ac8a9cdf92f775d33b88565b2d4e3141e7b03e05d627c54f20ed WatchSource:0}: Error finding container f51c9e6a8f80ac8a9cdf92f775d33b88565b2d4e3141e7b03e05d627c54f20ed: Status 404 returned error can't find the container with id f51c9e6a8f80ac8a9cdf92f775d33b88565b2d4e3141e7b03e05d627c54f20ed
Sep 10 14:26:36.958963 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:26:36.958926    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-957xl" event={"ID":"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798","Type":"ContainerStarted","Data":"f51c9e6a8f80ac8a9cdf92f775d33b88565b2d4e3141e7b03e05d627c54f20ed"}
Sep 10 14:26:37.222203 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:26:37.222130    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c677db8b6-w2fld"]
Sep 10 14:26:38.969182 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:26:38.969142    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-957xl" event={"ID":"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798","Type":"ContainerStarted","Data":"a040b9f20639d193cd4f75e0fd317798ce6b6731a840bc29e05bfaea225c3eda"}
Sep 10 14:26:38.969768 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:26:38.969203    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-957xl" event={"ID":"54e8b08a-0bfa-4e3f-87e3-d6a5e886c798","Type":"ContainerStarted","Data":"8baf03a97a6d4f5bc2b3b13e0d988c838c1c773c2d1a40311a0cf3a2ab64b4bf"}
Sep 10 14:26:38.969768 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:26:38.969245    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-957xl"
Sep 10 14:26:38.992475 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:26:38.992411    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-957xl" podStartSLOduration=374.081313051 podStartE2EDuration="6m15.992394026s" totalImagesPullingTime="1.911080975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:20:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:26:38.990809538 +0000 UTC m=+411.032684020" watchObservedRunningTime="2026-09-10 14:26:38.992394026 +0000 UTC m=+411.034268506"
Sep 10 14:26:48.978215 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:26:48.978181    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-957xl"
Sep 10 14:27:02.251665 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:02.251598    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7c677db8b6-w2fld" podUID="62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b" containerName="console" containerID="cri-o://7455402e2047bd0447d8f1b16f4c2a44ede8e735de44c5c8f63fc051fe3afdb7" gracePeriod=15
Sep 10 14:27:02.489793 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:02.489759    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:27:02.568718 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:02.568607    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-oauth-serving-cert\") pod \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") "
Sep 10 14:27:02.568718 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:02.568649    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-service-ca\") pod \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") "
Sep 10 14:27:02.568718 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:02.568679    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-console-serving-cert\") pod \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") "
Sep 10 14:27:02.568975 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:02.568726    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-console-config\") pod \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") "
Sep 10 14:27:02.568975 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:02.568759    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmrvn\" (UniqueName: \"kubernetes.io/projected/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-kube-api-access-kmrvn\") pod \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") "
Sep 10 14:27:02.568975 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:02.568790    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-console-oauth-config\") pod \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") "
Sep 10 14:27:02.568975 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:02.568832    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-trusted-ca-bundle\") pod \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\" (UID: \"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b\") "
Sep 10 14:27:02.569176 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:02.569149    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-service-ca" (OuterVolumeSpecName: "service-ca") pod "62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b" (UID: "62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:27:02.569280 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:02.569253    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b" (UID: "62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:27:02.569430 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:02.569269    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-console-config" (OuterVolumeSpecName: "console-config") pod "62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b" (UID: "62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:27:02.569430 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:02.569327    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b" (UID: "62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:27:02.571105 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:02.571073    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-kube-api-access-kmrvn" (OuterVolumeSpecName: "kube-api-access-kmrvn") pod "62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b" (UID: "62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b"). InnerVolumeSpecName "kube-api-access-kmrvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:27:02.571245 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:02.571217    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b" (UID: "62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:27:02.571245 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:02.571234    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b" (UID: "62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:27:02.670053 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:02.670013    2547 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-console-oauth-config\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:27:02.670053 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:02.670043    2547 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-trusted-ca-bundle\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:27:02.670053 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:02.670053    2547 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-oauth-serving-cert\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:27:02.670053 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:02.670064    2547 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-service-ca\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:27:02.670307 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:02.670072    2547 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-console-serving-cert\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:27:02.670307 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:02.670081    2547 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-console-config\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:27:02.670307 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:02.670090    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmrvn\" (UniqueName: \"kubernetes.io/projected/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b-kube-api-access-kmrvn\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:27:03.057090 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:03.057052    2547 generic.go:356] "Generic (PLEG): container finished" podID="62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b" containerID="7455402e2047bd0447d8f1b16f4c2a44ede8e735de44c5c8f63fc051fe3afdb7" exitCode=0
Sep 10 14:27:03.057286 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:03.057108    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c677db8b6-w2fld"
Sep 10 14:27:03.057286 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:03.057128    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c677db8b6-w2fld" event={"ID":"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b","Type":"ContainerDied","Data":"7455402e2047bd0447d8f1b16f4c2a44ede8e735de44c5c8f63fc051fe3afdb7"}
Sep 10 14:27:03.057286 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:03.057168    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c677db8b6-w2fld" event={"ID":"62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b","Type":"ContainerDied","Data":"d12cb1d8010eff895e05fbb7e1703427f88d999d6af6a50f44e1fbb0ddd49297"}
Sep 10 14:27:03.057286 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:03.057187    2547 scope.go:117] "RemoveContainer" containerID="7455402e2047bd0447d8f1b16f4c2a44ede8e735de44c5c8f63fc051fe3afdb7"
Sep 10 14:27:03.065497 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:03.065477    2547 scope.go:117] "RemoveContainer" containerID="7455402e2047bd0447d8f1b16f4c2a44ede8e735de44c5c8f63fc051fe3afdb7"
Sep 10 14:27:03.065800 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:27:03.065777    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7455402e2047bd0447d8f1b16f4c2a44ede8e735de44c5c8f63fc051fe3afdb7\": container with ID starting with 7455402e2047bd0447d8f1b16f4c2a44ede8e735de44c5c8f63fc051fe3afdb7 not found: ID does not exist" containerID="7455402e2047bd0447d8f1b16f4c2a44ede8e735de44c5c8f63fc051fe3afdb7"
Sep 10 14:27:03.065852 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:03.065808    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7455402e2047bd0447d8f1b16f4c2a44ede8e735de44c5c8f63fc051fe3afdb7"} err="failed to get container status \"7455402e2047bd0447d8f1b16f4c2a44ede8e735de44c5c8f63fc051fe3afdb7\": rpc error: code = NotFound desc = could not find container \"7455402e2047bd0447d8f1b16f4c2a44ede8e735de44c5c8f63fc051fe3afdb7\": container with ID starting with 7455402e2047bd0447d8f1b16f4c2a44ede8e735de44c5c8f63fc051fe3afdb7 not found: ID does not exist"
Sep 10 14:27:03.078153 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:03.078124    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c677db8b6-w2fld"]
Sep 10 14:27:03.082131 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:03.082108    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c677db8b6-w2fld"]
Sep 10 14:27:04.419822 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:27:04.419787    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b" path="/var/lib/kubelet/pods/62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b/volumes"
Sep 10 14:29:48.315818 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:29:48.315788    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mg9d6_359d40e9-f428-4cc3-b4e4-991c081d664a/console-operator/1.log"
Sep 10 14:29:48.316318 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:29:48.316161    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mg9d6_359d40e9-f428-4cc3-b4e4-991c081d664a/console-operator/1.log"
Sep 10 14:30:43.630509 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.630476    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65df4d6d55-f2ncz"]
Sep 10 14:30:43.631028 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.630935    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b" containerName="console"
Sep 10 14:30:43.631028 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.630956    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b" containerName="console"
Sep 10 14:30:43.631141 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.631059    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="62e3fe6d-e07f-40e0-9beb-0bdb3fe50d1b" containerName="console"
Sep 10 14:30:43.633963 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.633942    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:43.636434 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.636412    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 14:30:43.636546 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.636417    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 14:30:43.636612 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.636541    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 14:30:43.637777 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.637754    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 14:30:43.637878 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.637790    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 14:30:43.637941 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.637915    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zf5dh\""
Sep 10 14:30:43.641983 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.641961    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 14:30:43.653666 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.653615    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65df4d6d55-f2ncz"]
Sep 10 14:30:43.692439 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.692407    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28fd9ffc-32ef-49c2-870b-c0b40ca95592-console-oauth-config\") pod \"console-65df4d6d55-f2ncz\" (UID: \"28fd9ffc-32ef-49c2-870b-c0b40ca95592\") " pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:43.692598 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.692445    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28fd9ffc-32ef-49c2-870b-c0b40ca95592-console-serving-cert\") pod \"console-65df4d6d55-f2ncz\" (UID: \"28fd9ffc-32ef-49c2-870b-c0b40ca95592\") " pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:43.692598 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.692468    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g44bk\" (UniqueName: \"kubernetes.io/projected/28fd9ffc-32ef-49c2-870b-c0b40ca95592-kube-api-access-g44bk\") pod \"console-65df4d6d55-f2ncz\" (UID: \"28fd9ffc-32ef-49c2-870b-c0b40ca95592\") " pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:43.692598 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.692498    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28fd9ffc-32ef-49c2-870b-c0b40ca95592-oauth-serving-cert\") pod \"console-65df4d6d55-f2ncz\" (UID: \"28fd9ffc-32ef-49c2-870b-c0b40ca95592\") " pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:43.692598 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.692528    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28fd9ffc-32ef-49c2-870b-c0b40ca95592-console-config\") pod \"console-65df4d6d55-f2ncz\" (UID: \"28fd9ffc-32ef-49c2-870b-c0b40ca95592\") " pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:43.692753 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.692621    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28fd9ffc-32ef-49c2-870b-c0b40ca95592-trusted-ca-bundle\") pod \"console-65df4d6d55-f2ncz\" (UID: \"28fd9ffc-32ef-49c2-870b-c0b40ca95592\") " pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:43.692753 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.692644    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28fd9ffc-32ef-49c2-870b-c0b40ca95592-service-ca\") pod \"console-65df4d6d55-f2ncz\" (UID: \"28fd9ffc-32ef-49c2-870b-c0b40ca95592\") " pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:43.793344 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.793303    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28fd9ffc-32ef-49c2-870b-c0b40ca95592-oauth-serving-cert\") pod \"console-65df4d6d55-f2ncz\" (UID: \"28fd9ffc-32ef-49c2-870b-c0b40ca95592\") " pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:43.793344 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.793349    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28fd9ffc-32ef-49c2-870b-c0b40ca95592-console-config\") pod \"console-65df4d6d55-f2ncz\" (UID: \"28fd9ffc-32ef-49c2-870b-c0b40ca95592\") " pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:43.793555 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.793525    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28fd9ffc-32ef-49c2-870b-c0b40ca95592-trusted-ca-bundle\") pod \"console-65df4d6d55-f2ncz\" (UID: \"28fd9ffc-32ef-49c2-870b-c0b40ca95592\") " pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:43.793555 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.793543    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28fd9ffc-32ef-49c2-870b-c0b40ca95592-service-ca\") pod \"console-65df4d6d55-f2ncz\" (UID: \"28fd9ffc-32ef-49c2-870b-c0b40ca95592\") " pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:43.793638 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.793572    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28fd9ffc-32ef-49c2-870b-c0b40ca95592-console-oauth-config\") pod \"console-65df4d6d55-f2ncz\" (UID: \"28fd9ffc-32ef-49c2-870b-c0b40ca95592\") " pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:43.793638 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.793593    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28fd9ffc-32ef-49c2-870b-c0b40ca95592-console-serving-cert\") pod \"console-65df4d6d55-f2ncz\" (UID: \"28fd9ffc-32ef-49c2-870b-c0b40ca95592\") " pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:43.793638 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.793608    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g44bk\" (UniqueName: \"kubernetes.io/projected/28fd9ffc-32ef-49c2-870b-c0b40ca95592-kube-api-access-g44bk\") pod \"console-65df4d6d55-f2ncz\" (UID: \"28fd9ffc-32ef-49c2-870b-c0b40ca95592\") " pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:43.794227 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.794188    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28fd9ffc-32ef-49c2-870b-c0b40ca95592-oauth-serving-cert\") pod \"console-65df4d6d55-f2ncz\" (UID: \"28fd9ffc-32ef-49c2-870b-c0b40ca95592\") " pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:43.794348 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.794188    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28fd9ffc-32ef-49c2-870b-c0b40ca95592-console-config\") pod \"console-65df4d6d55-f2ncz\" (UID: \"28fd9ffc-32ef-49c2-870b-c0b40ca95592\") " pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:43.794348 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.794283    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28fd9ffc-32ef-49c2-870b-c0b40ca95592-service-ca\") pod \"console-65df4d6d55-f2ncz\" (UID: \"28fd9ffc-32ef-49c2-870b-c0b40ca95592\") " pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:43.794507 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.794484    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28fd9ffc-32ef-49c2-870b-c0b40ca95592-trusted-ca-bundle\") pod \"console-65df4d6d55-f2ncz\" (UID: \"28fd9ffc-32ef-49c2-870b-c0b40ca95592\") " pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:43.796101 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.796077    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28fd9ffc-32ef-49c2-870b-c0b40ca95592-console-oauth-config\") pod \"console-65df4d6d55-f2ncz\" (UID: \"28fd9ffc-32ef-49c2-870b-c0b40ca95592\") " pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:43.796205 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.796186    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28fd9ffc-32ef-49c2-870b-c0b40ca95592-console-serving-cert\") pod \"console-65df4d6d55-f2ncz\" (UID: \"28fd9ffc-32ef-49c2-870b-c0b40ca95592\") " pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:43.801319 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.801294    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g44bk\" (UniqueName: \"kubernetes.io/projected/28fd9ffc-32ef-49c2-870b-c0b40ca95592-kube-api-access-g44bk\") pod \"console-65df4d6d55-f2ncz\" (UID: \"28fd9ffc-32ef-49c2-870b-c0b40ca95592\") " pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:43.943847 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:43.943813    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:44.216371 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:44.216285    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65df4d6d55-f2ncz"]
Sep 10 14:30:44.219149 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:30:44.219121    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28fd9ffc_32ef_49c2_870b_c0b40ca95592.slice/crio-c2bad857bf6fc6f8850d647b8044d6e6ee01f621a34b46cb0b77f4a58d807cea WatchSource:0}: Error finding container c2bad857bf6fc6f8850d647b8044d6e6ee01f621a34b46cb0b77f4a58d807cea: Status 404 returned error can't find the container with id c2bad857bf6fc6f8850d647b8044d6e6ee01f621a34b46cb0b77f4a58d807cea
Sep 10 14:30:44.846073 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:44.846036    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65df4d6d55-f2ncz" event={"ID":"28fd9ffc-32ef-49c2-870b-c0b40ca95592","Type":"ContainerStarted","Data":"0027e4769ffa60aa754dad1c9a7c97e5de6598fb3942a95d8d18e5493ebec2b5"}
Sep 10 14:30:44.846422 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:44.846090    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65df4d6d55-f2ncz" event={"ID":"28fd9ffc-32ef-49c2-870b-c0b40ca95592","Type":"ContainerStarted","Data":"c2bad857bf6fc6f8850d647b8044d6e6ee01f621a34b46cb0b77f4a58d807cea"}
Sep 10 14:30:44.863516 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:44.863464    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65df4d6d55-f2ncz" podStartSLOduration=1.8634523330000001 podStartE2EDuration="1.863452333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:30:44.862047812 +0000 UTC m=+656.903922293" watchObservedRunningTime="2026-09-10 14:30:44.863452333 +0000 UTC m=+656.905326815"
Sep 10 14:30:53.944745 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:53.944691    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:53.945174 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:53.944769    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:53.949419 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:53.949395    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:54.889915 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:54.889886    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65df4d6d55-f2ncz"
Sep 10 14:30:57.290751 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:57.290715    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v"]
Sep 10 14:30:57.294411 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:57.294394    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v"
Sep 10 14:30:57.297959 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:57.297939    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 14:30:57.299214 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:57.299197    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 14:30:57.299287 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:57.299261    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hntw6\""
Sep 10 14:30:57.305436 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:57.305411    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v"]
Sep 10 14:30:57.397898 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:57.397861    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28b9b5d5-8cb7-4273-9056-10fd9d404bc6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v\" (UID: \"28b9b5d5-8cb7-4273-9056-10fd9d404bc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v"
Sep 10 14:30:57.398058 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:57.397904    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdrr7\" (UniqueName: \"kubernetes.io/projected/28b9b5d5-8cb7-4273-9056-10fd9d404bc6-kube-api-access-fdrr7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v\" (UID: \"28b9b5d5-8cb7-4273-9056-10fd9d404bc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v"
Sep 10 14:30:57.398058 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:57.397965    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28b9b5d5-8cb7-4273-9056-10fd9d404bc6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v\" (UID: \"28b9b5d5-8cb7-4273-9056-10fd9d404bc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v"
Sep 10 14:30:57.499308 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:57.499247    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28b9b5d5-8cb7-4273-9056-10fd9d404bc6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v\" (UID: \"28b9b5d5-8cb7-4273-9056-10fd9d404bc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v"
Sep 10 14:30:57.499308 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:57.499322    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28b9b5d5-8cb7-4273-9056-10fd9d404bc6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v\" (UID: \"28b9b5d5-8cb7-4273-9056-10fd9d404bc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v"
Sep 10 14:30:57.499545 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:57.499348    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdrr7\" (UniqueName: \"kubernetes.io/projected/28b9b5d5-8cb7-4273-9056-10fd9d404bc6-kube-api-access-fdrr7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v\" (UID: \"28b9b5d5-8cb7-4273-9056-10fd9d404bc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v"
Sep 10 14:30:57.499771 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:57.499746    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28b9b5d5-8cb7-4273-9056-10fd9d404bc6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v\" (UID: \"28b9b5d5-8cb7-4273-9056-10fd9d404bc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v"
Sep 10 14:30:57.499771 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:57.499760    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28b9b5d5-8cb7-4273-9056-10fd9d404bc6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v\" (UID: \"28b9b5d5-8cb7-4273-9056-10fd9d404bc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v"
Sep 10 14:30:57.507722 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:57.507681    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdrr7\" (UniqueName: \"kubernetes.io/projected/28b9b5d5-8cb7-4273-9056-10fd9d404bc6-kube-api-access-fdrr7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v\" (UID: \"28b9b5d5-8cb7-4273-9056-10fd9d404bc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v"
Sep 10 14:30:57.603878 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:57.603790    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v"
Sep 10 14:30:57.873022 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:57.872994    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v"]
Sep 10 14:30:57.875417 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:30:57.875392    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b9b5d5_8cb7_4273_9056_10fd9d404bc6.slice/crio-3d397acc92d3a16acb2dabe3e5137a3c23d404f6d5adb9e06e09d2fb0a0747d3 WatchSource:0}: Error finding container 3d397acc92d3a16acb2dabe3e5137a3c23d404f6d5adb9e06e09d2fb0a0747d3: Status 404 returned error can't find the container with id 3d397acc92d3a16acb2dabe3e5137a3c23d404f6d5adb9e06e09d2fb0a0747d3
Sep 10 14:30:57.877386 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:57.877371    2547 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:30:57.898227 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:30:57.898202    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v" event={"ID":"28b9b5d5-8cb7-4273-9056-10fd9d404bc6","Type":"ContainerStarted","Data":"3d397acc92d3a16acb2dabe3e5137a3c23d404f6d5adb9e06e09d2fb0a0747d3"}
Sep 10 14:31:04.930919 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:04.930884    2547 generic.go:356] "Generic (PLEG): container finished" podID="28b9b5d5-8cb7-4273-9056-10fd9d404bc6" containerID="3e9346918c4796c51d02d7d8fc894b421b9f318f11c567747cc93ae67891d967" exitCode=0
Sep 10 14:31:04.931289 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:04.930948    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v" event={"ID":"28b9b5d5-8cb7-4273-9056-10fd9d404bc6","Type":"ContainerDied","Data":"3e9346918c4796c51d02d7d8fc894b421b9f318f11c567747cc93ae67891d967"}
Sep 10 14:31:07.944789 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:07.944754    2547 generic.go:356] "Generic (PLEG): container finished" podID="28b9b5d5-8cb7-4273-9056-10fd9d404bc6" containerID="2acd53f17f0dc0f5d30b9832b20b38f67497cb42a3eef6aecdf381c30e06cba7" exitCode=0
Sep 10 14:31:07.945179 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:07.944839    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v" event={"ID":"28b9b5d5-8cb7-4273-9056-10fd9d404bc6","Type":"ContainerDied","Data":"2acd53f17f0dc0f5d30b9832b20b38f67497cb42a3eef6aecdf381c30e06cba7"}
Sep 10 14:31:14.978295 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:14.978206    2547 generic.go:356] "Generic (PLEG): container finished" podID="28b9b5d5-8cb7-4273-9056-10fd9d404bc6" containerID="5ad94faaf69c758f4fb35efd26893a82f5996a27780fe3884c6307bd1511b989" exitCode=0
Sep 10 14:31:14.978295 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:14.978276    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v" event={"ID":"28b9b5d5-8cb7-4273-9056-10fd9d404bc6","Type":"ContainerDied","Data":"5ad94faaf69c758f4fb35efd26893a82f5996a27780fe3884c6307bd1511b989"}
Sep 10 14:31:16.149404 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:16.149371    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v"
Sep 10 14:31:16.255893 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:16.255798    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28b9b5d5-8cb7-4273-9056-10fd9d404bc6-util\") pod \"28b9b5d5-8cb7-4273-9056-10fd9d404bc6\" (UID: \"28b9b5d5-8cb7-4273-9056-10fd9d404bc6\") "
Sep 10 14:31:16.255893 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:16.255848    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdrr7\" (UniqueName: \"kubernetes.io/projected/28b9b5d5-8cb7-4273-9056-10fd9d404bc6-kube-api-access-fdrr7\") pod \"28b9b5d5-8cb7-4273-9056-10fd9d404bc6\" (UID: \"28b9b5d5-8cb7-4273-9056-10fd9d404bc6\") "
Sep 10 14:31:16.256078 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:16.255940    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28b9b5d5-8cb7-4273-9056-10fd9d404bc6-bundle\") pod \"28b9b5d5-8cb7-4273-9056-10fd9d404bc6\" (UID: \"28b9b5d5-8cb7-4273-9056-10fd9d404bc6\") "
Sep 10 14:31:16.256552 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:16.256524    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b9b5d5-8cb7-4273-9056-10fd9d404bc6-bundle" (OuterVolumeSpecName: "bundle") pod "28b9b5d5-8cb7-4273-9056-10fd9d404bc6" (UID: "28b9b5d5-8cb7-4273-9056-10fd9d404bc6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:31:16.258054 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:16.258029    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b9b5d5-8cb7-4273-9056-10fd9d404bc6-kube-api-access-fdrr7" (OuterVolumeSpecName: "kube-api-access-fdrr7") pod "28b9b5d5-8cb7-4273-9056-10fd9d404bc6" (UID: "28b9b5d5-8cb7-4273-9056-10fd9d404bc6"). InnerVolumeSpecName "kube-api-access-fdrr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:31:16.261375 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:16.261349    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b9b5d5-8cb7-4273-9056-10fd9d404bc6-util" (OuterVolumeSpecName: "util") pod "28b9b5d5-8cb7-4273-9056-10fd9d404bc6" (UID: "28b9b5d5-8cb7-4273-9056-10fd9d404bc6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:31:16.356684 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:16.356649    2547 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28b9b5d5-8cb7-4273-9056-10fd9d404bc6-util\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:31:16.356684 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:16.356681    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdrr7\" (UniqueName: \"kubernetes.io/projected/28b9b5d5-8cb7-4273-9056-10fd9d404bc6-kube-api-access-fdrr7\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:31:16.356684 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:16.356691    2547 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28b9b5d5-8cb7-4273-9056-10fd9d404bc6-bundle\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:31:16.988129 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:16.988097    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v"
Sep 10 14:31:16.988129 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:16.988109    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404852v" event={"ID":"28b9b5d5-8cb7-4273-9056-10fd9d404bc6","Type":"ContainerDied","Data":"3d397acc92d3a16acb2dabe3e5137a3c23d404f6d5adb9e06e09d2fb0a0747d3"}
Sep 10 14:31:16.988394 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:16.988150    2547 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d397acc92d3a16acb2dabe3e5137a3c23d404f6d5adb9e06e09d2fb0a0747d3"
Sep 10 14:31:25.190451 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.190416    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hwskw"]
Sep 10 14:31:25.190823 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.190690    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28b9b5d5-8cb7-4273-9056-10fd9d404bc6" containerName="extract"
Sep 10 14:31:25.190823 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.190721    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="28b9b5d5-8cb7-4273-9056-10fd9d404bc6" containerName="extract"
Sep 10 14:31:25.190823 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.190735    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28b9b5d5-8cb7-4273-9056-10fd9d404bc6" containerName="util"
Sep 10 14:31:25.190823 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.190741    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="28b9b5d5-8cb7-4273-9056-10fd9d404bc6" containerName="util"
Sep 10 14:31:25.190823 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.190746    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28b9b5d5-8cb7-4273-9056-10fd9d404bc6" containerName="pull"
Sep 10 14:31:25.190823 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.190752    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="28b9b5d5-8cb7-4273-9056-10fd9d404bc6" containerName="pull"
Sep 10 14:31:25.190823 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.190805    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="28b9b5d5-8cb7-4273-9056-10fd9d404bc6" containerName="extract"
Sep 10 14:31:25.192620 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.192603    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-hwskw"
Sep 10 14:31:25.195051 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.195031    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 14:31:25.195177 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.195073    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 14:31:25.206133 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.206108    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hwskw"]
Sep 10 14:31:25.322891 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.322846    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8f902f8-1676-434c-b16e-d44c0f82bd39-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hwskw\" (UID: \"e8f902f8-1676-434c-b16e-d44c0f82bd39\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hwskw"
Sep 10 14:31:25.323069 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.322916    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m77h\" (UniqueName: \"kubernetes.io/projected/e8f902f8-1676-434c-b16e-d44c0f82bd39-kube-api-access-5m77h\") pod \"cert-manager-webhook-6486ffd997-hwskw\" (UID: \"e8f902f8-1676-434c-b16e-d44c0f82bd39\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hwskw"
Sep 10 14:31:25.423817 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.423778    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8f902f8-1676-434c-b16e-d44c0f82bd39-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hwskw\" (UID: \"e8f902f8-1676-434c-b16e-d44c0f82bd39\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hwskw"
Sep 10 14:31:25.423980 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.423837    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m77h\" (UniqueName: \"kubernetes.io/projected/e8f902f8-1676-434c-b16e-d44c0f82bd39-kube-api-access-5m77h\") pod \"cert-manager-webhook-6486ffd997-hwskw\" (UID: \"e8f902f8-1676-434c-b16e-d44c0f82bd39\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hwskw"
Sep 10 14:31:25.438847 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.438805    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8f902f8-1676-434c-b16e-d44c0f82bd39-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hwskw\" (UID: \"e8f902f8-1676-434c-b16e-d44c0f82bd39\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hwskw"
Sep 10 14:31:25.438986 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.438914    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m77h\" (UniqueName: \"kubernetes.io/projected/e8f902f8-1676-434c-b16e-d44c0f82bd39-kube-api-access-5m77h\") pod \"cert-manager-webhook-6486ffd997-hwskw\" (UID: \"e8f902f8-1676-434c-b16e-d44c0f82bd39\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hwskw"
Sep 10 14:31:25.521571 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.521480    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-hwskw"
Sep 10 14:31:25.809293 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.809265    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq"]
Sep 10 14:31:25.811596 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:31:25.811571    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8f902f8_1676_434c_b16e_d44c0f82bd39.slice/crio-b6352e4692bf557eed175ca17259adb656dc5c155ac6929095a27fbb02b8a787 WatchSource:0}: Error finding container b6352e4692bf557eed175ca17259adb656dc5c155ac6929095a27fbb02b8a787: Status 404 returned error can't find the container with id b6352e4692bf557eed175ca17259adb656dc5c155ac6929095a27fbb02b8a787
Sep 10 14:31:25.812231 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.812215    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hwskw"]
Sep 10 14:31:25.812326 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.812315    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq"
Sep 10 14:31:25.816280 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.816255    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hntw6\""
Sep 10 14:31:25.816854 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.816833    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 14:31:25.817289 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.817272    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 14:31:25.826573 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.826552    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq"]
Sep 10 14:31:25.927138 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.927101    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eee510fd-baa1-41d0-9f41-d0b49f49e358-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq\" (UID: \"eee510fd-baa1-41d0-9f41-d0b49f49e358\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq"
Sep 10 14:31:25.927310 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.927146    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eee510fd-baa1-41d0-9f41-d0b49f49e358-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq\" (UID: \"eee510fd-baa1-41d0-9f41-d0b49f49e358\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq"
Sep 10 14:31:25.927310 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:25.927169    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6zj6\" (UniqueName: \"kubernetes.io/projected/eee510fd-baa1-41d0-9f41-d0b49f49e358-kube-api-access-k6zj6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq\" (UID: \"eee510fd-baa1-41d0-9f41-d0b49f49e358\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq"
Sep 10 14:31:26.022959 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:26.022919    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-hwskw" event={"ID":"e8f902f8-1676-434c-b16e-d44c0f82bd39","Type":"ContainerStarted","Data":"b6352e4692bf557eed175ca17259adb656dc5c155ac6929095a27fbb02b8a787"}
Sep 10 14:31:26.028305 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:26.028280    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eee510fd-baa1-41d0-9f41-d0b49f49e358-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq\" (UID: \"eee510fd-baa1-41d0-9f41-d0b49f49e358\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq"
Sep 10 14:31:26.028417 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:26.028330    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eee510fd-baa1-41d0-9f41-d0b49f49e358-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq\" (UID: \"eee510fd-baa1-41d0-9f41-d0b49f49e358\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq"
Sep 10 14:31:26.028417 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:26.028359    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6zj6\" (UniqueName: \"kubernetes.io/projected/eee510fd-baa1-41d0-9f41-d0b49f49e358-kube-api-access-k6zj6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq\" (UID: \"eee510fd-baa1-41d0-9f41-d0b49f49e358\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq"
Sep 10 14:31:26.028668 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:26.028647    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eee510fd-baa1-41d0-9f41-d0b49f49e358-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq\" (UID: \"eee510fd-baa1-41d0-9f41-d0b49f49e358\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq"
Sep 10 14:31:26.028760 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:26.028681    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eee510fd-baa1-41d0-9f41-d0b49f49e358-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq\" (UID: \"eee510fd-baa1-41d0-9f41-d0b49f49e358\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq"
Sep 10 14:31:26.047735 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:26.047692    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6zj6\" (UniqueName: \"kubernetes.io/projected/eee510fd-baa1-41d0-9f41-d0b49f49e358-kube-api-access-k6zj6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq\" (UID: \"eee510fd-baa1-41d0-9f41-d0b49f49e358\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq"
Sep 10 14:31:26.125829 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:26.125749    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq"
Sep 10 14:31:26.415283 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:31:26.415248    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee510fd_baa1_41d0_9f41_d0b49f49e358.slice/crio-d55c15f9923df760184934197e2dc8f87396cc7f00c60739ba94831c42b6a824 WatchSource:0}: Error finding container d55c15f9923df760184934197e2dc8f87396cc7f00c60739ba94831c42b6a824: Status 404 returned error can't find the container with id d55c15f9923df760184934197e2dc8f87396cc7f00c60739ba94831c42b6a824
Sep 10 14:31:26.420143 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:26.420123    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq"]
Sep 10 14:31:27.028456 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:27.028412    2547 generic.go:356] "Generic (PLEG): container finished" podID="eee510fd-baa1-41d0-9f41-d0b49f49e358" containerID="5191b533451baa03725471c53ba5596538e4b373da68329f7f97952be1b9f364" exitCode=0
Sep 10 14:31:27.028635 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:27.028506    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq" event={"ID":"eee510fd-baa1-41d0-9f41-d0b49f49e358","Type":"ContainerDied","Data":"5191b533451baa03725471c53ba5596538e4b373da68329f7f97952be1b9f364"}
Sep 10 14:31:27.028724 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:27.028637    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq" event={"ID":"eee510fd-baa1-41d0-9f41-d0b49f49e358","Type":"ContainerStarted","Data":"d55c15f9923df760184934197e2dc8f87396cc7f00c60739ba94831c42b6a824"}
Sep 10 14:31:28.055429 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:28.055389    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-wrgj5"]
Sep 10 14:31:28.057964 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:28.057942    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wrgj5"
Sep 10 14:31:28.060513 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:28.060489    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-wvl9z\""
Sep 10 14:31:28.068917 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:28.068893    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-wrgj5"]
Sep 10 14:31:28.146515 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:28.146472    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a1bb2160-9b9d-4cbe-beaf-4cd94fb09959-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-wrgj5\" (UID: \"a1bb2160-9b9d-4cbe-beaf-4cd94fb09959\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wrgj5"
Sep 10 14:31:28.146681 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:28.146528    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tvpl\" (UniqueName: \"kubernetes.io/projected/a1bb2160-9b9d-4cbe-beaf-4cd94fb09959-kube-api-access-6tvpl\") pod \"cert-manager-cainjector-559cdd8c87-wrgj5\" (UID: \"a1bb2160-9b9d-4cbe-beaf-4cd94fb09959\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wrgj5"
Sep 10 14:31:28.247952 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:28.247911    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a1bb2160-9b9d-4cbe-beaf-4cd94fb09959-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-wrgj5\" (UID: \"a1bb2160-9b9d-4cbe-beaf-4cd94fb09959\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wrgj5"
Sep 10 14:31:28.247952 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:28.247955    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tvpl\" (UniqueName: \"kubernetes.io/projected/a1bb2160-9b9d-4cbe-beaf-4cd94fb09959-kube-api-access-6tvpl\") pod \"cert-manager-cainjector-559cdd8c87-wrgj5\" (UID: \"a1bb2160-9b9d-4cbe-beaf-4cd94fb09959\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wrgj5"
Sep 10 14:31:28.256359 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:28.256323    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a1bb2160-9b9d-4cbe-beaf-4cd94fb09959-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-wrgj5\" (UID: \"a1bb2160-9b9d-4cbe-beaf-4cd94fb09959\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wrgj5"
Sep 10 14:31:28.256359 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:28.256355    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tvpl\" (UniqueName: \"kubernetes.io/projected/a1bb2160-9b9d-4cbe-beaf-4cd94fb09959-kube-api-access-6tvpl\") pod \"cert-manager-cainjector-559cdd8c87-wrgj5\" (UID: \"a1bb2160-9b9d-4cbe-beaf-4cd94fb09959\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wrgj5"
Sep 10 14:31:28.368449 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:28.368426    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wrgj5"
Sep 10 14:31:28.638374 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:28.638296    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-wrgj5"]
Sep 10 14:31:28.642061 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:31:28.642031    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1bb2160_9b9d_4cbe_beaf_4cd94fb09959.slice/crio-e4331019efeba02bd76f3296639156bf9f5c002861ce25caf81979ea63b23efb WatchSource:0}: Error finding container e4331019efeba02bd76f3296639156bf9f5c002861ce25caf81979ea63b23efb: Status 404 returned error can't find the container with id e4331019efeba02bd76f3296639156bf9f5c002861ce25caf81979ea63b23efb
Sep 10 14:31:29.039775 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:29.039734    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-hwskw" event={"ID":"e8f902f8-1676-434c-b16e-d44c0f82bd39","Type":"ContainerStarted","Data":"97dec24bcdeeaacdfb9bb690b7a6d1df91c6deb13185d53810284f39c2f2b380"}
Sep 10 14:31:29.039969 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:29.039810    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-hwskw"
Sep 10 14:31:29.041504 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:29.041474    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wrgj5" event={"ID":"a1bb2160-9b9d-4cbe-beaf-4cd94fb09959","Type":"ContainerStarted","Data":"9bb78d932830e8ba7a17d8ad7af1742a0531d39a75e52b49a90661e9581c8a71"}
Sep 10 14:31:29.041504 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:29.041512    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wrgj5" event={"ID":"a1bb2160-9b9d-4cbe-beaf-4cd94fb09959","Type":"ContainerStarted","Data":"e4331019efeba02bd76f3296639156bf9f5c002861ce25caf81979ea63b23efb"}
Sep 10 14:31:29.058625 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:29.058575    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-hwskw" podStartSLOduration=1.526950653 podStartE2EDuration="4.058560418s" totalImagesPullingTime="2.531609765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:31:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:31:29.057174173 +0000 UTC m=+701.099048655" watchObservedRunningTime="2026-09-10 14:31:29.058560418 +0000 UTC m=+701.100434898"
Sep 10 14:31:29.076581 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:29.076523    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wrgj5" podStartSLOduration=1.076508501 podStartE2EDuration="1.076508501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:31:29.074910993 +0000 UTC m=+701.116785479" watchObservedRunningTime="2026-09-10 14:31:29.076508501 +0000 UTC m=+701.118382981"
Sep 10 14:31:30.047160 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:30.047122    2547 generic.go:356] "Generic (PLEG): container finished" podID="eee510fd-baa1-41d0-9f41-d0b49f49e358" containerID="636a9542124e385a9cb5ffdad4b5cf4e0e20d3ae1ab0c96eb2a8581a85e4ed53" exitCode=0
Sep 10 14:31:30.047326 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:30.047203    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq" event={"ID":"eee510fd-baa1-41d0-9f41-d0b49f49e358","Type":"ContainerDied","Data":"636a9542124e385a9cb5ffdad4b5cf4e0e20d3ae1ab0c96eb2a8581a85e4ed53"}
Sep 10 14:31:31.057786 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:31.057748    2547 generic.go:356] "Generic (PLEG): container finished" podID="eee510fd-baa1-41d0-9f41-d0b49f49e358" containerID="9e5595d213d34b0efe55455fd85ad7056c9d00f9ca72444e66694369d93bad17" exitCode=0
Sep 10 14:31:31.058157 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:31.057827    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq" event={"ID":"eee510fd-baa1-41d0-9f41-d0b49f49e358","Type":"ContainerDied","Data":"9e5595d213d34b0efe55455fd85ad7056c9d00f9ca72444e66694369d93bad17"}
Sep 10 14:31:32.196747 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:32.196723    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq"
Sep 10 14:31:32.278916 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:32.278882    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eee510fd-baa1-41d0-9f41-d0b49f49e358-bundle\") pod \"eee510fd-baa1-41d0-9f41-d0b49f49e358\" (UID: \"eee510fd-baa1-41d0-9f41-d0b49f49e358\") "
Sep 10 14:31:32.279056 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:32.278925    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6zj6\" (UniqueName: \"kubernetes.io/projected/eee510fd-baa1-41d0-9f41-d0b49f49e358-kube-api-access-k6zj6\") pod \"eee510fd-baa1-41d0-9f41-d0b49f49e358\" (UID: \"eee510fd-baa1-41d0-9f41-d0b49f49e358\") "
Sep 10 14:31:32.279056 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:32.279003    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eee510fd-baa1-41d0-9f41-d0b49f49e358-util\") pod \"eee510fd-baa1-41d0-9f41-d0b49f49e358\" (UID: \"eee510fd-baa1-41d0-9f41-d0b49f49e358\") "
Sep 10 14:31:32.279373 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:32.279345    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee510fd-baa1-41d0-9f41-d0b49f49e358-bundle" (OuterVolumeSpecName: "bundle") pod "eee510fd-baa1-41d0-9f41-d0b49f49e358" (UID: "eee510fd-baa1-41d0-9f41-d0b49f49e358"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:31:32.281080 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:32.281057    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee510fd-baa1-41d0-9f41-d0b49f49e358-kube-api-access-k6zj6" (OuterVolumeSpecName: "kube-api-access-k6zj6") pod "eee510fd-baa1-41d0-9f41-d0b49f49e358" (UID: "eee510fd-baa1-41d0-9f41-d0b49f49e358"). InnerVolumeSpecName "kube-api-access-k6zj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:31:32.283446 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:32.283420    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee510fd-baa1-41d0-9f41-d0b49f49e358-util" (OuterVolumeSpecName: "util") pod "eee510fd-baa1-41d0-9f41-d0b49f49e358" (UID: "eee510fd-baa1-41d0-9f41-d0b49f49e358"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:31:32.380132 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:32.380051    2547 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eee510fd-baa1-41d0-9f41-d0b49f49e358-util\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:31:32.380132 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:32.380083    2547 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eee510fd-baa1-41d0-9f41-d0b49f49e358-bundle\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:31:32.380132 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:32.380106    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6zj6\" (UniqueName: \"kubernetes.io/projected/eee510fd-baa1-41d0-9f41-d0b49f49e358-kube-api-access-k6zj6\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:31:33.067354 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:33.067324    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq"
Sep 10 14:31:33.067354 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:33.067343    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flknjq" event={"ID":"eee510fd-baa1-41d0-9f41-d0b49f49e358","Type":"ContainerDied","Data":"d55c15f9923df760184934197e2dc8f87396cc7f00c60739ba94831c42b6a824"}
Sep 10 14:31:33.067549 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:33.067384    2547 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d55c15f9923df760184934197e2dc8f87396cc7f00c60739ba94831c42b6a824"
Sep 10 14:31:35.050328 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:35.050296    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-hwskw"
Sep 10 14:31:42.511302 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:42.511223    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz"]
Sep 10 14:31:42.511625 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:42.511509    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eee510fd-baa1-41d0-9f41-d0b49f49e358" containerName="util"
Sep 10 14:31:42.511625 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:42.511519    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="eee510fd-baa1-41d0-9f41-d0b49f49e358" containerName="util"
Sep 10 14:31:42.511625 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:42.511531    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eee510fd-baa1-41d0-9f41-d0b49f49e358" containerName="pull"
Sep 10 14:31:42.511625 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:42.511537    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="eee510fd-baa1-41d0-9f41-d0b49f49e358" containerName="pull"
Sep 10 14:31:42.511625 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:42.511549    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eee510fd-baa1-41d0-9f41-d0b49f49e358" containerName="extract"
Sep 10 14:31:42.511625 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:42.511555    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="eee510fd-baa1-41d0-9f41-d0b49f49e358" containerName="extract"
Sep 10 14:31:42.511625 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:42.511603    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="eee510fd-baa1-41d0-9f41-d0b49f49e358" containerName="extract"
Sep 10 14:31:42.515303 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:42.515285    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz"
Sep 10 14:31:42.518653 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:42.518621    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 14:31:42.519722 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:42.519683    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 14:31:42.519847 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:42.519829    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hntw6\""
Sep 10 14:31:42.524357 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:42.524332    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz"]
Sep 10 14:31:42.553742 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:42.553717    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17c18556-f38c-4106-849c-e5d3af291c2d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz\" (UID: \"17c18556-f38c-4106-849c-e5d3af291c2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz"
Sep 10 14:31:42.553861 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:42.553747    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkgks\" (UniqueName: \"kubernetes.io/projected/17c18556-f38c-4106-849c-e5d3af291c2d-kube-api-access-nkgks\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz\" (UID: \"17c18556-f38c-4106-849c-e5d3af291c2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz"
Sep 10 14:31:42.553861 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:42.553776    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17c18556-f38c-4106-849c-e5d3af291c2d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz\" (UID: \"17c18556-f38c-4106-849c-e5d3af291c2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz"
Sep 10 14:31:42.654229 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:42.654189    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17c18556-f38c-4106-849c-e5d3af291c2d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz\" (UID: \"17c18556-f38c-4106-849c-e5d3af291c2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz"
Sep 10 14:31:42.654229 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:42.654230    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkgks\" (UniqueName: \"kubernetes.io/projected/17c18556-f38c-4106-849c-e5d3af291c2d-kube-api-access-nkgks\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz\" (UID: \"17c18556-f38c-4106-849c-e5d3af291c2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz"
Sep 10 14:31:42.654438 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:42.654333    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17c18556-f38c-4106-849c-e5d3af291c2d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz\" (UID: \"17c18556-f38c-4106-849c-e5d3af291c2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz"
Sep 10 14:31:42.654596 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:42.654571    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17c18556-f38c-4106-849c-e5d3af291c2d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz\" (UID: \"17c18556-f38c-4106-849c-e5d3af291c2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz"
Sep 10 14:31:42.654648 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:42.654605    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17c18556-f38c-4106-849c-e5d3af291c2d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz\" (UID: \"17c18556-f38c-4106-849c-e5d3af291c2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz"
Sep 10 14:31:42.662932 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:42.662905    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkgks\" (UniqueName: \"kubernetes.io/projected/17c18556-f38c-4106-849c-e5d3af291c2d-kube-api-access-nkgks\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz\" (UID: \"17c18556-f38c-4106-849c-e5d3af291c2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz"
Sep 10 14:31:42.824881 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:42.824784    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz"
Sep 10 14:31:43.093629 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:43.093547    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz"]
Sep 10 14:31:43.096627 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:31:43.096597    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c18556_f38c_4106_849c_e5d3af291c2d.slice/crio-ec7064224739c92c9d2e51a326be4ad2c3dadb89ab28f0e4c3a12baa29a56349 WatchSource:0}: Error finding container ec7064224739c92c9d2e51a326be4ad2c3dadb89ab28f0e4c3a12baa29a56349: Status 404 returned error can't find the container with id ec7064224739c92c9d2e51a326be4ad2c3dadb89ab28f0e4c3a12baa29a56349
Sep 10 14:31:43.110077 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:43.110050    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz" event={"ID":"17c18556-f38c-4106-849c-e5d3af291c2d","Type":"ContainerStarted","Data":"ec7064224739c92c9d2e51a326be4ad2c3dadb89ab28f0e4c3a12baa29a56349"}
Sep 10 14:31:44.115672 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:44.115583    2547 generic.go:356] "Generic (PLEG): container finished" podID="17c18556-f38c-4106-849c-e5d3af291c2d" containerID="2e5d7420b65299e707f5511b56729c15a111e22fec8f9b1919cf2e9d662e6b23" exitCode=0
Sep 10 14:31:44.115672 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:44.115642    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz" event={"ID":"17c18556-f38c-4106-849c-e5d3af291c2d","Type":"ContainerDied","Data":"2e5d7420b65299e707f5511b56729c15a111e22fec8f9b1919cf2e9d662e6b23"}
Sep 10 14:31:45.121364 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:45.121277    2547 generic.go:356] "Generic (PLEG): container finished" podID="17c18556-f38c-4106-849c-e5d3af291c2d" containerID="fed150f4ec111207e84c7750275f2a8eb0d36289ce9c73507d874946d83acab2" exitCode=0
Sep 10 14:31:45.121364 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:45.121323    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz" event={"ID":"17c18556-f38c-4106-849c-e5d3af291c2d","Type":"ContainerDied","Data":"fed150f4ec111207e84c7750275f2a8eb0d36289ce9c73507d874946d83acab2"}
Sep 10 14:31:46.127291 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:46.127252    2547 generic.go:356] "Generic (PLEG): container finished" podID="17c18556-f38c-4106-849c-e5d3af291c2d" containerID="b8bb45b697a00b777a4f026e920297355b75d7dcc90dc65ef9e44fc9152a9175" exitCode=0
Sep 10 14:31:46.127656 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:46.127343    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz" event={"ID":"17c18556-f38c-4106-849c-e5d3af291c2d","Type":"ContainerDied","Data":"b8bb45b697a00b777a4f026e920297355b75d7dcc90dc65ef9e44fc9152a9175"}
Sep 10 14:31:47.275419 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:47.275384    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz"
Sep 10 14:31:47.286382 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:47.286356    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17c18556-f38c-4106-849c-e5d3af291c2d-util\") pod \"17c18556-f38c-4106-849c-e5d3af291c2d\" (UID: \"17c18556-f38c-4106-849c-e5d3af291c2d\") "
Sep 10 14:31:47.286512 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:47.286396    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17c18556-f38c-4106-849c-e5d3af291c2d-bundle\") pod \"17c18556-f38c-4106-849c-e5d3af291c2d\" (UID: \"17c18556-f38c-4106-849c-e5d3af291c2d\") "
Sep 10 14:31:47.286512 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:47.286436    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkgks\" (UniqueName: \"kubernetes.io/projected/17c18556-f38c-4106-849c-e5d3af291c2d-kube-api-access-nkgks\") pod \"17c18556-f38c-4106-849c-e5d3af291c2d\" (UID: \"17c18556-f38c-4106-849c-e5d3af291c2d\") "
Sep 10 14:31:47.287117 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:47.287082    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c18556-f38c-4106-849c-e5d3af291c2d-bundle" (OuterVolumeSpecName: "bundle") pod "17c18556-f38c-4106-849c-e5d3af291c2d" (UID: "17c18556-f38c-4106-849c-e5d3af291c2d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:31:47.288504 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:47.288478    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c18556-f38c-4106-849c-e5d3af291c2d-kube-api-access-nkgks" (OuterVolumeSpecName: "kube-api-access-nkgks") pod "17c18556-f38c-4106-849c-e5d3af291c2d" (UID: "17c18556-f38c-4106-849c-e5d3af291c2d"). InnerVolumeSpecName "kube-api-access-nkgks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:31:47.292299 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:47.292255    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c18556-f38c-4106-849c-e5d3af291c2d-util" (OuterVolumeSpecName: "util") pod "17c18556-f38c-4106-849c-e5d3af291c2d" (UID: "17c18556-f38c-4106-849c-e5d3af291c2d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:31:47.387294 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:47.387209    2547 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17c18556-f38c-4106-849c-e5d3af291c2d-util\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:31:47.387294 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:47.387240    2547 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17c18556-f38c-4106-849c-e5d3af291c2d-bundle\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:31:47.387294 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:47.387262    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkgks\" (UniqueName: \"kubernetes.io/projected/17c18556-f38c-4106-849c-e5d3af291c2d-kube-api-access-nkgks\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:31:48.136399 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:48.136366    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz" event={"ID":"17c18556-f38c-4106-849c-e5d3af291c2d","Type":"ContainerDied","Data":"ec7064224739c92c9d2e51a326be4ad2c3dadb89ab28f0e4c3a12baa29a56349"}
Sep 10 14:31:48.136565 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:48.136407    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj58zz"
Sep 10 14:31:48.136565 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:48.136418    2547 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec7064224739c92c9d2e51a326be4ad2c3dadb89ab28f0e4c3a12baa29a56349"
Sep 10 14:31:53.530432 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:53.530397    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5"]
Sep 10 14:31:53.530940 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:53.530858    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17c18556-f38c-4106-849c-e5d3af291c2d" containerName="extract"
Sep 10 14:31:53.530940 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:53.530877    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="17c18556-f38c-4106-849c-e5d3af291c2d" containerName="extract"
Sep 10 14:31:53.530940 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:53.530910    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17c18556-f38c-4106-849c-e5d3af291c2d" containerName="util"
Sep 10 14:31:53.530940 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:53.530919    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="17c18556-f38c-4106-849c-e5d3af291c2d" containerName="util"
Sep 10 14:31:53.530940 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:53.530933    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17c18556-f38c-4106-849c-e5d3af291c2d" containerName="pull"
Sep 10 14:31:53.530940 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:53.530942    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="17c18556-f38c-4106-849c-e5d3af291c2d" containerName="pull"
Sep 10 14:31:53.531242 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:53.531017    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="17c18556-f38c-4106-849c-e5d3af291c2d" containerName="extract"
Sep 10 14:31:53.533644 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:53.533617    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5"
Sep 10 14:31:53.537674 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:53.537654    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 14:31:53.538784 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:53.538755    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hntw6\""
Sep 10 14:31:53.538784 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:53.538783    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 14:31:53.543021 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:53.542997    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5"]
Sep 10 14:31:53.635212 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:53.635175    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44kqm\" (UniqueName: \"kubernetes.io/projected/b0f4e922-d9f9-4be9-b318-7edb6d0ba51e-kube-api-access-44kqm\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5\" (UID: \"b0f4e922-d9f9-4be9-b318-7edb6d0ba51e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5"
Sep 10 14:31:53.635212 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:53.635215    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0f4e922-d9f9-4be9-b318-7edb6d0ba51e-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5\" (UID: \"b0f4e922-d9f9-4be9-b318-7edb6d0ba51e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5"
Sep 10 14:31:53.635418 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:53.635282    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0f4e922-d9f9-4be9-b318-7edb6d0ba51e-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5\" (UID: \"b0f4e922-d9f9-4be9-b318-7edb6d0ba51e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5"
Sep 10 14:31:53.736027 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:53.735991    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44kqm\" (UniqueName: \"kubernetes.io/projected/b0f4e922-d9f9-4be9-b318-7edb6d0ba51e-kube-api-access-44kqm\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5\" (UID: \"b0f4e922-d9f9-4be9-b318-7edb6d0ba51e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5"
Sep 10 14:31:53.736027 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:53.736031    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0f4e922-d9f9-4be9-b318-7edb6d0ba51e-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5\" (UID: \"b0f4e922-d9f9-4be9-b318-7edb6d0ba51e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5"
Sep 10 14:31:53.736235 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:53.736074    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0f4e922-d9f9-4be9-b318-7edb6d0ba51e-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5\" (UID: \"b0f4e922-d9f9-4be9-b318-7edb6d0ba51e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5"
Sep 10 14:31:53.736494 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:53.736472    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0f4e922-d9f9-4be9-b318-7edb6d0ba51e-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5\" (UID: \"b0f4e922-d9f9-4be9-b318-7edb6d0ba51e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5"
Sep 10 14:31:53.736538 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:53.736500    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0f4e922-d9f9-4be9-b318-7edb6d0ba51e-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5\" (UID: \"b0f4e922-d9f9-4be9-b318-7edb6d0ba51e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5"
Sep 10 14:31:53.745032 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:53.745010    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44kqm\" (UniqueName: \"kubernetes.io/projected/b0f4e922-d9f9-4be9-b318-7edb6d0ba51e-kube-api-access-44kqm\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5\" (UID: \"b0f4e922-d9f9-4be9-b318-7edb6d0ba51e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5"
Sep 10 14:31:53.842635 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:53.842547    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5"
Sep 10 14:31:54.112219 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:54.112029    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5"]
Sep 10 14:31:54.114969 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:31:54.114941    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f4e922_d9f9_4be9_b318_7edb6d0ba51e.slice/crio-36dfb730b65521eee78a9b725782ffcd6aac27858ba77127741365b3db67002e WatchSource:0}: Error finding container 36dfb730b65521eee78a9b725782ffcd6aac27858ba77127741365b3db67002e: Status 404 returned error can't find the container with id 36dfb730b65521eee78a9b725782ffcd6aac27858ba77127741365b3db67002e
Sep 10 14:31:54.162078 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:54.162052    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5" event={"ID":"b0f4e922-d9f9-4be9-b318-7edb6d0ba51e","Type":"ContainerStarted","Data":"36dfb730b65521eee78a9b725782ffcd6aac27858ba77127741365b3db67002e"}
Sep 10 14:31:55.167629 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.167598    2547 generic.go:356] "Generic (PLEG): container finished" podID="b0f4e922-d9f9-4be9-b318-7edb6d0ba51e" containerID="1ea3c897b3bb274ed2510fdd66bbd4d0f0fbf9986b5c28529c734dbac1d4273a" exitCode=0
Sep 10 14:31:55.167999 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.167650    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5" event={"ID":"b0f4e922-d9f9-4be9-b318-7edb6d0ba51e","Type":"ContainerDied","Data":"1ea3c897b3bb274ed2510fdd66bbd4d0f0fbf9986b5c28529c734dbac1d4273a"}
Sep 10 14:31:55.348727 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.348679    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7"]
Sep 10 14:31:55.351019 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.350996    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7"
Sep 10 14:31:55.354398 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.354376    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 14:31:55.354890 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.354862    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 14:31:55.354990 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.354955    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 14:31:55.355073 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.355056    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6jvst\""
Sep 10 14:31:55.355264 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.355248    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 14:31:55.378877 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.378850    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7"]
Sep 10 14:31:55.449235 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.449146    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6snwp\" (UniqueName: \"kubernetes.io/projected/2e2642b2-98c3-43ff-b568-4a0932c2eb6f-kube-api-access-6snwp\") pod \"opendatahub-operator-controller-manager-755bd7f47-vrmm7\" (UID: \"2e2642b2-98c3-43ff-b568-4a0932c2eb6f\") " pod="opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7"
Sep 10 14:31:55.449405 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.449290    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2e2642b2-98c3-43ff-b568-4a0932c2eb6f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-755bd7f47-vrmm7\" (UID: \"2e2642b2-98c3-43ff-b568-4a0932c2eb6f\") " pod="opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7"
Sep 10 14:31:55.449405 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.449346    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e2642b2-98c3-43ff-b568-4a0932c2eb6f-webhook-cert\") pod \"opendatahub-operator-controller-manager-755bd7f47-vrmm7\" (UID: \"2e2642b2-98c3-43ff-b568-4a0932c2eb6f\") " pod="opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7"
Sep 10 14:31:55.449523 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.449441    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e2642b2-98c3-43ff-b568-4a0932c2eb6f-tmp\") pod \"opendatahub-operator-controller-manager-755bd7f47-vrmm7\" (UID: \"2e2642b2-98c3-43ff-b568-4a0932c2eb6f\") " pod="opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7"
Sep 10 14:31:55.449564 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.449532    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2e2642b2-98c3-43ff-b568-4a0932c2eb6f-manifests\") pod \"opendatahub-operator-controller-manager-755bd7f47-vrmm7\" (UID: \"2e2642b2-98c3-43ff-b568-4a0932c2eb6f\") " pod="opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7"
Sep 10 14:31:55.550691 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.550650    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e2642b2-98c3-43ff-b568-4a0932c2eb6f-tmp\") pod \"opendatahub-operator-controller-manager-755bd7f47-vrmm7\" (UID: \"2e2642b2-98c3-43ff-b568-4a0932c2eb6f\") " pod="opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7"
Sep 10 14:31:55.550691 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.550721    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2e2642b2-98c3-43ff-b568-4a0932c2eb6f-manifests\") pod \"opendatahub-operator-controller-manager-755bd7f47-vrmm7\" (UID: \"2e2642b2-98c3-43ff-b568-4a0932c2eb6f\") " pod="opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7"
Sep 10 14:31:55.550968 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.550750    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6snwp\" (UniqueName: \"kubernetes.io/projected/2e2642b2-98c3-43ff-b568-4a0932c2eb6f-kube-api-access-6snwp\") pod \"opendatahub-operator-controller-manager-755bd7f47-vrmm7\" (UID: \"2e2642b2-98c3-43ff-b568-4a0932c2eb6f\") " pod="opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7"
Sep 10 14:31:55.550968 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.550797    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2e2642b2-98c3-43ff-b568-4a0932c2eb6f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-755bd7f47-vrmm7\" (UID: \"2e2642b2-98c3-43ff-b568-4a0932c2eb6f\") " pod="opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7"
Sep 10 14:31:55.550968 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.550817    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e2642b2-98c3-43ff-b568-4a0932c2eb6f-webhook-cert\") pod \"opendatahub-operator-controller-manager-755bd7f47-vrmm7\" (UID: \"2e2642b2-98c3-43ff-b568-4a0932c2eb6f\") " pod="opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7"
Sep 10 14:31:55.551170 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.551147    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2e2642b2-98c3-43ff-b568-4a0932c2eb6f-manifests\") pod \"opendatahub-operator-controller-manager-755bd7f47-vrmm7\" (UID: \"2e2642b2-98c3-43ff-b568-4a0932c2eb6f\") " pod="opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7"
Sep 10 14:31:55.552900 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.552879    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e2642b2-98c3-43ff-b568-4a0932c2eb6f-tmp\") pod \"opendatahub-operator-controller-manager-755bd7f47-vrmm7\" (UID: \"2e2642b2-98c3-43ff-b568-4a0932c2eb6f\") " pod="opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7"
Sep 10 14:31:55.553144 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.553125    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e2642b2-98c3-43ff-b568-4a0932c2eb6f-webhook-cert\") pod \"opendatahub-operator-controller-manager-755bd7f47-vrmm7\" (UID: \"2e2642b2-98c3-43ff-b568-4a0932c2eb6f\") " pod="opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7"
Sep 10 14:31:55.553206 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.553142    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2e2642b2-98c3-43ff-b568-4a0932c2eb6f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-755bd7f47-vrmm7\" (UID: \"2e2642b2-98c3-43ff-b568-4a0932c2eb6f\") " pod="opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7"
Sep 10 14:31:55.559066 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.559035    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6snwp\" (UniqueName: \"kubernetes.io/projected/2e2642b2-98c3-43ff-b568-4a0932c2eb6f-kube-api-access-6snwp\") pod \"opendatahub-operator-controller-manager-755bd7f47-vrmm7\" (UID: \"2e2642b2-98c3-43ff-b568-4a0932c2eb6f\") " pod="opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7"
Sep 10 14:31:55.661061 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.661019    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7"
Sep 10 14:31:55.941429 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:55.941398    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7"]
Sep 10 14:31:55.944296 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:31:55.944260    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2642b2_98c3_43ff_b568_4a0932c2eb6f.slice/crio-08cb789dc1de5324ff9f243e31b36b197ad71d2bdf5072101554f63db93fe9e8 WatchSource:0}: Error finding container 08cb789dc1de5324ff9f243e31b36b197ad71d2bdf5072101554f63db93fe9e8: Status 404 returned error can't find the container with id 08cb789dc1de5324ff9f243e31b36b197ad71d2bdf5072101554f63db93fe9e8
Sep 10 14:31:56.174250 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:56.174215    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7" event={"ID":"2e2642b2-98c3-43ff-b568-4a0932c2eb6f","Type":"ContainerStarted","Data":"08cb789dc1de5324ff9f243e31b36b197ad71d2bdf5072101554f63db93fe9e8"}
Sep 10 14:31:56.175921 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:56.175892    2547 generic.go:356] "Generic (PLEG): container finished" podID="b0f4e922-d9f9-4be9-b318-7edb6d0ba51e" containerID="4d8952e1d1c8180d39114ee8d5301871c93d2a983937547937bfd441eccdb4b5" exitCode=0
Sep 10 14:31:56.176056 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:56.175977    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5" event={"ID":"b0f4e922-d9f9-4be9-b318-7edb6d0ba51e","Type":"ContainerDied","Data":"4d8952e1d1c8180d39114ee8d5301871c93d2a983937547937bfd441eccdb4b5"}
Sep 10 14:31:57.185194 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:57.185154    2547 generic.go:356] "Generic (PLEG): container finished" podID="b0f4e922-d9f9-4be9-b318-7edb6d0ba51e" containerID="d86fe263efe3b941e2bde0e2819a75c68a0411d6b1d2685be7641f579ddeca32" exitCode=0
Sep 10 14:31:57.185635 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:57.185237    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5" event={"ID":"b0f4e922-d9f9-4be9-b318-7edb6d0ba51e","Type":"ContainerDied","Data":"d86fe263efe3b941e2bde0e2819a75c68a0411d6b1d2685be7641f579ddeca32"}
Sep 10 14:31:59.066933 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:59.066902    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5"
Sep 10 14:31:59.184642 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:59.184618    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0f4e922-d9f9-4be9-b318-7edb6d0ba51e-bundle\") pod \"b0f4e922-d9f9-4be9-b318-7edb6d0ba51e\" (UID: \"b0f4e922-d9f9-4be9-b318-7edb6d0ba51e\") "
Sep 10 14:31:59.184790 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:59.184649    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0f4e922-d9f9-4be9-b318-7edb6d0ba51e-util\") pod \"b0f4e922-d9f9-4be9-b318-7edb6d0ba51e\" (UID: \"b0f4e922-d9f9-4be9-b318-7edb6d0ba51e\") "
Sep 10 14:31:59.184790 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:59.184675    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44kqm\" (UniqueName: \"kubernetes.io/projected/b0f4e922-d9f9-4be9-b318-7edb6d0ba51e-kube-api-access-44kqm\") pod \"b0f4e922-d9f9-4be9-b318-7edb6d0ba51e\" (UID: \"b0f4e922-d9f9-4be9-b318-7edb6d0ba51e\") "
Sep 10 14:31:59.185321 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:59.185299    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f4e922-d9f9-4be9-b318-7edb6d0ba51e-bundle" (OuterVolumeSpecName: "bundle") pod "b0f4e922-d9f9-4be9-b318-7edb6d0ba51e" (UID: "b0f4e922-d9f9-4be9-b318-7edb6d0ba51e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:31:59.186561 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:59.186540    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f4e922-d9f9-4be9-b318-7edb6d0ba51e-kube-api-access-44kqm" (OuterVolumeSpecName: "kube-api-access-44kqm") pod "b0f4e922-d9f9-4be9-b318-7edb6d0ba51e" (UID: "b0f4e922-d9f9-4be9-b318-7edb6d0ba51e"). InnerVolumeSpecName "kube-api-access-44kqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:31:59.189647 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:59.189626    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f4e922-d9f9-4be9-b318-7edb6d0ba51e-util" (OuterVolumeSpecName: "util") pod "b0f4e922-d9f9-4be9-b318-7edb6d0ba51e" (UID: "b0f4e922-d9f9-4be9-b318-7edb6d0ba51e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:31:59.195752 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:59.195726    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7" event={"ID":"2e2642b2-98c3-43ff-b568-4a0932c2eb6f","Type":"ContainerStarted","Data":"238e298712c34fc61b39be2b112dc2b1b2044b93468bc75bfae694c4984f34a7"}
Sep 10 14:31:59.197454 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:59.197433    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5" event={"ID":"b0f4e922-d9f9-4be9-b318-7edb6d0ba51e","Type":"ContainerDied","Data":"36dfb730b65521eee78a9b725782ffcd6aac27858ba77127741365b3db67002e"}
Sep 10 14:31:59.197454 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:59.197441    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0t5vk5"
Sep 10 14:31:59.197563 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:59.197465    2547 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36dfb730b65521eee78a9b725782ffcd6aac27858ba77127741365b3db67002e"
Sep 10 14:31:59.285454 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:59.285420    2547 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0f4e922-d9f9-4be9-b318-7edb6d0ba51e-bundle\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:31:59.285454 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:59.285447    2547 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0f4e922-d9f9-4be9-b318-7edb6d0ba51e-util\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:31:59.285454 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:31:59.285456    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44kqm\" (UniqueName: \"kubernetes.io/projected/b0f4e922-d9f9-4be9-b318-7edb6d0ba51e-kube-api-access-44kqm\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:32:01.208351 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:01.208311    2547 generic.go:356] "Generic (PLEG): container finished" podID="2e2642b2-98c3-43ff-b568-4a0932c2eb6f" containerID="238e298712c34fc61b39be2b112dc2b1b2044b93468bc75bfae694c4984f34a7" exitCode=0
Sep 10 14:32:01.208834 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:01.208390    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7" event={"ID":"2e2642b2-98c3-43ff-b568-4a0932c2eb6f","Type":"ContainerDied","Data":"238e298712c34fc61b39be2b112dc2b1b2044b93468bc75bfae694c4984f34a7"}
Sep 10 14:32:02.214729 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:02.214674    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7" event={"ID":"2e2642b2-98c3-43ff-b568-4a0932c2eb6f","Type":"ContainerStarted","Data":"bf392e8b06f05c97769bc8a42e40511838419b13497dc52d7212e23476b1f5b1"}
Sep 10 14:32:02.215106 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:02.214907    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7"
Sep 10 14:32:02.239252 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:02.239201    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7" podStartSLOduration=2.50334124 podStartE2EDuration="7.23918745s" totalImagesPullingTime="3.652161699s" totalInitContainerRuntime="1.083684511s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:31:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:32:02.237726308 +0000 UTC m=+734.279600788" watchObservedRunningTime="2026-09-10 14:32:02.23918745 +0000 UTC m=+734.281061930"
Sep 10 14:32:13.221570 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:13.221538    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-755bd7f47-vrmm7"
Sep 10 14:32:16.029476 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:16.029439    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6f69648866-w6nq7"]
Sep 10 14:32:16.029893 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:16.029779    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0f4e922-d9f9-4be9-b318-7edb6d0ba51e" containerName="util"
Sep 10 14:32:16.029893 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:16.029792    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0f4e922-d9f9-4be9-b318-7edb6d0ba51e" containerName="util"
Sep 10 14:32:16.029893 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:16.029804    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0f4e922-d9f9-4be9-b318-7edb6d0ba51e" containerName="pull"
Sep 10 14:32:16.029893 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:16.029810    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0f4e922-d9f9-4be9-b318-7edb6d0ba51e" containerName="pull"
Sep 10 14:32:16.029893 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:16.029819    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0f4e922-d9f9-4be9-b318-7edb6d0ba51e" containerName="extract"
Sep 10 14:32:16.029893 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:16.029825    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0f4e922-d9f9-4be9-b318-7edb6d0ba51e" containerName="extract"
Sep 10 14:32:16.029893 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:16.029874    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0f4e922-d9f9-4be9-b318-7edb6d0ba51e" containerName="extract"
Sep 10 14:32:16.032951 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:16.032935    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f69648866-w6nq7"
Sep 10 14:32:16.035430 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:16.035407    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vgprh\""
Sep 10 14:32:16.035706 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:16.035673    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 14:32:16.036425 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:16.036409    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 14:32:16.044288 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:16.044259    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f69648866-w6nq7"]
Sep 10 14:32:16.122871 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:16.122839    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2891fd11-751e-473b-aba3-ff3570a980bc-tmp\") pod \"kube-auth-proxy-6f69648866-w6nq7\" (UID: \"2891fd11-751e-473b-aba3-ff3570a980bc\") " pod="openshift-ingress/kube-auth-proxy-6f69648866-w6nq7"
Sep 10 14:32:16.123055 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:16.122886    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5dxm\" (UniqueName: \"kubernetes.io/projected/2891fd11-751e-473b-aba3-ff3570a980bc-kube-api-access-s5dxm\") pod \"kube-auth-proxy-6f69648866-w6nq7\" (UID: \"2891fd11-751e-473b-aba3-ff3570a980bc\") " pod="openshift-ingress/kube-auth-proxy-6f69648866-w6nq7"
Sep 10 14:32:16.123055 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:16.122931    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2891fd11-751e-473b-aba3-ff3570a980bc-tls-certs\") pod \"kube-auth-proxy-6f69648866-w6nq7\" (UID: \"2891fd11-751e-473b-aba3-ff3570a980bc\") " pod="openshift-ingress/kube-auth-proxy-6f69648866-w6nq7"
Sep 10 14:32:16.224219 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:16.224182    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2891fd11-751e-473b-aba3-ff3570a980bc-tmp\") pod \"kube-auth-proxy-6f69648866-w6nq7\" (UID: \"2891fd11-751e-473b-aba3-ff3570a980bc\") " pod="openshift-ingress/kube-auth-proxy-6f69648866-w6nq7"
Sep 10 14:32:16.224367 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:16.224227    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5dxm\" (UniqueName: \"kubernetes.io/projected/2891fd11-751e-473b-aba3-ff3570a980bc-kube-api-access-s5dxm\") pod \"kube-auth-proxy-6f69648866-w6nq7\" (UID: \"2891fd11-751e-473b-aba3-ff3570a980bc\") " pod="openshift-ingress/kube-auth-proxy-6f69648866-w6nq7"
Sep 10 14:32:16.224367 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:16.224276    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2891fd11-751e-473b-aba3-ff3570a980bc-tls-certs\") pod \"kube-auth-proxy-6f69648866-w6nq7\" (UID: \"2891fd11-751e-473b-aba3-ff3570a980bc\") " pod="openshift-ingress/kube-auth-proxy-6f69648866-w6nq7"
Sep 10 14:32:16.226481 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:16.226455    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2891fd11-751e-473b-aba3-ff3570a980bc-tmp\") pod \"kube-auth-proxy-6f69648866-w6nq7\" (UID: \"2891fd11-751e-473b-aba3-ff3570a980bc\") " pod="openshift-ingress/kube-auth-proxy-6f69648866-w6nq7"
Sep 10 14:32:16.226597 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:16.226580    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2891fd11-751e-473b-aba3-ff3570a980bc-tls-certs\") pod \"kube-auth-proxy-6f69648866-w6nq7\" (UID: \"2891fd11-751e-473b-aba3-ff3570a980bc\") " pod="openshift-ingress/kube-auth-proxy-6f69648866-w6nq7"
Sep 10 14:32:16.233324 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:16.233297    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5dxm\" (UniqueName: \"kubernetes.io/projected/2891fd11-751e-473b-aba3-ff3570a980bc-kube-api-access-s5dxm\") pod \"kube-auth-proxy-6f69648866-w6nq7\" (UID: \"2891fd11-751e-473b-aba3-ff3570a980bc\") " pod="openshift-ingress/kube-auth-proxy-6f69648866-w6nq7"
Sep 10 14:32:16.342268 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:16.342189    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f69648866-w6nq7"
Sep 10 14:32:16.615375 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:16.615346    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f69648866-w6nq7"]
Sep 10 14:32:16.617381 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:32:16.617354    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2891fd11_751e_473b_aba3_ff3570a980bc.slice/crio-40a5338f5b1afea7efbc5ae0034f834bf2e2fbe83c26031931a3beb2f8d60d45 WatchSource:0}: Error finding container 40a5338f5b1afea7efbc5ae0034f834bf2e2fbe83c26031931a3beb2f8d60d45: Status 404 returned error can't find the container with id 40a5338f5b1afea7efbc5ae0034f834bf2e2fbe83c26031931a3beb2f8d60d45
Sep 10 14:32:17.284431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:17.284377    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f69648866-w6nq7" event={"ID":"2891fd11-751e-473b-aba3-ff3570a980bc","Type":"ContainerStarted","Data":"40a5338f5b1afea7efbc5ae0034f834bf2e2fbe83c26031931a3beb2f8d60d45"}
Sep 10 14:32:20.300165 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:20.300128    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f69648866-w6nq7" event={"ID":"2891fd11-751e-473b-aba3-ff3570a980bc","Type":"ContainerStarted","Data":"467c43e11b6b4201752387c6fd474b60c9f3b971a21fd0aa0865f378530facec"}
Sep 10 14:32:20.318192 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:20.318139    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6f69648866-w6nq7" podStartSLOduration=1.494087916 podStartE2EDuration="4.318120578s" totalImagesPullingTime="2.824032662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:32:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:32:20.317446083 +0000 UTC m=+752.359320564" watchObservedRunningTime="2026-09-10 14:32:20.318120578 +0000 UTC m=+752.359995059"
Sep 10 14:32:23.041676 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.041642    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"]
Sep 10 14:32:23.045056 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.045038    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.047794 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.047771    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 14:32:23.047896 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.047786    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-pjsx8\""
Sep 10 14:32:23.063047 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.063023    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"]
Sep 10 14:32:23.183270 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.183241    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.183270 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.183274    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.183477 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.183306    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.183477 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.183344    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.183477 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.183386    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.183477 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.183411    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.183477 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.183431    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.183477 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.183448    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4cbs\" (UniqueName: \"kubernetes.io/projected/9c611680-f2cc-46f2-baec-d51cfd58e9e7-kube-api-access-h4cbs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.183477 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.183466    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.284747 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.284692    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.284940 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.284759    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.284940 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.284797    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.284940 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.284827    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.285089 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.284938    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.285089 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.284989    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4cbs\" (UniqueName: \"kubernetes.io/projected/9c611680-f2cc-46f2-baec-d51cfd58e9e7-kube-api-access-h4cbs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.285089 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.285015    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.285089 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.285065    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.285286 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.285098    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.285286 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.285195    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.285286 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.285275    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.285445 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.285394    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.285513 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.285492    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.285603 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.285581    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.287247 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.287226    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.287408 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.287388    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.296680 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.296624    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.298195 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.298174    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4cbs\" (UniqueName: \"kubernetes.io/projected/9c611680-f2cc-46f2-baec-d51cfd58e9e7-kube-api-access-h4cbs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpklcg\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.354972 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.354937    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:23.630745 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:23.630619    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"]
Sep 10 14:32:23.632913 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:32:23.632885    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c611680_f2cc_46f2_baec_d51cfd58e9e7.slice/crio-0379f8f53b0c0163b8aab96e6a6e2719da2e3755a3bce2be64f12d78ada34cbf WatchSource:0}: Error finding container 0379f8f53b0c0163b8aab96e6a6e2719da2e3755a3bce2be64f12d78ada34cbf: Status 404 returned error can't find the container with id 0379f8f53b0c0163b8aab96e6a6e2719da2e3755a3bce2be64f12d78ada34cbf
Sep 10 14:32:24.320097 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:24.320054    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg" event={"ID":"9c611680-f2cc-46f2-baec-d51cfd58e9e7","Type":"ContainerStarted","Data":"0379f8f53b0c0163b8aab96e6a6e2719da2e3755a3bce2be64f12d78ada34cbf"}
Sep 10 14:32:25.960961 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:25.960918    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-gcjhf"]
Sep 10 14:32:25.968097 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:25.968074    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-gcjhf"
Sep 10 14:32:25.971433 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:25.970880    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-64stz\""
Sep 10 14:32:25.971433 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:25.971196    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 10 14:32:25.972148 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:25.972125    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-gcjhf"]
Sep 10 14:32:26.110950 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:26.110916    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66trh\" (UniqueName: \"kubernetes.io/projected/447019ec-0133-4f4b-a530-b4fd4d693651-kube-api-access-66trh\") pod \"odh-model-controller-7998c67dc4-gcjhf\" (UID: \"447019ec-0133-4f4b-a530-b4fd4d693651\") " pod="opendatahub/odh-model-controller-7998c67dc4-gcjhf"
Sep 10 14:32:26.111113 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:26.110988    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/447019ec-0133-4f4b-a530-b4fd4d693651-cert\") pod \"odh-model-controller-7998c67dc4-gcjhf\" (UID: \"447019ec-0133-4f4b-a530-b4fd4d693651\") " pod="opendatahub/odh-model-controller-7998c67dc4-gcjhf"
Sep 10 14:32:26.212395 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:26.212299    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/447019ec-0133-4f4b-a530-b4fd4d693651-cert\") pod \"odh-model-controller-7998c67dc4-gcjhf\" (UID: \"447019ec-0133-4f4b-a530-b4fd4d693651\") " pod="opendatahub/odh-model-controller-7998c67dc4-gcjhf"
Sep 10 14:32:26.212563 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:26.212418    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66trh\" (UniqueName: \"kubernetes.io/projected/447019ec-0133-4f4b-a530-b4fd4d693651-kube-api-access-66trh\") pod \"odh-model-controller-7998c67dc4-gcjhf\" (UID: \"447019ec-0133-4f4b-a530-b4fd4d693651\") " pod="opendatahub/odh-model-controller-7998c67dc4-gcjhf"
Sep 10 14:32:26.215096 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:26.215070    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/447019ec-0133-4f4b-a530-b4fd4d693651-cert\") pod \"odh-model-controller-7998c67dc4-gcjhf\" (UID: \"447019ec-0133-4f4b-a530-b4fd4d693651\") " pod="opendatahub/odh-model-controller-7998c67dc4-gcjhf"
Sep 10 14:32:26.222465 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:26.222440    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66trh\" (UniqueName: \"kubernetes.io/projected/447019ec-0133-4f4b-a530-b4fd4d693651-kube-api-access-66trh\") pod \"odh-model-controller-7998c67dc4-gcjhf\" (UID: \"447019ec-0133-4f4b-a530-b4fd4d693651\") " pod="opendatahub/odh-model-controller-7998c67dc4-gcjhf"
Sep 10 14:32:26.281528 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:26.281488    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-gcjhf"
Sep 10 14:32:26.615893 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:26.615853    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:32:26.616001 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:26.615925    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:32:26.616001 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:26.615954    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:32:26.662053 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:26.662034    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-hrq7m"]
Sep 10 14:32:26.666673 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:26.666647    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-hrq7m"
Sep 10 14:32:26.669428 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:26.669409    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-4z9kj\""
Sep 10 14:32:26.669516 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:26.669415    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 10 14:32:26.679375 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:26.678766    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-hrq7m"]
Sep 10 14:32:26.809138 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:26.809110    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-gcjhf"]
Sep 10 14:32:26.811038 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:32:26.811007    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod447019ec_0133_4f4b_a530_b4fd4d693651.slice/crio-bcbd7dbd8b1bea90f60c647ec929dc1711330a80e4d1a713309486d32652970e WatchSource:0}: Error finding container bcbd7dbd8b1bea90f60c647ec929dc1711330a80e4d1a713309486d32652970e: Status 404 returned error can't find the container with id bcbd7dbd8b1bea90f60c647ec929dc1711330a80e4d1a713309486d32652970e
Sep 10 14:32:26.818373 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:26.818345    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9db08a11-abdf-415a-8f96-2c363bb4d1e9-tls-certs\") pod \"model-serving-api-54d89bd5fc-hrq7m\" (UID: \"9db08a11-abdf-415a-8f96-2c363bb4d1e9\") " pod="opendatahub/model-serving-api-54d89bd5fc-hrq7m"
Sep 10 14:32:26.818483 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:26.818466    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b229n\" (UniqueName: \"kubernetes.io/projected/9db08a11-abdf-415a-8f96-2c363bb4d1e9-kube-api-access-b229n\") pod \"model-serving-api-54d89bd5fc-hrq7m\" (UID: \"9db08a11-abdf-415a-8f96-2c363bb4d1e9\") " pod="opendatahub/model-serving-api-54d89bd5fc-hrq7m"
Sep 10 14:32:26.919347 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:26.919313    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b229n\" (UniqueName: \"kubernetes.io/projected/9db08a11-abdf-415a-8f96-2c363bb4d1e9-kube-api-access-b229n\") pod \"model-serving-api-54d89bd5fc-hrq7m\" (UID: \"9db08a11-abdf-415a-8f96-2c363bb4d1e9\") " pod="opendatahub/model-serving-api-54d89bd5fc-hrq7m"
Sep 10 14:32:26.919477 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:26.919365    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9db08a11-abdf-415a-8f96-2c363bb4d1e9-tls-certs\") pod \"model-serving-api-54d89bd5fc-hrq7m\" (UID: \"9db08a11-abdf-415a-8f96-2c363bb4d1e9\") " pod="opendatahub/model-serving-api-54d89bd5fc-hrq7m"
Sep 10 14:32:26.919477 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:32:26.919467    2547 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 10 14:32:26.919560 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:32:26.919531    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9db08a11-abdf-415a-8f96-2c363bb4d1e9-tls-certs podName:9db08a11-abdf-415a-8f96-2c363bb4d1e9 nodeName:}" failed. No retries permitted until 2026-09-10 14:32:27.419511923 +0000 UTC m=+759.461386386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9db08a11-abdf-415a-8f96-2c363bb4d1e9-tls-certs") pod "model-serving-api-54d89bd5fc-hrq7m" (UID: "9db08a11-abdf-415a-8f96-2c363bb4d1e9") : secret "model-serving-api-tls" not found
Sep 10 14:32:26.927772 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:26.927748    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b229n\" (UniqueName: \"kubernetes.io/projected/9db08a11-abdf-415a-8f96-2c363bb4d1e9-kube-api-access-b229n\") pod \"model-serving-api-54d89bd5fc-hrq7m\" (UID: \"9db08a11-abdf-415a-8f96-2c363bb4d1e9\") " pod="opendatahub/model-serving-api-54d89bd5fc-hrq7m"
Sep 10 14:32:27.336863 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:27.336826    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg" event={"ID":"9c611680-f2cc-46f2-baec-d51cfd58e9e7","Type":"ContainerStarted","Data":"9403c5703e1b871f1c80b0cb48aa462b474e9619add92e42bf618dd4deb13923"}
Sep 10 14:32:27.338081 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:27.338054    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-gcjhf" event={"ID":"447019ec-0133-4f4b-a530-b4fd4d693651","Type":"ContainerStarted","Data":"bcbd7dbd8b1bea90f60c647ec929dc1711330a80e4d1a713309486d32652970e"}
Sep 10 14:32:27.355167 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:27.355123    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:27.356479 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:27.356444    2547 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body=
Sep 10 14:32:27.356624 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:27.356495    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg" podUID="9c611680-f2cc-46f2-baec-d51cfd58e9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Sep 10 14:32:27.357068 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:27.357022    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg" podStartSLOduration=1.376413826 podStartE2EDuration="4.357007971s" totalImagesPullingTime="2.980594145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:32:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:32:27.355096785 +0000 UTC m=+759.396971266" watchObservedRunningTime="2026-09-10 14:32:27.357007971 +0000 UTC m=+759.398882452"
Sep 10 14:32:27.422906 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:27.422872    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9db08a11-abdf-415a-8f96-2c363bb4d1e9-tls-certs\") pod \"model-serving-api-54d89bd5fc-hrq7m\" (UID: \"9db08a11-abdf-415a-8f96-2c363bb4d1e9\") " pod="opendatahub/model-serving-api-54d89bd5fc-hrq7m"
Sep 10 14:32:27.425401 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:27.425375    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9db08a11-abdf-415a-8f96-2c363bb4d1e9-tls-certs\") pod \"model-serving-api-54d89bd5fc-hrq7m\" (UID: \"9db08a11-abdf-415a-8f96-2c363bb4d1e9\") " pod="opendatahub/model-serving-api-54d89bd5fc-hrq7m"
Sep 10 14:32:27.577997 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:27.577965    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-hrq7m"
Sep 10 14:32:27.853808 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:27.853777    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-hrq7m"]
Sep 10 14:32:28.345549 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:28.345507    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-hrq7m" event={"ID":"9db08a11-abdf-415a-8f96-2c363bb4d1e9","Type":"ContainerStarted","Data":"505b33c1cd50054bfaad8a7b620ae5ff95b720f1970db5cc8a196f69a840d488"}
Sep 10 14:32:28.356156 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:28.356131    2547 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body=
Sep 10 14:32:28.356308 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:28.356184    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg" podUID="9c611680-f2cc-46f2-baec-d51cfd58e9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Sep 10 14:32:28.604801 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:28.604353    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"]
Sep 10 14:32:29.355727 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:29.355673    2547 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body=
Sep 10 14:32:29.356157 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:29.355744    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg" podUID="9c611680-f2cc-46f2-baec-d51cfd58e9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Sep 10 14:32:30.355252 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:30.355216    2547 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body=
Sep 10 14:32:30.355358 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:30.355270    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg" podUID="9c611680-f2cc-46f2-baec-d51cfd58e9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Sep 10 14:32:30.356195 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:30.356165    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg" podUID="9c611680-f2cc-46f2-baec-d51cfd58e9e7" containerName="istio-proxy" containerID="cri-o://9403c5703e1b871f1c80b0cb48aa462b474e9619add92e42bf618dd4deb13923" gracePeriod=30
Sep 10 14:32:31.361939 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:31.361895    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-gcjhf" event={"ID":"447019ec-0133-4f4b-a530-b4fd4d693651","Type":"ContainerStarted","Data":"5201c912c0dab599fe8e424e720f94286397a590415b1bbeefffe76971912cf3"}
Sep 10 14:32:31.362494 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:31.362022    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-gcjhf"
Sep 10 14:32:31.363389 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:31.363366    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-hrq7m" event={"ID":"9db08a11-abdf-415a-8f96-2c363bb4d1e9","Type":"ContainerStarted","Data":"69e95cdb4247f5d1395ad17c5210a77cde48963dbf37bf5123ca2db66e31ccf3"}
Sep 10 14:32:31.363497 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:31.363417    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-hrq7m"
Sep 10 14:32:31.383149 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:31.383102    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-gcjhf" podStartSLOduration=2.855392885 podStartE2EDuration="6.383087402s" totalImagesPullingTime="3.527694517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:32:31.381976671 +0000 UTC m=+763.423851179" watchObservedRunningTime="2026-09-10 14:32:31.383087402 +0000 UTC m=+763.424961910"
Sep 10 14:32:31.397900 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:31.397859    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-hrq7m" podStartSLOduration=2.919865229 podStartE2EDuration="5.397844263s" totalImagesPullingTime="2.477979034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:32:31.397283775 +0000 UTC m=+763.439158258" watchObservedRunningTime="2026-09-10 14:32:31.397844263 +0000 UTC m=+763.439718745"
Sep 10 14:32:33.036224 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:33.036189    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8f5db98bf-rjb2s"]
Sep 10 14:32:33.041856 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:33.041836    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8f5db98bf-rjb2s"
Sep 10 14:32:33.045673 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:33.045643    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 14:32:33.045673 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:33.045665    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jr6k8\""
Sep 10 14:32:33.045876 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:33.045716    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 14:32:33.046004 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:33.045981    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 14:32:33.046133 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:33.045985    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 14:32:33.046133 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:33.046028    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:32:33.054386 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:33.054362    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8f5db98bf-rjb2s"]
Sep 10 14:32:33.180859 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:33.180810    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e2c33dd1-671a-4509-9c50-205b60962d1c-manager-config\") pod \"lws-controller-manager-8f5db98bf-rjb2s\" (UID: \"e2c33dd1-671a-4509-9c50-205b60962d1c\") " pod="openshift-lws-operator/lws-controller-manager-8f5db98bf-rjb2s"
Sep 10 14:32:33.181022 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:33.180874    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2c33dd1-671a-4509-9c50-205b60962d1c-cert\") pod \"lws-controller-manager-8f5db98bf-rjb2s\" (UID: \"e2c33dd1-671a-4509-9c50-205b60962d1c\") " pod="openshift-lws-operator/lws-controller-manager-8f5db98bf-rjb2s"
Sep 10 14:32:33.181022 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:33.180892    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2c33dd1-671a-4509-9c50-205b60962d1c-metrics-cert\") pod \"lws-controller-manager-8f5db98bf-rjb2s\" (UID: \"e2c33dd1-671a-4509-9c50-205b60962d1c\") " pod="openshift-lws-operator/lws-controller-manager-8f5db98bf-rjb2s"
Sep 10 14:32:33.181022 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:33.180953    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q9v9\" (UniqueName: \"kubernetes.io/projected/e2c33dd1-671a-4509-9c50-205b60962d1c-kube-api-access-7q9v9\") pod \"lws-controller-manager-8f5db98bf-rjb2s\" (UID: \"e2c33dd1-671a-4509-9c50-205b60962d1c\") " pod="openshift-lws-operator/lws-controller-manager-8f5db98bf-rjb2s"
Sep 10 14:32:33.281909 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:33.281874    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e2c33dd1-671a-4509-9c50-205b60962d1c-manager-config\") pod \"lws-controller-manager-8f5db98bf-rjb2s\" (UID: \"e2c33dd1-671a-4509-9c50-205b60962d1c\") " pod="openshift-lws-operator/lws-controller-manager-8f5db98bf-rjb2s"
Sep 10 14:32:33.281909 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:33.281923    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2c33dd1-671a-4509-9c50-205b60962d1c-cert\") pod \"lws-controller-manager-8f5db98bf-rjb2s\" (UID: \"e2c33dd1-671a-4509-9c50-205b60962d1c\") " pod="openshift-lws-operator/lws-controller-manager-8f5db98bf-rjb2s"
Sep 10 14:32:33.282100 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:33.281939    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2c33dd1-671a-4509-9c50-205b60962d1c-metrics-cert\") pod \"lws-controller-manager-8f5db98bf-rjb2s\" (UID: \"e2c33dd1-671a-4509-9c50-205b60962d1c\") " pod="openshift-lws-operator/lws-controller-manager-8f5db98bf-rjb2s"
Sep 10 14:32:33.282100 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:33.281979    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q9v9\" (UniqueName: \"kubernetes.io/projected/e2c33dd1-671a-4509-9c50-205b60962d1c-kube-api-access-7q9v9\") pod \"lws-controller-manager-8f5db98bf-rjb2s\" (UID: \"e2c33dd1-671a-4509-9c50-205b60962d1c\") " pod="openshift-lws-operator/lws-controller-manager-8f5db98bf-rjb2s"
Sep 10 14:32:33.282625 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:33.282590    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e2c33dd1-671a-4509-9c50-205b60962d1c-manager-config\") pod \"lws-controller-manager-8f5db98bf-rjb2s\" (UID: \"e2c33dd1-671a-4509-9c50-205b60962d1c\") " pod="openshift-lws-operator/lws-controller-manager-8f5db98bf-rjb2s"
Sep 10 14:32:33.284452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:33.284430    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2c33dd1-671a-4509-9c50-205b60962d1c-cert\") pod \"lws-controller-manager-8f5db98bf-rjb2s\" (UID: \"e2c33dd1-671a-4509-9c50-205b60962d1c\") " pod="openshift-lws-operator/lws-controller-manager-8f5db98bf-rjb2s"
Sep 10 14:32:33.284534 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:33.284454    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2c33dd1-671a-4509-9c50-205b60962d1c-metrics-cert\") pod \"lws-controller-manager-8f5db98bf-rjb2s\" (UID: \"e2c33dd1-671a-4509-9c50-205b60962d1c\") " pod="openshift-lws-operator/lws-controller-manager-8f5db98bf-rjb2s"
Sep 10 14:32:33.295831 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:33.295777    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q9v9\" (UniqueName: \"kubernetes.io/projected/e2c33dd1-671a-4509-9c50-205b60962d1c-kube-api-access-7q9v9\") pod \"lws-controller-manager-8f5db98bf-rjb2s\" (UID: \"e2c33dd1-671a-4509-9c50-205b60962d1c\") " pod="openshift-lws-operator/lws-controller-manager-8f5db98bf-rjb2s"
Sep 10 14:32:33.351323 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:33.351283    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8f5db98bf-rjb2s"
Sep 10 14:32:33.625583 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:33.625504    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8f5db98bf-rjb2s"]
Sep 10 14:32:33.627841 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:32:33.627815    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c33dd1_671a_4509_9c50_205b60962d1c.slice/crio-82ddb95b629e9c2c2479b99c1d3c551d06343702ddc8d73118e197d0175efd88 WatchSource:0}: Error finding container 82ddb95b629e9c2c2479b99c1d3c551d06343702ddc8d73118e197d0175efd88: Status 404 returned error can't find the container with id 82ddb95b629e9c2c2479b99c1d3c551d06343702ddc8d73118e197d0175efd88
Sep 10 14:32:34.379737 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:34.379692    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8f5db98bf-rjb2s" event={"ID":"e2c33dd1-671a-4509-9c50-205b60962d1c","Type":"ContainerStarted","Data":"82ddb95b629e9c2c2479b99c1d3c551d06343702ddc8d73118e197d0175efd88"}
Sep 10 14:32:35.597835 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.597809    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:35.703030 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.702993    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-workload-certs\") pod \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") "
Sep 10 14:32:35.703030 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.703032    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4cbs\" (UniqueName: \"kubernetes.io/projected/9c611680-f2cc-46f2-baec-d51cfd58e9e7-kube-api-access-h4cbs\") pod \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") "
Sep 10 14:32:35.703235 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.703059    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-workload-socket\") pod \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") "
Sep 10 14:32:35.703235 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.703086    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istio-token\") pod \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") "
Sep 10 14:32:35.703235 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.703112    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istio-podinfo\") pod \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") "
Sep 10 14:32:35.703235 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.703188    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istio-envoy\") pod \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") "
Sep 10 14:32:35.703437 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.703266    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istio-data\") pod \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") "
Sep 10 14:32:35.703437 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.703289    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istiod-ca-cert\") pod \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") "
Sep 10 14:32:35.703437 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.703334    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-credential-socket\") pod \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\" (UID: \"9c611680-f2cc-46f2-baec-d51cfd58e9e7\") "
Sep 10 14:32:35.703437 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.703347    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9c611680-f2cc-46f2-baec-d51cfd58e9e7" (UID: "9c611680-f2cc-46f2-baec-d51cfd58e9e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:32:35.703437 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.703380    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9c611680-f2cc-46f2-baec-d51cfd58e9e7" (UID: "9c611680-f2cc-46f2-baec-d51cfd58e9e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:32:35.703677 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.703575    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9c611680-f2cc-46f2-baec-d51cfd58e9e7" (UID: "9c611680-f2cc-46f2-baec-d51cfd58e9e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:32:35.703677 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.703590    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istio-data" (OuterVolumeSpecName: "istio-data") pod "9c611680-f2cc-46f2-baec-d51cfd58e9e7" (UID: "9c611680-f2cc-46f2-baec-d51cfd58e9e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:32:35.703677 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.703602    2547 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-workload-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:32:35.703677 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.703624    2547 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-workload-socket\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:32:35.703865 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.703808    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9c611680-f2cc-46f2-baec-d51cfd58e9e7" (UID: "9c611680-f2cc-46f2-baec-d51cfd58e9e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:32:35.705908 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.705887    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istio-token" (OuterVolumeSpecName: "istio-token") pod "9c611680-f2cc-46f2-baec-d51cfd58e9e7" (UID: "9c611680-f2cc-46f2-baec-d51cfd58e9e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:32:35.705980 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.705913    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c611680-f2cc-46f2-baec-d51cfd58e9e7-kube-api-access-h4cbs" (OuterVolumeSpecName: "kube-api-access-h4cbs") pod "9c611680-f2cc-46f2-baec-d51cfd58e9e7" (UID: "9c611680-f2cc-46f2-baec-d51cfd58e9e7"). InnerVolumeSpecName "kube-api-access-h4cbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:32:35.705980 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.705918    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9c611680-f2cc-46f2-baec-d51cfd58e9e7" (UID: "9c611680-f2cc-46f2-baec-d51cfd58e9e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 14:32:35.705980 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.705921    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9c611680-f2cc-46f2-baec-d51cfd58e9e7" (UID: "9c611680-f2cc-46f2-baec-d51cfd58e9e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:32:35.804259 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.804222    2547 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istio-podinfo\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:32:35.804259 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.804260    2547 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istio-envoy\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:32:35.804452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.804281    2547 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istio-data\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:32:35.804452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.804294    2547 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istiod-ca-cert\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:32:35.804452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.804308    2547 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c611680-f2cc-46f2-baec-d51cfd58e9e7-credential-socket\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:32:35.804452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.804320    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4cbs\" (UniqueName: \"kubernetes.io/projected/9c611680-f2cc-46f2-baec-d51cfd58e9e7-kube-api-access-h4cbs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:32:35.804452 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:35.804333    2547 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c611680-f2cc-46f2-baec-d51cfd58e9e7-istio-token\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:32:36.392076 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:36.392035    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8f5db98bf-rjb2s" event={"ID":"e2c33dd1-671a-4509-9c50-205b60962d1c","Type":"ContainerStarted","Data":"d3711ef6b58075360bd13c5c921b7570daf476eb9498e95c0490dd02bd301467"}
Sep 10 14:32:36.392256 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:36.392102    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8f5db98bf-rjb2s"
Sep 10 14:32:36.393353 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:36.393330    2547 generic.go:356] "Generic (PLEG): container finished" podID="9c611680-f2cc-46f2-baec-d51cfd58e9e7" containerID="9403c5703e1b871f1c80b0cb48aa462b474e9619add92e42bf618dd4deb13923" exitCode=0
Sep 10 14:32:36.393437 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:36.393372    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"
Sep 10 14:32:36.393481 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:36.393371    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg" event={"ID":"9c611680-f2cc-46f2-baec-d51cfd58e9e7","Type":"ContainerDied","Data":"9403c5703e1b871f1c80b0cb48aa462b474e9619add92e42bf618dd4deb13923"}
Sep 10 14:32:36.393593 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:36.393574    2547 scope.go:117] "RemoveContainer" containerID="9403c5703e1b871f1c80b0cb48aa462b474e9619add92e42bf618dd4deb13923"
Sep 10 14:32:36.397708 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:36.393514    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg" event={"ID":"9c611680-f2cc-46f2-baec-d51cfd58e9e7","Type":"ContainerDied","Data":"0379f8f53b0c0163b8aab96e6a6e2719da2e3755a3bce2be64f12d78ada34cbf"}
Sep 10 14:32:36.408649 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:36.408628    2547 scope.go:117] "RemoveContainer" containerID="9403c5703e1b871f1c80b0cb48aa462b474e9619add92e42bf618dd4deb13923"
Sep 10 14:32:36.408927 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:32:36.408908    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9403c5703e1b871f1c80b0cb48aa462b474e9619add92e42bf618dd4deb13923\": container with ID starting with 9403c5703e1b871f1c80b0cb48aa462b474e9619add92e42bf618dd4deb13923 not found: ID does not exist" containerID="9403c5703e1b871f1c80b0cb48aa462b474e9619add92e42bf618dd4deb13923"
Sep 10 14:32:36.408990 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:36.408933    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9403c5703e1b871f1c80b0cb48aa462b474e9619add92e42bf618dd4deb13923"} err="failed to get container status \"9403c5703e1b871f1c80b0cb48aa462b474e9619add92e42bf618dd4deb13923\": rpc error: code = NotFound desc = could not find container \"9403c5703e1b871f1c80b0cb48aa462b474e9619add92e42bf618dd4deb13923\": container with ID starting with 9403c5703e1b871f1c80b0cb48aa462b474e9619add92e42bf618dd4deb13923 not found: ID does not exist"
Sep 10 14:32:36.424787 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:36.424738    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8f5db98bf-rjb2s" podStartSLOduration=1.738989717 podStartE2EDuration="3.424725015s" totalImagesPullingTime="1.685735298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:32:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:32:36.423234218 +0000 UTC m=+768.465108699" watchObservedRunningTime="2026-09-10 14:32:36.424725015 +0000 UTC m=+768.466599495"
Sep 10 14:32:36.445509 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:36.445486    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"]
Sep 10 14:32:36.451635 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:36.451609    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpklcg"]
Sep 10 14:32:38.421996 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:38.421961    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c611680-f2cc-46f2-baec-d51cfd58e9e7" path="/var/lib/kubelet/pods/9c611680-f2cc-46f2-baec-d51cfd58e9e7/volumes"
Sep 10 14:32:42.371461 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:42.371421    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-gcjhf"
Sep 10 14:32:42.373323 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:42.373301    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-hrq7m"
Sep 10 14:32:47.402235 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:32:47.402206    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8f5db98bf-rjb2s"
Sep 10 14:33:30.700222 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.700185    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd"]
Sep 10 14:33:30.700744 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.700510    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c611680-f2cc-46f2-baec-d51cfd58e9e7" containerName="istio-proxy"
Sep 10 14:33:30.700744 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.700523    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c611680-f2cc-46f2-baec-d51cfd58e9e7" containerName="istio-proxy"
Sep 10 14:33:30.700744 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.700585    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c611680-f2cc-46f2-baec-d51cfd58e9e7" containerName="istio-proxy"
Sep 10 14:33:30.703779 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.703762    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd"
Sep 10 14:33:30.706564 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.706543    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hntw6\""
Sep 10 14:33:30.706692 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.706548    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 14:33:30.707320 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.707306    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 14:33:30.712502 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.712475    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd"]
Sep 10 14:33:30.805340 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.805305    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m"]
Sep 10 14:33:30.808752 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.808736    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m"
Sep 10 14:33:30.818339 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.818317    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m"]
Sep 10 14:33:30.858374 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.858345    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/791776a6-9a65-4d4b-ad77-2d9260ec70c6-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd\" (UID: \"791776a6-9a65-4d4b-ad77-2d9260ec70c6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd"
Sep 10 14:33:30.858492 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.858395    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/791776a6-9a65-4d4b-ad77-2d9260ec70c6-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd\" (UID: \"791776a6-9a65-4d4b-ad77-2d9260ec70c6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd"
Sep 10 14:33:30.858492 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.858429    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gxl6\" (UniqueName: \"kubernetes.io/projected/791776a6-9a65-4d4b-ad77-2d9260ec70c6-kube-api-access-4gxl6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd\" (UID: \"791776a6-9a65-4d4b-ad77-2d9260ec70c6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd"
Sep 10 14:33:30.897810 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.897781    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg"]
Sep 10 14:33:30.901340 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.901323    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg"
Sep 10 14:33:30.911263 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.911240    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg"]
Sep 10 14:33:30.959603 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.959527    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/791776a6-9a65-4d4b-ad77-2d9260ec70c6-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd\" (UID: \"791776a6-9a65-4d4b-ad77-2d9260ec70c6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd"
Sep 10 14:33:30.959771 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.959685    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gxl6\" (UniqueName: \"kubernetes.io/projected/791776a6-9a65-4d4b-ad77-2d9260ec70c6-kube-api-access-4gxl6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd\" (UID: \"791776a6-9a65-4d4b-ad77-2d9260ec70c6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd"
Sep 10 14:33:30.959771 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.959757    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fe83afe-38ab-4386-9653-e6385a966d37-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m\" (UID: \"1fe83afe-38ab-4386-9653-e6385a966d37\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m"
Sep 10 14:33:30.959901 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.959814    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fe83afe-38ab-4386-9653-e6385a966d37-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m\" (UID: \"1fe83afe-38ab-4386-9653-e6385a966d37\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m"
Sep 10 14:33:30.959901 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.959866    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/791776a6-9a65-4d4b-ad77-2d9260ec70c6-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd\" (UID: \"791776a6-9a65-4d4b-ad77-2d9260ec70c6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd"
Sep 10 14:33:30.960014 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.959908    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/791776a6-9a65-4d4b-ad77-2d9260ec70c6-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd\" (UID: \"791776a6-9a65-4d4b-ad77-2d9260ec70c6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd"
Sep 10 14:33:30.960014 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.959945    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpv4l\" (UniqueName: \"kubernetes.io/projected/1fe83afe-38ab-4386-9653-e6385a966d37-kube-api-access-tpv4l\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m\" (UID: \"1fe83afe-38ab-4386-9653-e6385a966d37\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m"
Sep 10 14:33:30.960179 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.960160    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/791776a6-9a65-4d4b-ad77-2d9260ec70c6-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd\" (UID: \"791776a6-9a65-4d4b-ad77-2d9260ec70c6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd"
Sep 10 14:33:30.968491 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:30.968467    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gxl6\" (UniqueName: \"kubernetes.io/projected/791776a6-9a65-4d4b-ad77-2d9260ec70c6-kube-api-access-4gxl6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd\" (UID: \"791776a6-9a65-4d4b-ad77-2d9260ec70c6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd"
Sep 10 14:33:31.013892 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.013856    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd"
Sep 10 14:33:31.016296 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.016266    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn"]
Sep 10 14:33:31.019840 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.019824    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn"
Sep 10 14:33:31.045400 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.045369    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn"]
Sep 10 14:33:31.062255 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.061013    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b0dcbf8-5244-4e8e-8f09-c33ad474f824-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg\" (UID: \"5b0dcbf8-5244-4e8e-8f09-c33ad474f824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg"
Sep 10 14:33:31.062255 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.061075    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b0dcbf8-5244-4e8e-8f09-c33ad474f824-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg\" (UID: \"5b0dcbf8-5244-4e8e-8f09-c33ad474f824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg"
Sep 10 14:33:31.062255 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.061107    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbjxs\" (UniqueName: \"kubernetes.io/projected/5b0dcbf8-5244-4e8e-8f09-c33ad474f824-kube-api-access-fbjxs\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg\" (UID: \"5b0dcbf8-5244-4e8e-8f09-c33ad474f824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg"
Sep 10 14:33:31.062255 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.061139    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fe83afe-38ab-4386-9653-e6385a966d37-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m\" (UID: \"1fe83afe-38ab-4386-9653-e6385a966d37\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m"
Sep 10 14:33:31.062255 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.061236    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fe83afe-38ab-4386-9653-e6385a966d37-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m\" (UID: \"1fe83afe-38ab-4386-9653-e6385a966d37\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m"
Sep 10 14:33:31.062255 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.061316    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpv4l\" (UniqueName: \"kubernetes.io/projected/1fe83afe-38ab-4386-9653-e6385a966d37-kube-api-access-tpv4l\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m\" (UID: \"1fe83afe-38ab-4386-9653-e6385a966d37\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m"
Sep 10 14:33:31.062255 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.061513    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fe83afe-38ab-4386-9653-e6385a966d37-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m\" (UID: \"1fe83afe-38ab-4386-9653-e6385a966d37\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m"
Sep 10 14:33:31.062255 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.061758    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fe83afe-38ab-4386-9653-e6385a966d37-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m\" (UID: \"1fe83afe-38ab-4386-9653-e6385a966d37\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m"
Sep 10 14:33:31.082163 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.082135    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpv4l\" (UniqueName: \"kubernetes.io/projected/1fe83afe-38ab-4386-9653-e6385a966d37-kube-api-access-tpv4l\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m\" (UID: \"1fe83afe-38ab-4386-9653-e6385a966d37\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m"
Sep 10 14:33:31.117782 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.117755    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m"
Sep 10 14:33:31.162025 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.161994    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dfb8070-bff2-428a-b7d4-d54dfdedfbfa-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn\" (UID: \"0dfb8070-bff2-428a-b7d4-d54dfdedfbfa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn"
Sep 10 14:33:31.162163 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.162044    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b0dcbf8-5244-4e8e-8f09-c33ad474f824-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg\" (UID: \"5b0dcbf8-5244-4e8e-8f09-c33ad474f824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg"
Sep 10 14:33:31.162163 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.162064    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b0dcbf8-5244-4e8e-8f09-c33ad474f824-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg\" (UID: \"5b0dcbf8-5244-4e8e-8f09-c33ad474f824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg"
Sep 10 14:33:31.162163 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.162083    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dfb8070-bff2-428a-b7d4-d54dfdedfbfa-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn\" (UID: \"0dfb8070-bff2-428a-b7d4-d54dfdedfbfa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn"
Sep 10 14:33:31.162163 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.162111    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbjxs\" (UniqueName: \"kubernetes.io/projected/5b0dcbf8-5244-4e8e-8f09-c33ad474f824-kube-api-access-fbjxs\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg\" (UID: \"5b0dcbf8-5244-4e8e-8f09-c33ad474f824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg"
Sep 10 14:33:31.162163 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.162153    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5rpj\" (UniqueName: \"kubernetes.io/projected/0dfb8070-bff2-428a-b7d4-d54dfdedfbfa-kube-api-access-d5rpj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn\" (UID: \"0dfb8070-bff2-428a-b7d4-d54dfdedfbfa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn"
Sep 10 14:33:31.162380 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.162350    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b0dcbf8-5244-4e8e-8f09-c33ad474f824-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg\" (UID: \"5b0dcbf8-5244-4e8e-8f09-c33ad474f824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg"
Sep 10 14:33:31.162419 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.162408    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b0dcbf8-5244-4e8e-8f09-c33ad474f824-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg\" (UID: \"5b0dcbf8-5244-4e8e-8f09-c33ad474f824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg"
Sep 10 14:33:31.171834 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.171791    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbjxs\" (UniqueName: \"kubernetes.io/projected/5b0dcbf8-5244-4e8e-8f09-c33ad474f824-kube-api-access-fbjxs\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg\" (UID: \"5b0dcbf8-5244-4e8e-8f09-c33ad474f824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg"
Sep 10 14:33:31.211182 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.211124    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg"
Sep 10 14:33:31.262829 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.262799    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dfb8070-bff2-428a-b7d4-d54dfdedfbfa-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn\" (UID: \"0dfb8070-bff2-428a-b7d4-d54dfdedfbfa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn"
Sep 10 14:33:31.262926 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.262860    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dfb8070-bff2-428a-b7d4-d54dfdedfbfa-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn\" (UID: \"0dfb8070-bff2-428a-b7d4-d54dfdedfbfa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn"
Sep 10 14:33:31.262926 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.262915    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5rpj\" (UniqueName: \"kubernetes.io/projected/0dfb8070-bff2-428a-b7d4-d54dfdedfbfa-kube-api-access-d5rpj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn\" (UID: \"0dfb8070-bff2-428a-b7d4-d54dfdedfbfa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn"
Sep 10 14:33:31.263359 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.263213    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dfb8070-bff2-428a-b7d4-d54dfdedfbfa-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn\" (UID: \"0dfb8070-bff2-428a-b7d4-d54dfdedfbfa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn"
Sep 10 14:33:31.263359 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.263261    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dfb8070-bff2-428a-b7d4-d54dfdedfbfa-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn\" (UID: \"0dfb8070-bff2-428a-b7d4-d54dfdedfbfa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn"
Sep 10 14:33:31.272397 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.272372    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5rpj\" (UniqueName: \"kubernetes.io/projected/0dfb8070-bff2-428a-b7d4-d54dfdedfbfa-kube-api-access-d5rpj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn\" (UID: \"0dfb8070-bff2-428a-b7d4-d54dfdedfbfa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn"
Sep 10 14:33:31.288592 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.288564    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd"]
Sep 10 14:33:31.291765 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:33:31.291741    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791776a6_9a65_4d4b_ad77_2d9260ec70c6.slice/crio-69a1a6c58e577f7004e4874c87f43b6b32dabee48422b3d3fada0c1c4ffb1170 WatchSource:0}: Error finding container 69a1a6c58e577f7004e4874c87f43b6b32dabee48422b3d3fada0c1c4ffb1170: Status 404 returned error can't find the container with id 69a1a6c58e577f7004e4874c87f43b6b32dabee48422b3d3fada0c1c4ffb1170
Sep 10 14:33:31.341949 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.341929    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn"
Sep 10 14:33:31.389939 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.389902    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m"]
Sep 10 14:33:31.391220 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:33:31.391188    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe83afe_38ab_4386_9653_e6385a966d37.slice/crio-e39f42a436e7a719b2a9ef77c506841ec790caa42fa6fe50b984cb958fdb002b WatchSource:0}: Error finding container e39f42a436e7a719b2a9ef77c506841ec790caa42fa6fe50b984cb958fdb002b: Status 404 returned error can't find the container with id e39f42a436e7a719b2a9ef77c506841ec790caa42fa6fe50b984cb958fdb002b
Sep 10 14:33:31.484518 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.484493    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg"]
Sep 10 14:33:31.486404 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:33:31.486380    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0dcbf8_5244_4e8e_8f09_c33ad474f824.slice/crio-97ba4fd22d67de69ea389bfc5872f326b211467b11d18dc3b3c88e6f99e81239 WatchSource:0}: Error finding container 97ba4fd22d67de69ea389bfc5872f326b211467b11d18dc3b3c88e6f99e81239: Status 404 returned error can't find the container with id 97ba4fd22d67de69ea389bfc5872f326b211467b11d18dc3b3c88e6f99e81239
Sep 10 14:33:31.626354 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.626329    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn"]
Sep 10 14:33:31.628311 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:33:31.628273    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dfb8070_bff2_428a_b7d4_d54dfdedfbfa.slice/crio-fd6a6474c8f72b75edc6a8af3c63db226a8d3775d1b32011e8b5b48ebbfe0b58 WatchSource:0}: Error finding container fd6a6474c8f72b75edc6a8af3c63db226a8d3775d1b32011e8b5b48ebbfe0b58: Status 404 returned error can't find the container with id fd6a6474c8f72b75edc6a8af3c63db226a8d3775d1b32011e8b5b48ebbfe0b58
Sep 10 14:33:31.660609 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.660574    2547 generic.go:356] "Generic (PLEG): container finished" podID="5b0dcbf8-5244-4e8e-8f09-c33ad474f824" containerID="c482a37e80a7613fd6cfd847dc0c7902fda8482f7edd08aab9e3a1eed8046494" exitCode=0
Sep 10 14:33:31.660796 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.660657    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg" event={"ID":"5b0dcbf8-5244-4e8e-8f09-c33ad474f824","Type":"ContainerDied","Data":"c482a37e80a7613fd6cfd847dc0c7902fda8482f7edd08aab9e3a1eed8046494"}
Sep 10 14:33:31.660796 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.660763    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg" event={"ID":"5b0dcbf8-5244-4e8e-8f09-c33ad474f824","Type":"ContainerStarted","Data":"97ba4fd22d67de69ea389bfc5872f326b211467b11d18dc3b3c88e6f99e81239"}
Sep 10 14:33:31.662391 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.662366    2547 generic.go:356] "Generic (PLEG): container finished" podID="791776a6-9a65-4d4b-ad77-2d9260ec70c6" containerID="e29a18e6042b1975362bbc8c673ab4b3bf4ed1114b26eef275630a61a4b1ca8f" exitCode=0
Sep 10 14:33:31.662467 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.662445    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd" event={"ID":"791776a6-9a65-4d4b-ad77-2d9260ec70c6","Type":"ContainerDied","Data":"e29a18e6042b1975362bbc8c673ab4b3bf4ed1114b26eef275630a61a4b1ca8f"}
Sep 10 14:33:31.662523 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.662491    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd" event={"ID":"791776a6-9a65-4d4b-ad77-2d9260ec70c6","Type":"ContainerStarted","Data":"69a1a6c58e577f7004e4874c87f43b6b32dabee48422b3d3fada0c1c4ffb1170"}
Sep 10 14:33:31.664675 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.664474    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn" event={"ID":"0dfb8070-bff2-428a-b7d4-d54dfdedfbfa","Type":"ContainerStarted","Data":"fd6a6474c8f72b75edc6a8af3c63db226a8d3775d1b32011e8b5b48ebbfe0b58"}
Sep 10 14:33:31.666134 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.666111    2547 generic.go:356] "Generic (PLEG): container finished" podID="1fe83afe-38ab-4386-9653-e6385a966d37" containerID="40b35729f13ef57825f4ac52119d80c0465aba84bb1f7cb9c76d61ec57054242" exitCode=0
Sep 10 14:33:31.666223 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.666184    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m" event={"ID":"1fe83afe-38ab-4386-9653-e6385a966d37","Type":"ContainerDied","Data":"40b35729f13ef57825f4ac52119d80c0465aba84bb1f7cb9c76d61ec57054242"}
Sep 10 14:33:31.666223 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:31.666214    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m" event={"ID":"1fe83afe-38ab-4386-9653-e6385a966d37","Type":"ContainerStarted","Data":"e39f42a436e7a719b2a9ef77c506841ec790caa42fa6fe50b984cb958fdb002b"}
Sep 10 14:33:32.672422 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:32.672387    2547 generic.go:356] "Generic (PLEG): container finished" podID="0dfb8070-bff2-428a-b7d4-d54dfdedfbfa" containerID="d14dcc51273cd82930813310544825e343c68b12a453923c8177c901d9ba3c62" exitCode=0
Sep 10 14:33:32.672828 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:32.672456    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn" event={"ID":"0dfb8070-bff2-428a-b7d4-d54dfdedfbfa","Type":"ContainerDied","Data":"d14dcc51273cd82930813310544825e343c68b12a453923c8177c901d9ba3c62"}
Sep 10 14:33:33.678879 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:33.678831    2547 generic.go:356] "Generic (PLEG): container finished" podID="791776a6-9a65-4d4b-ad77-2d9260ec70c6" containerID="bc17ad007d7e0534a67cb3ab8b2d714e98df1e6b32942007e6badcd37d4d6526" exitCode=0
Sep 10 14:33:33.679328 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:33.678898    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd" event={"ID":"791776a6-9a65-4d4b-ad77-2d9260ec70c6","Type":"ContainerDied","Data":"bc17ad007d7e0534a67cb3ab8b2d714e98df1e6b32942007e6badcd37d4d6526"}
Sep 10 14:33:33.680889 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:33.680867    2547 generic.go:356] "Generic (PLEG): container finished" podID="1fe83afe-38ab-4386-9653-e6385a966d37" containerID="0dd248b769205aeedcf4f4458b3d03c828d7ebe21d7c0042a6277a3c3a18bca6" exitCode=0
Sep 10 14:33:33.680970 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:33.680925    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m" event={"ID":"1fe83afe-38ab-4386-9653-e6385a966d37","Type":"ContainerDied","Data":"0dd248b769205aeedcf4f4458b3d03c828d7ebe21d7c0042a6277a3c3a18bca6"}
Sep 10 14:33:34.688203 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:34.688172    2547 generic.go:356] "Generic (PLEG): container finished" podID="1fe83afe-38ab-4386-9653-e6385a966d37" containerID="39e32dee42504098b11c27b819018c463e94f1184e1b6c3e5d66a7ea6bedcd21" exitCode=0
Sep 10 14:33:34.688621 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:34.688257    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m" event={"ID":"1fe83afe-38ab-4386-9653-e6385a966d37","Type":"ContainerDied","Data":"39e32dee42504098b11c27b819018c463e94f1184e1b6c3e5d66a7ea6bedcd21"}
Sep 10 14:33:34.690332 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:34.690308    2547 generic.go:356] "Generic (PLEG): container finished" podID="791776a6-9a65-4d4b-ad77-2d9260ec70c6" containerID="53c3aa8d5be86ef941ce7a614ab19bf8be299c1445f13a2fd3b68894712d6aaa" exitCode=0
Sep 10 14:33:34.690445 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:34.690346    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd" event={"ID":"791776a6-9a65-4d4b-ad77-2d9260ec70c6","Type":"ContainerDied","Data":"53c3aa8d5be86ef941ce7a614ab19bf8be299c1445f13a2fd3b68894712d6aaa"}
Sep 10 14:33:35.833053 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:35.833031    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m"
Sep 10 14:33:35.862316 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:35.862292    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd"
Sep 10 14:33:36.001025 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:36.000927    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpv4l\" (UniqueName: \"kubernetes.io/projected/1fe83afe-38ab-4386-9653-e6385a966d37-kube-api-access-tpv4l\") pod \"1fe83afe-38ab-4386-9653-e6385a966d37\" (UID: \"1fe83afe-38ab-4386-9653-e6385a966d37\") "
Sep 10 14:33:36.001025 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:36.000984    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fe83afe-38ab-4386-9653-e6385a966d37-util\") pod \"1fe83afe-38ab-4386-9653-e6385a966d37\" (UID: \"1fe83afe-38ab-4386-9653-e6385a966d37\") "
Sep 10 14:33:36.001209 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:36.001058    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/791776a6-9a65-4d4b-ad77-2d9260ec70c6-util\") pod \"791776a6-9a65-4d4b-ad77-2d9260ec70c6\" (UID: \"791776a6-9a65-4d4b-ad77-2d9260ec70c6\") "
Sep 10 14:33:36.001209 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:36.001113    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fe83afe-38ab-4386-9653-e6385a966d37-bundle\") pod \"1fe83afe-38ab-4386-9653-e6385a966d37\" (UID: \"1fe83afe-38ab-4386-9653-e6385a966d37\") "
Sep 10 14:33:36.001209 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:36.001149    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/791776a6-9a65-4d4b-ad77-2d9260ec70c6-bundle\") pod \"791776a6-9a65-4d4b-ad77-2d9260ec70c6\" (UID: \"791776a6-9a65-4d4b-ad77-2d9260ec70c6\") "
Sep 10 14:33:36.001209 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:36.001182    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gxl6\" (UniqueName: \"kubernetes.io/projected/791776a6-9a65-4d4b-ad77-2d9260ec70c6-kube-api-access-4gxl6\") pod \"791776a6-9a65-4d4b-ad77-2d9260ec70c6\" (UID: \"791776a6-9a65-4d4b-ad77-2d9260ec70c6\") "
Sep 10 14:33:36.001625 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:36.001592    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe83afe-38ab-4386-9653-e6385a966d37-bundle" (OuterVolumeSpecName: "bundle") pod "1fe83afe-38ab-4386-9653-e6385a966d37" (UID: "1fe83afe-38ab-4386-9653-e6385a966d37"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:33:36.001771 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:36.001748    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791776a6-9a65-4d4b-ad77-2d9260ec70c6-bundle" (OuterVolumeSpecName: "bundle") pod "791776a6-9a65-4d4b-ad77-2d9260ec70c6" (UID: "791776a6-9a65-4d4b-ad77-2d9260ec70c6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:33:36.003614 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:36.003587    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/791776a6-9a65-4d4b-ad77-2d9260ec70c6-kube-api-access-4gxl6" (OuterVolumeSpecName: "kube-api-access-4gxl6") pod "791776a6-9a65-4d4b-ad77-2d9260ec70c6" (UID: "791776a6-9a65-4d4b-ad77-2d9260ec70c6"). InnerVolumeSpecName "kube-api-access-4gxl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:33:36.003734 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:36.003624    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe83afe-38ab-4386-9653-e6385a966d37-kube-api-access-tpv4l" (OuterVolumeSpecName: "kube-api-access-tpv4l") pod "1fe83afe-38ab-4386-9653-e6385a966d37" (UID: "1fe83afe-38ab-4386-9653-e6385a966d37"). InnerVolumeSpecName "kube-api-access-tpv4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:33:36.006765 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:36.006742    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe83afe-38ab-4386-9653-e6385a966d37-util" (OuterVolumeSpecName: "util") pod "1fe83afe-38ab-4386-9653-e6385a966d37" (UID: "1fe83afe-38ab-4386-9653-e6385a966d37"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:33:36.006964 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:36.006945    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791776a6-9a65-4d4b-ad77-2d9260ec70c6-util" (OuterVolumeSpecName: "util") pod "791776a6-9a65-4d4b-ad77-2d9260ec70c6" (UID: "791776a6-9a65-4d4b-ad77-2d9260ec70c6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:33:36.101829 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:36.101801    2547 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/791776a6-9a65-4d4b-ad77-2d9260ec70c6-bundle\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:33:36.101829 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:36.101825    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gxl6\" (UniqueName: \"kubernetes.io/projected/791776a6-9a65-4d4b-ad77-2d9260ec70c6-kube-api-access-4gxl6\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:33:36.101829 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:36.101835    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpv4l\" (UniqueName: \"kubernetes.io/projected/1fe83afe-38ab-4386-9653-e6385a966d37-kube-api-access-tpv4l\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:33:36.102032 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:36.101843    2547 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fe83afe-38ab-4386-9653-e6385a966d37-util\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:33:36.102032 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:36.101851    2547 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/791776a6-9a65-4d4b-ad77-2d9260ec70c6-util\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:33:36.102032 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:36.101859    2547 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fe83afe-38ab-4386-9653-e6385a966d37-bundle\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:33:36.701838 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:36.701755    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd"
Sep 10 14:33:36.701975 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:36.701753    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chz5nd" event={"ID":"791776a6-9a65-4d4b-ad77-2d9260ec70c6","Type":"ContainerDied","Data":"69a1a6c58e577f7004e4874c87f43b6b32dabee48422b3d3fada0c1c4ffb1170"}
Sep 10 14:33:36.701975 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:36.701922    2547 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69a1a6c58e577f7004e4874c87f43b6b32dabee48422b3d3fada0c1c4ffb1170"
Sep 10 14:33:36.703802 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:36.703771    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m" event={"ID":"1fe83afe-38ab-4386-9653-e6385a966d37","Type":"ContainerDied","Data":"e39f42a436e7a719b2a9ef77c506841ec790caa42fa6fe50b984cb958fdb002b"}
Sep 10 14:33:36.703942 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:36.703817    2547 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e39f42a436e7a719b2a9ef77c506841ec790caa42fa6fe50b984cb958fdb002b"
Sep 10 14:33:36.703942 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:36.703837    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jxh7m"
Sep 10 14:33:43.738015 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:43.737966    2547 generic.go:356] "Generic (PLEG): container finished" podID="5b0dcbf8-5244-4e8e-8f09-c33ad474f824" containerID="044f4a15c42dce5f9a9bbc2771614f3284c872be668ec490b47b0e518c27703e" exitCode=0
Sep 10 14:33:43.738399 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:43.738058    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg" event={"ID":"5b0dcbf8-5244-4e8e-8f09-c33ad474f824","Type":"ContainerDied","Data":"044f4a15c42dce5f9a9bbc2771614f3284c872be668ec490b47b0e518c27703e"}
Sep 10 14:33:43.740025 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:43.740000    2547 generic.go:356] "Generic (PLEG): container finished" podID="0dfb8070-bff2-428a-b7d4-d54dfdedfbfa" containerID="62da4dd34dff32c2f57c55352148123d17e899cc6b9c9008851088e3200d5c14" exitCode=0
Sep 10 14:33:43.740109 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:43.740044    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn" event={"ID":"0dfb8070-bff2-428a-b7d4-d54dfdedfbfa","Type":"ContainerDied","Data":"62da4dd34dff32c2f57c55352148123d17e899cc6b9c9008851088e3200d5c14"}
Sep 10 14:33:44.747015 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:44.746985    2547 generic.go:356] "Generic (PLEG): container finished" podID="5b0dcbf8-5244-4e8e-8f09-c33ad474f824" containerID="235a8d3a8e0a1bd456bc7c85d5032961e6a5e5b56a07ef24a9236487f798a234" exitCode=0
Sep 10 14:33:44.747399 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:44.747082    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg" event={"ID":"5b0dcbf8-5244-4e8e-8f09-c33ad474f824","Type":"ContainerDied","Data":"235a8d3a8e0a1bd456bc7c85d5032961e6a5e5b56a07ef24a9236487f798a234"}
Sep 10 14:33:44.749078 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:44.749055    2547 generic.go:356] "Generic (PLEG): container finished" podID="0dfb8070-bff2-428a-b7d4-d54dfdedfbfa" containerID="a19cdf331f53dfc1b40bc1e76bed8f86d2b1f4a0bd80b99e7c0036c6f14aca8f" exitCode=0
Sep 10 14:33:44.749191 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:44.749108    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn" event={"ID":"0dfb8070-bff2-428a-b7d4-d54dfdedfbfa","Type":"ContainerDied","Data":"a19cdf331f53dfc1b40bc1e76bed8f86d2b1f4a0bd80b99e7c0036c6f14aca8f"}
Sep 10 14:33:45.898225 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:45.898189    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn"
Sep 10 14:33:45.924461 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:45.924432    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg"
Sep 10 14:33:45.982115 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:45.982088    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5rpj\" (UniqueName: \"kubernetes.io/projected/0dfb8070-bff2-428a-b7d4-d54dfdedfbfa-kube-api-access-d5rpj\") pod \"0dfb8070-bff2-428a-b7d4-d54dfdedfbfa\" (UID: \"0dfb8070-bff2-428a-b7d4-d54dfdedfbfa\") "
Sep 10 14:33:45.982274 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:45.982155    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b0dcbf8-5244-4e8e-8f09-c33ad474f824-bundle\") pod \"5b0dcbf8-5244-4e8e-8f09-c33ad474f824\" (UID: \"5b0dcbf8-5244-4e8e-8f09-c33ad474f824\") "
Sep 10 14:33:45.982274 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:45.982178    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dfb8070-bff2-428a-b7d4-d54dfdedfbfa-util\") pod \"0dfb8070-bff2-428a-b7d4-d54dfdedfbfa\" (UID: \"0dfb8070-bff2-428a-b7d4-d54dfdedfbfa\") "
Sep 10 14:33:45.982274 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:45.982217    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b0dcbf8-5244-4e8e-8f09-c33ad474f824-util\") pod \"5b0dcbf8-5244-4e8e-8f09-c33ad474f824\" (UID: \"5b0dcbf8-5244-4e8e-8f09-c33ad474f824\") "
Sep 10 14:33:45.982274 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:45.982235    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dfb8070-bff2-428a-b7d4-d54dfdedfbfa-bundle\") pod \"0dfb8070-bff2-428a-b7d4-d54dfdedfbfa\" (UID: \"0dfb8070-bff2-428a-b7d4-d54dfdedfbfa\") "
Sep 10 14:33:45.982274 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:45.982262    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbjxs\" (UniqueName: \"kubernetes.io/projected/5b0dcbf8-5244-4e8e-8f09-c33ad474f824-kube-api-access-fbjxs\") pod \"5b0dcbf8-5244-4e8e-8f09-c33ad474f824\" (UID: \"5b0dcbf8-5244-4e8e-8f09-c33ad474f824\") "
Sep 10 14:33:45.982739 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:45.982710    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b0dcbf8-5244-4e8e-8f09-c33ad474f824-bundle" (OuterVolumeSpecName: "bundle") pod "5b0dcbf8-5244-4e8e-8f09-c33ad474f824" (UID: "5b0dcbf8-5244-4e8e-8f09-c33ad474f824"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:33:45.983064 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:45.983022    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dfb8070-bff2-428a-b7d4-d54dfdedfbfa-bundle" (OuterVolumeSpecName: "bundle") pod "0dfb8070-bff2-428a-b7d4-d54dfdedfbfa" (UID: "0dfb8070-bff2-428a-b7d4-d54dfdedfbfa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:33:45.984384 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:45.984355    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dfb8070-bff2-428a-b7d4-d54dfdedfbfa-kube-api-access-d5rpj" (OuterVolumeSpecName: "kube-api-access-d5rpj") pod "0dfb8070-bff2-428a-b7d4-d54dfdedfbfa" (UID: "0dfb8070-bff2-428a-b7d4-d54dfdedfbfa"). InnerVolumeSpecName "kube-api-access-d5rpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:33:45.984678 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:45.984656    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0dcbf8-5244-4e8e-8f09-c33ad474f824-kube-api-access-fbjxs" (OuterVolumeSpecName: "kube-api-access-fbjxs") pod "5b0dcbf8-5244-4e8e-8f09-c33ad474f824" (UID: "5b0dcbf8-5244-4e8e-8f09-c33ad474f824"). InnerVolumeSpecName "kube-api-access-fbjxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:33:45.987359 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:45.987338    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dfb8070-bff2-428a-b7d4-d54dfdedfbfa-util" (OuterVolumeSpecName: "util") pod "0dfb8070-bff2-428a-b7d4-d54dfdedfbfa" (UID: "0dfb8070-bff2-428a-b7d4-d54dfdedfbfa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:33:45.987776 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:45.987755    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b0dcbf8-5244-4e8e-8f09-c33ad474f824-util" (OuterVolumeSpecName: "util") pod "5b0dcbf8-5244-4e8e-8f09-c33ad474f824" (UID: "5b0dcbf8-5244-4e8e-8f09-c33ad474f824"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:33:46.083057 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:46.082965    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbjxs\" (UniqueName: \"kubernetes.io/projected/5b0dcbf8-5244-4e8e-8f09-c33ad474f824-kube-api-access-fbjxs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:33:46.083057 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:46.083005    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5rpj\" (UniqueName: \"kubernetes.io/projected/0dfb8070-bff2-428a-b7d4-d54dfdedfbfa-kube-api-access-d5rpj\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:33:46.083057 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:46.083015    2547 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b0dcbf8-5244-4e8e-8f09-c33ad474f824-bundle\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:33:46.083057 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:46.083024    2547 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dfb8070-bff2-428a-b7d4-d54dfdedfbfa-util\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:33:46.083057 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:46.083031    2547 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b0dcbf8-5244-4e8e-8f09-c33ad474f824-util\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:33:46.083057 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:46.083041    2547 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dfb8070-bff2-428a-b7d4-d54dfdedfbfa-bundle\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:33:46.761975 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:46.761894    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn"
Sep 10 14:33:46.761975 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:46.761925    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asztsn" event={"ID":"0dfb8070-bff2-428a-b7d4-d54dfdedfbfa","Type":"ContainerDied","Data":"fd6a6474c8f72b75edc6a8af3c63db226a8d3775d1b32011e8b5b48ebbfe0b58"}
Sep 10 14:33:46.761975 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:46.761973    2547 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd6a6474c8f72b75edc6a8af3c63db226a8d3775d1b32011e8b5b48ebbfe0b58"
Sep 10 14:33:46.763959 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:46.763933    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg" event={"ID":"5b0dcbf8-5244-4e8e-8f09-c33ad474f824","Type":"ContainerDied","Data":"97ba4fd22d67de69ea389bfc5872f326b211467b11d18dc3b3c88e6f99e81239"}
Sep 10 14:33:46.764085 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:46.764022    2547 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97ba4fd22d67de69ea389bfc5872f326b211467b11d18dc3b3c88e6f99e81239"
Sep 10 14:33:46.764085 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:33:46.763977    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78tgxg"
Sep 10 14:34:48.350373 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:34:48.350289    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mg9d6_359d40e9-f428-4cc3-b4e4-991c081d664a/console-operator/1.log"
Sep 10 14:34:48.351754 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:34:48.351729    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mg9d6_359d40e9-f428-4cc3-b4e4-991c081d664a/console-operator/1.log"
Sep 10 14:35:04.464007 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.463966    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:35:04.464483 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464449    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fe83afe-38ab-4386-9653-e6385a966d37" containerName="pull"
Sep 10 14:35:04.464483 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464468    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fe83afe-38ab-4386-9653-e6385a966d37" containerName="pull"
Sep 10 14:35:04.464483 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464478    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b0dcbf8-5244-4e8e-8f09-c33ad474f824" containerName="extract"
Sep 10 14:35:04.464654 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464486    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b0dcbf8-5244-4e8e-8f09-c33ad474f824" containerName="extract"
Sep 10 14:35:04.464654 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464497    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791776a6-9a65-4d4b-ad77-2d9260ec70c6" containerName="pull"
Sep 10 14:35:04.464654 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464506    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="791776a6-9a65-4d4b-ad77-2d9260ec70c6" containerName="pull"
Sep 10 14:35:04.464654 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464517    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b0dcbf8-5244-4e8e-8f09-c33ad474f824" containerName="pull"
Sep 10 14:35:04.464654 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464525    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b0dcbf8-5244-4e8e-8f09-c33ad474f824" containerName="pull"
Sep 10 14:35:04.464654 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464539    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dfb8070-bff2-428a-b7d4-d54dfdedfbfa" containerName="pull"
Sep 10 14:35:04.464654 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464547    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dfb8070-bff2-428a-b7d4-d54dfdedfbfa" containerName="pull"
Sep 10 14:35:04.464654 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464557    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dfb8070-bff2-428a-b7d4-d54dfdedfbfa" containerName="extract"
Sep 10 14:35:04.464654 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464566    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dfb8070-bff2-428a-b7d4-d54dfdedfbfa" containerName="extract"
Sep 10 14:35:04.464654 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464583    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fe83afe-38ab-4386-9653-e6385a966d37" containerName="util"
Sep 10 14:35:04.464654 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464591    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fe83afe-38ab-4386-9653-e6385a966d37" containerName="util"
Sep 10 14:35:04.464654 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464602    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fe83afe-38ab-4386-9653-e6385a966d37" containerName="extract"
Sep 10 14:35:04.464654 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464610    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fe83afe-38ab-4386-9653-e6385a966d37" containerName="extract"
Sep 10 14:35:04.464654 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464622    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dfb8070-bff2-428a-b7d4-d54dfdedfbfa" containerName="util"
Sep 10 14:35:04.464654 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464630    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dfb8070-bff2-428a-b7d4-d54dfdedfbfa" containerName="util"
Sep 10 14:35:04.464654 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464644    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791776a6-9a65-4d4b-ad77-2d9260ec70c6" containerName="util"
Sep 10 14:35:04.464654 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464651    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="791776a6-9a65-4d4b-ad77-2d9260ec70c6" containerName="util"
Sep 10 14:35:04.464654 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464659    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b0dcbf8-5244-4e8e-8f09-c33ad474f824" containerName="util"
Sep 10 14:35:04.465483 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464667    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b0dcbf8-5244-4e8e-8f09-c33ad474f824" containerName="util"
Sep 10 14:35:04.465483 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464684    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791776a6-9a65-4d4b-ad77-2d9260ec70c6" containerName="extract"
Sep 10 14:35:04.465483 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464692    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="791776a6-9a65-4d4b-ad77-2d9260ec70c6" containerName="extract"
Sep 10 14:35:04.465483 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464806    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b0dcbf8-5244-4e8e-8f09-c33ad474f824" containerName="extract"
Sep 10 14:35:04.465483 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464819    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="791776a6-9a65-4d4b-ad77-2d9260ec70c6" containerName="extract"
Sep 10 14:35:04.465483 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464827    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dfb8070-bff2-428a-b7d4-d54dfdedfbfa" containerName="extract"
Sep 10 14:35:04.465483 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.464840    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fe83afe-38ab-4386-9653-e6385a966d37" containerName="extract"
Sep 10 14:35:04.468166 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.468146    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"
Sep 10 14:35:04.470601 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.470578    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 14:35:04.470744 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.470653    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dmbns\""
Sep 10 14:35:04.470827 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.470760    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 14:35:04.471745 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.471725    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 14:35:04.476884 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.476863    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:35:04.507997 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.507959    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbz2n\" (UniqueName: \"kubernetes.io/projected/9f409f9a-6846-4ea2-874e-401a2d71f247-kube-api-access-tbz2n\") pod \"limitador-limitador-84994f7ffc-c6mlk\" (UID: \"9f409f9a-6846-4ea2-874e-401a2d71f247\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"
Sep 10 14:35:04.508181 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.508017    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9f409f9a-6846-4ea2-874e-401a2d71f247-config-file\") pod \"limitador-limitador-84994f7ffc-c6mlk\" (UID: \"9f409f9a-6846-4ea2-874e-401a2d71f247\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"
Sep 10 14:35:04.513075 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.513044    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:35:04.608711 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.608664    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbz2n\" (UniqueName: \"kubernetes.io/projected/9f409f9a-6846-4ea2-874e-401a2d71f247-kube-api-access-tbz2n\") pod \"limitador-limitador-84994f7ffc-c6mlk\" (UID: \"9f409f9a-6846-4ea2-874e-401a2d71f247\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"
Sep 10 14:35:04.608895 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.608744    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9f409f9a-6846-4ea2-874e-401a2d71f247-config-file\") pod \"limitador-limitador-84994f7ffc-c6mlk\" (UID: \"9f409f9a-6846-4ea2-874e-401a2d71f247\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"
Sep 10 14:35:04.609435 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.609404    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9f409f9a-6846-4ea2-874e-401a2d71f247-config-file\") pod \"limitador-limitador-84994f7ffc-c6mlk\" (UID: \"9f409f9a-6846-4ea2-874e-401a2d71f247\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"
Sep 10 14:35:04.616561 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.616529    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbz2n\" (UniqueName: \"kubernetes.io/projected/9f409f9a-6846-4ea2-874e-401a2d71f247-kube-api-access-tbz2n\") pod \"limitador-limitador-84994f7ffc-c6mlk\" (UID: \"9f409f9a-6846-4ea2-874e-401a2d71f247\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"
Sep 10 14:35:04.779148 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:04.779060    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"
Sep 10 14:35:05.052069 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:05.052039    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:35:05.053515 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:35:05.053485    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f409f9a_6846_4ea2_874e_401a2d71f247.slice/crio-844d04c7ba925d9b595b4971241cd1fb29eff8b0c737342458ecb4dd519f05d4 WatchSource:0}: Error finding container 844d04c7ba925d9b595b4971241cd1fb29eff8b0c737342458ecb4dd519f05d4: Status 404 returned error can't find the container with id 844d04c7ba925d9b595b4971241cd1fb29eff8b0c737342458ecb4dd519f05d4
Sep 10 14:35:05.054232 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:05.054212    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-n86c8"]
Sep 10 14:35:05.058833 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:05.058809    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-n86c8"
Sep 10 14:35:05.061835 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:05.061353    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-d85tr\""
Sep 10 14:35:05.063996 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:05.063971    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-n86c8"]
Sep 10 14:35:05.113281 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:05.113246    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shf8k\" (UniqueName: \"kubernetes.io/projected/0c4b8b39-ffe0-4764-bc80-305c7e09fc68-kube-api-access-shf8k\") pod \"authorino-59c974bd48-n86c8\" (UID: \"0c4b8b39-ffe0-4764-bc80-305c7e09fc68\") " pod="kuadrant-system/authorino-59c974bd48-n86c8"
Sep 10 14:35:05.154226 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:05.154188    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-c6mlk" event={"ID":"9f409f9a-6846-4ea2-874e-401a2d71f247","Type":"ContainerStarted","Data":"844d04c7ba925d9b595b4971241cd1fb29eff8b0c737342458ecb4dd519f05d4"}
Sep 10 14:35:05.214435 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:05.214396    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shf8k\" (UniqueName: \"kubernetes.io/projected/0c4b8b39-ffe0-4764-bc80-305c7e09fc68-kube-api-access-shf8k\") pod \"authorino-59c974bd48-n86c8\" (UID: \"0c4b8b39-ffe0-4764-bc80-305c7e09fc68\") " pod="kuadrant-system/authorino-59c974bd48-n86c8"
Sep 10 14:35:05.222434 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:05.222396    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shf8k\" (UniqueName: \"kubernetes.io/projected/0c4b8b39-ffe0-4764-bc80-305c7e09fc68-kube-api-access-shf8k\") pod \"authorino-59c974bd48-n86c8\" (UID: \"0c4b8b39-ffe0-4764-bc80-305c7e09fc68\") " pod="kuadrant-system/authorino-59c974bd48-n86c8"
Sep 10 14:35:05.369583 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:05.369498    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-n86c8"
Sep 10 14:35:05.641447 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:05.641422    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-n86c8"]
Sep 10 14:35:05.643364 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:35:05.643331    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c4b8b39_ffe0_4764_bc80_305c7e09fc68.slice/crio-386429563ca5768ce20c67270c191df99f155eca148d2e9e3dd406ed51115c04 WatchSource:0}: Error finding container 386429563ca5768ce20c67270c191df99f155eca148d2e9e3dd406ed51115c04: Status 404 returned error can't find the container with id 386429563ca5768ce20c67270c191df99f155eca148d2e9e3dd406ed51115c04
Sep 10 14:35:06.162874 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:06.162837    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-n86c8" event={"ID":"0c4b8b39-ffe0-4764-bc80-305c7e09fc68","Type":"ContainerStarted","Data":"386429563ca5768ce20c67270c191df99f155eca148d2e9e3dd406ed51115c04"}
Sep 10 14:35:08.174620 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:08.174578    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-n86c8" event={"ID":"0c4b8b39-ffe0-4764-bc80-305c7e09fc68","Type":"ContainerStarted","Data":"bfc30232c4ddc8b708cd8e546df62ae8d3a16c81ce6233f0180fbe81839bf65d"}
Sep 10 14:35:08.176113 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:08.176086    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-c6mlk" event={"ID":"9f409f9a-6846-4ea2-874e-401a2d71f247","Type":"ContainerStarted","Data":"c884e67d3d1785454bc6efcec3fd61b12d696716bafbfd15e31906d28d102c29"}
Sep 10 14:35:08.176230 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:08.176210    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"
Sep 10 14:35:08.187975 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:08.187926    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-n86c8" podStartSLOduration=0.85550067 podStartE2EDuration="3.187911035s" totalImagesPullingTime="2.332410365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:35:08.187600572 +0000 UTC m=+920.229475054" watchObservedRunningTime="2026-09-10 14:35:08.187911035 +0000 UTC m=+920.229785637"
Sep 10 14:35:08.204426 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:08.204377    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-c6mlk" podStartSLOduration=1.288763058 podStartE2EDuration="4.204357346s" totalImagesPullingTime="2.915594288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:35:08.20350452 +0000 UTC m=+920.245379002" watchObservedRunningTime="2026-09-10 14:35:08.204357346 +0000 UTC m=+920.246231826"
Sep 10 14:35:19.182332 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:19.182302    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"
Sep 10 14:35:31.844034 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:31.843985    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-qgx8n"]
Sep 10 14:35:31.849197 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:31.849179    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-qgx8n"
Sep 10 14:35:31.853636 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:31.853609    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-qgx8n"]
Sep 10 14:35:31.932672 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:31.932634    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-296qt\" (UniqueName: \"kubernetes.io/projected/1df0a550-0c00-4e9b-86e6-44c95e6be3b5-kube-api-access-296qt\") pod \"authorino-557f876f69-qgx8n\" (UID: \"1df0a550-0c00-4e9b-86e6-44c95e6be3b5\") " pod="kuadrant-system/authorino-557f876f69-qgx8n"
Sep 10 14:35:31.939325 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:31.939297    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-qgx8n"]
Sep 10 14:35:31.939575 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:35:31.939554    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-296qt], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-qgx8n" podUID="1df0a550-0c00-4e9b-86e6-44c95e6be3b5"
Sep 10 14:35:32.033066 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:32.033026    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-296qt\" (UniqueName: \"kubernetes.io/projected/1df0a550-0c00-4e9b-86e6-44c95e6be3b5-kube-api-access-296qt\") pod \"authorino-557f876f69-qgx8n\" (UID: \"1df0a550-0c00-4e9b-86e6-44c95e6be3b5\") " pod="kuadrant-system/authorino-557f876f69-qgx8n"
Sep 10 14:35:32.041181 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:32.041157    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-296qt\" (UniqueName: \"kubernetes.io/projected/1df0a550-0c00-4e9b-86e6-44c95e6be3b5-kube-api-access-296qt\") pod \"authorino-557f876f69-qgx8n\" (UID: \"1df0a550-0c00-4e9b-86e6-44c95e6be3b5\") " pod="kuadrant-system/authorino-557f876f69-qgx8n"
Sep 10 14:35:32.102787 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:32.102692    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-966dc8454-t252t"]
Sep 10 14:35:32.104919 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:32.104900    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-966dc8454-t252t"
Sep 10 14:35:32.107368 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:32.107348    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 14:35:32.113426 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:32.113405    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-966dc8454-t252t"]
Sep 10 14:35:32.235165 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:32.235122    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj4vm\" (UniqueName: \"kubernetes.io/projected/415eeb5f-8c49-4fc3-b6c2-1660e667fae3-kube-api-access-tj4vm\") pod \"authorino-966dc8454-t252t\" (UID: \"415eeb5f-8c49-4fc3-b6c2-1660e667fae3\") " pod="kuadrant-system/authorino-966dc8454-t252t"
Sep 10 14:35:32.235334 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:32.235236    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/415eeb5f-8c49-4fc3-b6c2-1660e667fae3-tls-cert\") pod \"authorino-966dc8454-t252t\" (UID: \"415eeb5f-8c49-4fc3-b6c2-1660e667fae3\") " pod="kuadrant-system/authorino-966dc8454-t252t"
Sep 10 14:35:32.297180 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:32.297151    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-qgx8n"
Sep 10 14:35:32.302629 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:32.302608    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-qgx8n"
Sep 10 14:35:32.335971 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:32.335941    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj4vm\" (UniqueName: \"kubernetes.io/projected/415eeb5f-8c49-4fc3-b6c2-1660e667fae3-kube-api-access-tj4vm\") pod \"authorino-966dc8454-t252t\" (UID: \"415eeb5f-8c49-4fc3-b6c2-1660e667fae3\") " pod="kuadrant-system/authorino-966dc8454-t252t"
Sep 10 14:35:32.336116 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:32.335998    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/415eeb5f-8c49-4fc3-b6c2-1660e667fae3-tls-cert\") pod \"authorino-966dc8454-t252t\" (UID: \"415eeb5f-8c49-4fc3-b6c2-1660e667fae3\") " pod="kuadrant-system/authorino-966dc8454-t252t"
Sep 10 14:35:32.338360 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:32.338338    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/415eeb5f-8c49-4fc3-b6c2-1660e667fae3-tls-cert\") pod \"authorino-966dc8454-t252t\" (UID: \"415eeb5f-8c49-4fc3-b6c2-1660e667fae3\") " pod="kuadrant-system/authorino-966dc8454-t252t"
Sep 10 14:35:32.344512 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:32.344489    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj4vm\" (UniqueName: \"kubernetes.io/projected/415eeb5f-8c49-4fc3-b6c2-1660e667fae3-kube-api-access-tj4vm\") pod \"authorino-966dc8454-t252t\" (UID: \"415eeb5f-8c49-4fc3-b6c2-1660e667fae3\") " pod="kuadrant-system/authorino-966dc8454-t252t"
Sep 10 14:35:32.415517 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:32.415487    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-966dc8454-t252t"
Sep 10 14:35:32.436652 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:32.436627    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-296qt\" (UniqueName: \"kubernetes.io/projected/1df0a550-0c00-4e9b-86e6-44c95e6be3b5-kube-api-access-296qt\") pod \"1df0a550-0c00-4e9b-86e6-44c95e6be3b5\" (UID: \"1df0a550-0c00-4e9b-86e6-44c95e6be3b5\") "
Sep 10 14:35:32.438563 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:32.438539    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df0a550-0c00-4e9b-86e6-44c95e6be3b5-kube-api-access-296qt" (OuterVolumeSpecName: "kube-api-access-296qt") pod "1df0a550-0c00-4e9b-86e6-44c95e6be3b5" (UID: "1df0a550-0c00-4e9b-86e6-44c95e6be3b5"). InnerVolumeSpecName "kube-api-access-296qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:35:32.537920 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:32.537881    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-296qt\" (UniqueName: \"kubernetes.io/projected/1df0a550-0c00-4e9b-86e6-44c95e6be3b5-kube-api-access-296qt\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:35:32.686344 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:32.686311    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-966dc8454-t252t"]
Sep 10 14:35:32.687942 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:35:32.687919    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod415eeb5f_8c49_4fc3_b6c2_1660e667fae3.slice/crio-5b189843e6bd01452bcf9d79a972530126a931321c0afad0b2e079bb2b70e78a WatchSource:0}: Error finding container 5b189843e6bd01452bcf9d79a972530126a931321c0afad0b2e079bb2b70e78a: Status 404 returned error can't find the container with id 5b189843e6bd01452bcf9d79a972530126a931321c0afad0b2e079bb2b70e78a
Sep 10 14:35:33.303712 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:33.303625    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-qgx8n"
Sep 10 14:35:33.303712 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:33.303664    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-966dc8454-t252t" event={"ID":"415eeb5f-8c49-4fc3-b6c2-1660e667fae3","Type":"ContainerStarted","Data":"5b189843e6bd01452bcf9d79a972530126a931321c0afad0b2e079bb2b70e78a"}
Sep 10 14:35:33.333905 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:33.333870    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-qgx8n"]
Sep 10 14:35:33.336982 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:33.336955    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-qgx8n"]
Sep 10 14:35:34.310364 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:34.310327    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-966dc8454-t252t" event={"ID":"415eeb5f-8c49-4fc3-b6c2-1660e667fae3","Type":"ContainerStarted","Data":"df964ed273f97009bf473144f77aa6e7e2e62d5e3b0a00a6cce5c59f0131d6bd"}
Sep 10 14:35:34.329039 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:34.328986    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-966dc8454-t252t" podStartSLOduration=1.636324686 podStartE2EDuration="2.328973902s" totalImagesPullingTime="692.649216ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:35:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:35:34.328374994 +0000 UTC m=+946.370249478" watchObservedRunningTime="2026-09-10 14:35:34.328973902 +0000 UTC m=+946.370848383"
Sep 10 14:35:34.351602 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:34.351565    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-n86c8"]
Sep 10 14:35:34.351824 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:34.351800    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-n86c8" podUID="0c4b8b39-ffe0-4764-bc80-305c7e09fc68" containerName="authorino" containerID="cri-o://bfc30232c4ddc8b708cd8e546df62ae8d3a16c81ce6233f0180fbe81839bf65d" gracePeriod=30
Sep 10 14:35:34.421564 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:34.421532    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1df0a550-0c00-4e9b-86e6-44c95e6be3b5" path="/var/lib/kubelet/pods/1df0a550-0c00-4e9b-86e6-44c95e6be3b5/volumes"
Sep 10 14:35:34.594286 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:34.594260    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-n86c8"
Sep 10 14:35:34.759666 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:34.759613    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shf8k\" (UniqueName: \"kubernetes.io/projected/0c4b8b39-ffe0-4764-bc80-305c7e09fc68-kube-api-access-shf8k\") pod \"0c4b8b39-ffe0-4764-bc80-305c7e09fc68\" (UID: \"0c4b8b39-ffe0-4764-bc80-305c7e09fc68\") "
Sep 10 14:35:34.761871 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:34.761841    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4b8b39-ffe0-4764-bc80-305c7e09fc68-kube-api-access-shf8k" (OuterVolumeSpecName: "kube-api-access-shf8k") pod "0c4b8b39-ffe0-4764-bc80-305c7e09fc68" (UID: "0c4b8b39-ffe0-4764-bc80-305c7e09fc68"). InnerVolumeSpecName "kube-api-access-shf8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:35:34.860645 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:34.860561    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shf8k\" (UniqueName: \"kubernetes.io/projected/0c4b8b39-ffe0-4764-bc80-305c7e09fc68-kube-api-access-shf8k\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:35:35.316225 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:35.316188    2547 generic.go:356] "Generic (PLEG): container finished" podID="0c4b8b39-ffe0-4764-bc80-305c7e09fc68" containerID="bfc30232c4ddc8b708cd8e546df62ae8d3a16c81ce6233f0180fbe81839bf65d" exitCode=0
Sep 10 14:35:35.316648 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:35.316233    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-n86c8"
Sep 10 14:35:35.316648 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:35.316277    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-n86c8" event={"ID":"0c4b8b39-ffe0-4764-bc80-305c7e09fc68","Type":"ContainerDied","Data":"bfc30232c4ddc8b708cd8e546df62ae8d3a16c81ce6233f0180fbe81839bf65d"}
Sep 10 14:35:35.316648 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:35.316325    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-n86c8" event={"ID":"0c4b8b39-ffe0-4764-bc80-305c7e09fc68","Type":"ContainerDied","Data":"386429563ca5768ce20c67270c191df99f155eca148d2e9e3dd406ed51115c04"}
Sep 10 14:35:35.316648 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:35.316343    2547 scope.go:117] "RemoveContainer" containerID="bfc30232c4ddc8b708cd8e546df62ae8d3a16c81ce6233f0180fbe81839bf65d"
Sep 10 14:35:35.326572 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:35.326544    2547 scope.go:117] "RemoveContainer" containerID="bfc30232c4ddc8b708cd8e546df62ae8d3a16c81ce6233f0180fbe81839bf65d"
Sep 10 14:35:35.326852 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:35:35.326834    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfc30232c4ddc8b708cd8e546df62ae8d3a16c81ce6233f0180fbe81839bf65d\": container with ID starting with bfc30232c4ddc8b708cd8e546df62ae8d3a16c81ce6233f0180fbe81839bf65d not found: ID does not exist" containerID="bfc30232c4ddc8b708cd8e546df62ae8d3a16c81ce6233f0180fbe81839bf65d"
Sep 10 14:35:35.326895 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:35.326862    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfc30232c4ddc8b708cd8e546df62ae8d3a16c81ce6233f0180fbe81839bf65d"} err="failed to get container status \"bfc30232c4ddc8b708cd8e546df62ae8d3a16c81ce6233f0180fbe81839bf65d\": rpc error: code = NotFound desc = could not find container \"bfc30232c4ddc8b708cd8e546df62ae8d3a16c81ce6233f0180fbe81839bf65d\": container with ID starting with bfc30232c4ddc8b708cd8e546df62ae8d3a16c81ce6233f0180fbe81839bf65d not found: ID does not exist"
Sep 10 14:35:35.338889 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:35.338863    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-n86c8"]
Sep 10 14:35:35.343243 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:35.343221    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-n86c8"]
Sep 10 14:35:36.421052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:35:36.421021    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c4b8b39-ffe0-4764-bc80-305c7e09fc68" path="/var/lib/kubelet/pods/0c4b8b39-ffe0-4764-bc80-305c7e09fc68/volumes"
Sep 10 14:36:14.963243 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:36:14.963205    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:36:31.951365 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:36:31.951327    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:36:37.919939 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:36:37.919900    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:36:47.348180 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:36:47.348131    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:37:04.236859 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:04.236819    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:37:27.931417 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:27.931370    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:37:37.724917 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:37.724874    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:37:42.317572 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.317496    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"]
Sep 10 14:37:42.317932 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.317881    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c4b8b39-ffe0-4764-bc80-305c7e09fc68" containerName="authorino"
Sep 10 14:37:42.317932 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.317894    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c4b8b39-ffe0-4764-bc80-305c7e09fc68" containerName="authorino"
Sep 10 14:37:42.318008 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.317972    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c4b8b39-ffe0-4764-bc80-305c7e09fc68" containerName="authorino"
Sep 10 14:37:42.321114 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.321095    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"
Sep 10 14:37:42.324776 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.324754    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 14:37:42.324905 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.324789    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-n6q9c\""
Sep 10 14:37:42.324905 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.324833    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 14:37:42.324905 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.324789    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 14:37:42.329737 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.329717    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"]
Sep 10 14:37:42.384780 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.384752    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e81a218-b560-4d8f-892e-1ff9a5ffb2b4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sjt57\" (UID: \"5e81a218-b560-4d8f-892e-1ff9a5ffb2b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"
Sep 10 14:37:42.384923 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.384784    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e81a218-b560-4d8f-892e-1ff9a5ffb2b4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sjt57\" (UID: \"5e81a218-b560-4d8f-892e-1ff9a5ffb2b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"
Sep 10 14:37:42.384923 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.384803    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e81a218-b560-4d8f-892e-1ff9a5ffb2b4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sjt57\" (UID: \"5e81a218-b560-4d8f-892e-1ff9a5ffb2b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"
Sep 10 14:37:42.384923 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.384837    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e81a218-b560-4d8f-892e-1ff9a5ffb2b4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sjt57\" (UID: \"5e81a218-b560-4d8f-892e-1ff9a5ffb2b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"
Sep 10 14:37:42.384923 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.384865    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e81a218-b560-4d8f-892e-1ff9a5ffb2b4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sjt57\" (UID: \"5e81a218-b560-4d8f-892e-1ff9a5ffb2b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"
Sep 10 14:37:42.384923 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.384884    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8wmz\" (UniqueName: \"kubernetes.io/projected/5e81a218-b560-4d8f-892e-1ff9a5ffb2b4-kube-api-access-l8wmz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sjt57\" (UID: \"5e81a218-b560-4d8f-892e-1ff9a5ffb2b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"
Sep 10 14:37:42.486044 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.485989    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e81a218-b560-4d8f-892e-1ff9a5ffb2b4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sjt57\" (UID: \"5e81a218-b560-4d8f-892e-1ff9a5ffb2b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"
Sep 10 14:37:42.486225 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.486069    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e81a218-b560-4d8f-892e-1ff9a5ffb2b4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sjt57\" (UID: \"5e81a218-b560-4d8f-892e-1ff9a5ffb2b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"
Sep 10 14:37:42.486225 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.486092    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8wmz\" (UniqueName: \"kubernetes.io/projected/5e81a218-b560-4d8f-892e-1ff9a5ffb2b4-kube-api-access-l8wmz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sjt57\" (UID: \"5e81a218-b560-4d8f-892e-1ff9a5ffb2b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"
Sep 10 14:37:42.486225 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.486153    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e81a218-b560-4d8f-892e-1ff9a5ffb2b4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sjt57\" (UID: \"5e81a218-b560-4d8f-892e-1ff9a5ffb2b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"
Sep 10 14:37:42.486225 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.486177    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e81a218-b560-4d8f-892e-1ff9a5ffb2b4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sjt57\" (UID: \"5e81a218-b560-4d8f-892e-1ff9a5ffb2b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"
Sep 10 14:37:42.486225 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.486201    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e81a218-b560-4d8f-892e-1ff9a5ffb2b4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sjt57\" (UID: \"5e81a218-b560-4d8f-892e-1ff9a5ffb2b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"
Sep 10 14:37:42.486561 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.486537    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e81a218-b560-4d8f-892e-1ff9a5ffb2b4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sjt57\" (UID: \"5e81a218-b560-4d8f-892e-1ff9a5ffb2b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"
Sep 10 14:37:42.486561 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.486553    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e81a218-b560-4d8f-892e-1ff9a5ffb2b4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sjt57\" (UID: \"5e81a218-b560-4d8f-892e-1ff9a5ffb2b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"
Sep 10 14:37:42.486740 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.486592    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e81a218-b560-4d8f-892e-1ff9a5ffb2b4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sjt57\" (UID: \"5e81a218-b560-4d8f-892e-1ff9a5ffb2b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"
Sep 10 14:37:42.488457 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.488431    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e81a218-b560-4d8f-892e-1ff9a5ffb2b4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sjt57\" (UID: \"5e81a218-b560-4d8f-892e-1ff9a5ffb2b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"
Sep 10 14:37:42.488626 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.488609    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e81a218-b560-4d8f-892e-1ff9a5ffb2b4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sjt57\" (UID: \"5e81a218-b560-4d8f-892e-1ff9a5ffb2b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"
Sep 10 14:37:42.493531 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.493507    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8wmz\" (UniqueName: \"kubernetes.io/projected/5e81a218-b560-4d8f-892e-1ff9a5ffb2b4-kube-api-access-l8wmz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sjt57\" (UID: \"5e81a218-b560-4d8f-892e-1ff9a5ffb2b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"
Sep 10 14:37:42.630984 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.630889    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"
Sep 10 14:37:42.900623 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.900595    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"]
Sep 10 14:37:42.902648 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:37:42.902613    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e81a218_b560_4d8f_892e_1ff9a5ffb2b4.slice/crio-7262a44a2687798d6ebca96c20b6a5b447d2a3c0e45db98db927284658d47e74 WatchSource:0}: Error finding container 7262a44a2687798d6ebca96c20b6a5b447d2a3c0e45db98db927284658d47e74: Status 404 returned error can't find the container with id 7262a44a2687798d6ebca96c20b6a5b447d2a3c0e45db98db927284658d47e74
Sep 10 14:37:42.904070 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.904047    2547 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:37:42.972792 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:42.972753    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57" event={"ID":"5e81a218-b560-4d8f-892e-1ff9a5ffb2b4","Type":"ContainerStarted","Data":"7262a44a2687798d6ebca96c20b6a5b447d2a3c0e45db98db927284658d47e74"}
Sep 10 14:37:43.979939 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:43.979903    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57" event={"ID":"5e81a218-b560-4d8f-892e-1ff9a5ffb2b4","Type":"ContainerStarted","Data":"6a1bbbedfdfe99e81b1a0aa4227f2c554617eddf23fa648f32adebcd09922e2e"}
Sep 10 14:37:43.998886 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:43.998828    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57" podStartSLOduration=1.036552749 podStartE2EDuration="1.9988115s" totalImagesPullingTime="962.258751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:37:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:37:43.996560843 +0000 UTC m=+1076.038435324" watchObservedRunningTime="2026-09-10 14:37:43.9988115 +0000 UTC m=+1076.040685980"
Sep 10 14:37:45.125717 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:45.125660    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:37:52.631541 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:52.631503    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"
Sep 10 14:37:52.631950 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:52.631553    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"
Sep 10 14:37:52.644328 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:52.644302    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"
Sep 10 14:37:53.034729 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:37:53.034687    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sjt57"
Sep 10 14:38:29.435282 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:29.435249    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-666d6cbfc5-6shpd"]
Sep 10 14:38:29.439907 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:29.439889    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-666d6cbfc5-6shpd"
Sep 10 14:38:29.452861 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:29.452061    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-666d6cbfc5-6shpd"]
Sep 10 14:38:29.498989 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:29.498951    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7adf78ec-2887-4fe9-97f9-75ed6e67a05b-tls-cert\") pod \"authorino-666d6cbfc5-6shpd\" (UID: \"7adf78ec-2887-4fe9-97f9-75ed6e67a05b\") " pod="kuadrant-system/authorino-666d6cbfc5-6shpd"
Sep 10 14:38:29.499156 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:29.499033    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw4hj\" (UniqueName: \"kubernetes.io/projected/7adf78ec-2887-4fe9-97f9-75ed6e67a05b-kube-api-access-nw4hj\") pod \"authorino-666d6cbfc5-6shpd\" (UID: \"7adf78ec-2887-4fe9-97f9-75ed6e67a05b\") " pod="kuadrant-system/authorino-666d6cbfc5-6shpd"
Sep 10 14:38:29.600250 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:29.600217    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7adf78ec-2887-4fe9-97f9-75ed6e67a05b-tls-cert\") pod \"authorino-666d6cbfc5-6shpd\" (UID: \"7adf78ec-2887-4fe9-97f9-75ed6e67a05b\") " pod="kuadrant-system/authorino-666d6cbfc5-6shpd"
Sep 10 14:38:29.600423 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:29.600282    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw4hj\" (UniqueName: \"kubernetes.io/projected/7adf78ec-2887-4fe9-97f9-75ed6e67a05b-kube-api-access-nw4hj\") pod \"authorino-666d6cbfc5-6shpd\" (UID: \"7adf78ec-2887-4fe9-97f9-75ed6e67a05b\") " pod="kuadrant-system/authorino-666d6cbfc5-6shpd"
Sep 10 14:38:29.602579 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:29.602557    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7adf78ec-2887-4fe9-97f9-75ed6e67a05b-tls-cert\") pod \"authorino-666d6cbfc5-6shpd\" (UID: \"7adf78ec-2887-4fe9-97f9-75ed6e67a05b\") " pod="kuadrant-system/authorino-666d6cbfc5-6shpd"
Sep 10 14:38:29.609392 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:29.609366    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw4hj\" (UniqueName: \"kubernetes.io/projected/7adf78ec-2887-4fe9-97f9-75ed6e67a05b-kube-api-access-nw4hj\") pod \"authorino-666d6cbfc5-6shpd\" (UID: \"7adf78ec-2887-4fe9-97f9-75ed6e67a05b\") " pod="kuadrant-system/authorino-666d6cbfc5-6shpd"
Sep 10 14:38:29.756232 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:29.756139    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-666d6cbfc5-6shpd"
Sep 10 14:38:30.026213 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:30.026182    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-666d6cbfc5-6shpd"]
Sep 10 14:38:30.027542 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:38:30.027509    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7adf78ec_2887_4fe9_97f9_75ed6e67a05b.slice/crio-f541b67a9dfe8605a07429cb87883e9bed643553ecb6d49c330cd53919a3076d WatchSource:0}: Error finding container f541b67a9dfe8605a07429cb87883e9bed643553ecb6d49c330cd53919a3076d: Status 404 returned error can't find the container with id f541b67a9dfe8605a07429cb87883e9bed643553ecb6d49c330cd53919a3076d
Sep 10 14:38:30.206896 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:30.206859    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-666d6cbfc5-6shpd" event={"ID":"7adf78ec-2887-4fe9-97f9-75ed6e67a05b","Type":"ContainerStarted","Data":"f541b67a9dfe8605a07429cb87883e9bed643553ecb6d49c330cd53919a3076d"}
Sep 10 14:38:31.216455 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:31.216359    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-666d6cbfc5-6shpd" event={"ID":"7adf78ec-2887-4fe9-97f9-75ed6e67a05b","Type":"ContainerStarted","Data":"47c22bd9dc8dda2b44ed700c4ea3fd7084f746b7afaa15b5d281ab68e2432d56"}
Sep 10 14:38:31.233507 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:31.233446    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-666d6cbfc5-6shpd" podStartSLOduration=1.336604218 podStartE2EDuration="2.233430753s" totalImagesPullingTime="896.826535ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:38:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:38:31.231631234 +0000 UTC m=+1123.273505717" watchObservedRunningTime="2026-09-10 14:38:31.233430753 +0000 UTC m=+1123.275305234"
Sep 10 14:38:31.263722 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:31.263360    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-966dc8454-t252t"]
Sep 10 14:38:31.264289 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:31.263905    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-966dc8454-t252t" podUID="415eeb5f-8c49-4fc3-b6c2-1660e667fae3" containerName="authorino" containerID="cri-o://df964ed273f97009bf473144f77aa6e7e2e62d5e3b0a00a6cce5c59f0131d6bd" gracePeriod=30
Sep 10 14:38:31.524380 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:31.524352    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-966dc8454-t252t"
Sep 10 14:38:31.617438 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:31.617408    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/415eeb5f-8c49-4fc3-b6c2-1660e667fae3-tls-cert\") pod \"415eeb5f-8c49-4fc3-b6c2-1660e667fae3\" (UID: \"415eeb5f-8c49-4fc3-b6c2-1660e667fae3\") "
Sep 10 14:38:31.617621 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:31.617464    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tj4vm\" (UniqueName: \"kubernetes.io/projected/415eeb5f-8c49-4fc3-b6c2-1660e667fae3-kube-api-access-tj4vm\") pod \"415eeb5f-8c49-4fc3-b6c2-1660e667fae3\" (UID: \"415eeb5f-8c49-4fc3-b6c2-1660e667fae3\") "
Sep 10 14:38:31.619371 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:31.619344    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415eeb5f-8c49-4fc3-b6c2-1660e667fae3-kube-api-access-tj4vm" (OuterVolumeSpecName: "kube-api-access-tj4vm") pod "415eeb5f-8c49-4fc3-b6c2-1660e667fae3" (UID: "415eeb5f-8c49-4fc3-b6c2-1660e667fae3"). InnerVolumeSpecName "kube-api-access-tj4vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:38:31.627518 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:31.627495    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/415eeb5f-8c49-4fc3-b6c2-1660e667fae3-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "415eeb5f-8c49-4fc3-b6c2-1660e667fae3" (UID: "415eeb5f-8c49-4fc3-b6c2-1660e667fae3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:38:31.718779 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:31.718745    2547 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/415eeb5f-8c49-4fc3-b6c2-1660e667fae3-tls-cert\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:38:31.718779 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:31.718772    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj4vm\" (UniqueName: \"kubernetes.io/projected/415eeb5f-8c49-4fc3-b6c2-1660e667fae3-kube-api-access-tj4vm\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:38:32.223298 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:32.223210    2547 generic.go:356] "Generic (PLEG): container finished" podID="415eeb5f-8c49-4fc3-b6c2-1660e667fae3" containerID="df964ed273f97009bf473144f77aa6e7e2e62d5e3b0a00a6cce5c59f0131d6bd" exitCode=0
Sep 10 14:38:32.223298 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:32.223256    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-966dc8454-t252t"
Sep 10 14:38:32.223820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:32.223306    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-966dc8454-t252t" event={"ID":"415eeb5f-8c49-4fc3-b6c2-1660e667fae3","Type":"ContainerDied","Data":"df964ed273f97009bf473144f77aa6e7e2e62d5e3b0a00a6cce5c59f0131d6bd"}
Sep 10 14:38:32.223820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:32.223352    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-966dc8454-t252t" event={"ID":"415eeb5f-8c49-4fc3-b6c2-1660e667fae3","Type":"ContainerDied","Data":"5b189843e6bd01452bcf9d79a972530126a931321c0afad0b2e079bb2b70e78a"}
Sep 10 14:38:32.223820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:32.223379    2547 scope.go:117] "RemoveContainer" containerID="df964ed273f97009bf473144f77aa6e7e2e62d5e3b0a00a6cce5c59f0131d6bd"
Sep 10 14:38:32.232673 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:32.232655    2547 scope.go:117] "RemoveContainer" containerID="df964ed273f97009bf473144f77aa6e7e2e62d5e3b0a00a6cce5c59f0131d6bd"
Sep 10 14:38:32.232992 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:38:32.232967    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df964ed273f97009bf473144f77aa6e7e2e62d5e3b0a00a6cce5c59f0131d6bd\": container with ID starting with df964ed273f97009bf473144f77aa6e7e2e62d5e3b0a00a6cce5c59f0131d6bd not found: ID does not exist" containerID="df964ed273f97009bf473144f77aa6e7e2e62d5e3b0a00a6cce5c59f0131d6bd"
Sep 10 14:38:32.233057 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:32.233005    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df964ed273f97009bf473144f77aa6e7e2e62d5e3b0a00a6cce5c59f0131d6bd"} err="failed to get container status \"df964ed273f97009bf473144f77aa6e7e2e62d5e3b0a00a6cce5c59f0131d6bd\": rpc error: code = NotFound desc = could not find container \"df964ed273f97009bf473144f77aa6e7e2e62d5e3b0a00a6cce5c59f0131d6bd\": container with ID starting with df964ed273f97009bf473144f77aa6e7e2e62d5e3b0a00a6cce5c59f0131d6bd not found: ID does not exist"
Sep 10 14:38:32.245655 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:32.245624    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-966dc8454-t252t"]
Sep 10 14:38:32.253779 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:32.253756    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-966dc8454-t252t"]
Sep 10 14:38:32.421226 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:32.421194    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="415eeb5f-8c49-4fc3-b6c2-1660e667fae3" path="/var/lib/kubelet/pods/415eeb5f-8c49-4fc3-b6c2-1660e667fae3/volumes"
Sep 10 14:38:48.356525 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.356493    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"]
Sep 10 14:38:48.356986 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.356874    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="415eeb5f-8c49-4fc3-b6c2-1660e667fae3" containerName="authorino"
Sep 10 14:38:48.356986 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.356886    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="415eeb5f-8c49-4fc3-b6c2-1660e667fae3" containerName="authorino"
Sep 10 14:38:48.356986 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.356947    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="415eeb5f-8c49-4fc3-b6c2-1660e667fae3" containerName="authorino"
Sep 10 14:38:48.360021 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.359998    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.363519 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.363497    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c892fe-openshift-default-dockercfg-95phc\""
Sep 10 14:38:48.363633 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.363540    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 14:38:48.378089 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.378068    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"]
Sep 10 14:38:48.463048 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.463014    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gv9c\" (UniqueName: \"kubernetes.io/projected/8159cea0-0584-4d7b-8532-a136531ec0ed-kube-api-access-5gv9c\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.463189 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.463053    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-istio-data\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.463189 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.463074    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-credential-socket\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.463189 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.463133    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8159cea0-0584-4d7b-8532-a136531ec0ed-istio-token\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.463189 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.463168    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-workload-socket\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.463323 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.463245    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8159cea0-0584-4d7b-8532-a136531ec0ed-istio-podinfo\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.463323 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.463271    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8159cea0-0584-4d7b-8532-a136531ec0ed-istiod-ca-cert\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.463323 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.463317    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-workload-certs\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.463419 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.463335    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-istio-envoy\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.563984 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.563946    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-workload-certs\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.563984 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.563985    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-istio-envoy\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.564226 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.564004    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gv9c\" (UniqueName: \"kubernetes.io/projected/8159cea0-0584-4d7b-8532-a136531ec0ed-kube-api-access-5gv9c\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.564226 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.564030    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-istio-data\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.564226 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.564057    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-credential-socket\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.564226 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.564084    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8159cea0-0584-4d7b-8532-a136531ec0ed-istio-token\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.564226 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.564110    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-workload-socket\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.564226 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.564181    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8159cea0-0584-4d7b-8532-a136531ec0ed-istio-podinfo\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.564226 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.564216    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8159cea0-0584-4d7b-8532-a136531ec0ed-istiod-ca-cert\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.564595 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.564412    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-workload-certs\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.564595 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.564530    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-credential-socket\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.564595 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.564580    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-workload-socket\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.564772 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.564652    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-istio-data\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.564924 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.564902    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8159cea0-0584-4d7b-8532-a136531ec0ed-istiod-ca-cert\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.566432 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.566411    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-istio-envoy\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.566590 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.566571    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8159cea0-0584-4d7b-8532-a136531ec0ed-istio-podinfo\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.571673 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.571648    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8159cea0-0584-4d7b-8532-a136531ec0ed-istio-token\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.571757 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.571688    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gv9c\" (UniqueName: \"kubernetes.io/projected/8159cea0-0584-4d7b-8532-a136531ec0ed-kube-api-access-5gv9c\") pod \"e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") " pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.670578 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.670547    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:48.949188 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.949152    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"]
Sep 10 14:38:48.951443 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:38:48.951413    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8159cea0_0584_4d7b_8532_a136531ec0ed.slice/crio-afdb4945ba9a37ce281caca2ec5c633ab4e38bbd0241e324bcac5524c5137981 WatchSource:0}: Error finding container afdb4945ba9a37ce281caca2ec5c633ab4e38bbd0241e324bcac5524c5137981: Status 404 returned error can't find the container with id afdb4945ba9a37ce281caca2ec5c633ab4e38bbd0241e324bcac5524c5137981
Sep 10 14:38:48.954088 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.954049    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:38:48.954176 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.954134    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:38:48.954227 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:48.954179    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:38:49.309824 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:49.309728    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77" event={"ID":"8159cea0-0584-4d7b-8532-a136531ec0ed","Type":"ContainerStarted","Data":"d6ae9daf38a0c235a5484020cf22226b1ad959cd3b91b748bd85e3a78627b84c"}
Sep 10 14:38:49.309824 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:49.309793    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77" event={"ID":"8159cea0-0584-4d7b-8532-a136531ec0ed","Type":"ContainerStarted","Data":"afdb4945ba9a37ce281caca2ec5c633ab4e38bbd0241e324bcac5524c5137981"}
Sep 10 14:38:49.332018 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:49.331968    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77" podStartSLOduration=1.331954412 podStartE2EDuration="1.331954412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:38:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:38:49.331387022 +0000 UTC m=+1141.373261503" watchObservedRunningTime="2026-09-10 14:38:49.331954412 +0000 UTC m=+1141.373828892"
Sep 10 14:38:49.671445 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:49.671411    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:50.049575 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:50.049506    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7709-61d100"]
Sep 10 14:38:50.052888 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:50.052872    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7709-61d100"
Sep 10 14:38:50.056074 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:50.056049    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4cjjz\""
Sep 10 14:38:50.068952 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:50.068922    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7709-61d100"]
Sep 10 14:38:50.177457 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:50.177417    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd9gz\" (UniqueName: \"kubernetes.io/projected/be7a5c08-5572-4b03-ae5a-8a93937db52d-kube-api-access-jd9gz\") pod \"test-curl-7709-61d100\" (UID: \"be7a5c08-5572-4b03-ae5a-8a93937db52d\") " pod="opendatahub/test-curl-7709-61d100"
Sep 10 14:38:50.278460 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:50.278419    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd9gz\" (UniqueName: \"kubernetes.io/projected/be7a5c08-5572-4b03-ae5a-8a93937db52d-kube-api-access-jd9gz\") pod \"test-curl-7709-61d100\" (UID: \"be7a5c08-5572-4b03-ae5a-8a93937db52d\") " pod="opendatahub/test-curl-7709-61d100"
Sep 10 14:38:50.286733 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:50.286686    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd9gz\" (UniqueName: \"kubernetes.io/projected/be7a5c08-5572-4b03-ae5a-8a93937db52d-kube-api-access-jd9gz\") pod \"test-curl-7709-61d100\" (UID: \"be7a5c08-5572-4b03-ae5a-8a93937db52d\") " pod="opendatahub/test-curl-7709-61d100"
Sep 10 14:38:50.362780 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:50.362683    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7709-61d100"
Sep 10 14:38:50.635841 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:50.635814    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7709-61d100"]
Sep 10 14:38:50.637604 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:38:50.637559    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe7a5c08_5572_4b03_ae5a_8a93937db52d.slice/crio-39e5a7ebd01c157e34710d03b5d25d70f7ad97b378067a8403e552af95b53f5d WatchSource:0}: Error finding container 39e5a7ebd01c157e34710d03b5d25d70f7ad97b378067a8403e552af95b53f5d: Status 404 returned error can't find the container with id 39e5a7ebd01c157e34710d03b5d25d70f7ad97b378067a8403e552af95b53f5d
Sep 10 14:38:50.676044 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:50.676021    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:51.322294 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:51.322254    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7709-61d100" event={"ID":"be7a5c08-5572-4b03-ae5a-8a93937db52d","Type":"ContainerStarted","Data":"39e5a7ebd01c157e34710d03b5d25d70f7ad97b378067a8403e552af95b53f5d"}
Sep 10 14:38:51.322466 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:51.322448    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:51.323537 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:51.323520    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:38:51.635554 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:51.635467    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:38:52.329158 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:52.329120    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7709-61d100" event={"ID":"be7a5c08-5572-4b03-ae5a-8a93937db52d","Type":"ContainerStarted","Data":"fde2c105aa9403798338997514c150adc2b14f04988d535cfab5e7e6e8ec8531"}
Sep 10 14:38:52.344578 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:52.344531    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7709-61d100" podStartSLOduration=0.854709017 podStartE2EDuration="2.344515865s" totalImagesPullingTime="1.489806848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:38:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:38:52.342212585 +0000 UTC m=+1144.384087067" watchObservedRunningTime="2026-09-10 14:38:52.344515865 +0000 UTC m=+1144.386390345"
Sep 10 14:38:52.894259 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:52.894162    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7709-61d100"]
Sep 10 14:38:52.896899 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:52.896799    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7709-61d100"]
Sep 10 14:38:53.009609 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:53.009575    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7709-68f56f"]
Sep 10 14:38:53.009945 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:53.009932    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be7a5c08-5572-4b03-ae5a-8a93937db52d" containerName="test-curl-7709-61d100"
Sep 10 14:38:53.009992 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:53.009946    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="be7a5c08-5572-4b03-ae5a-8a93937db52d" containerName="test-curl-7709-61d100"
Sep 10 14:38:53.010036 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:53.010027    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="be7a5c08-5572-4b03-ae5a-8a93937db52d" containerName="test-curl-7709-61d100"
Sep 10 14:38:53.013181 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:53.013166    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7709-68f56f"
Sep 10 14:38:53.021395 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:53.021366    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7709-68f56f"]
Sep 10 14:38:53.106708 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:53.106650    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq7cj\" (UniqueName: \"kubernetes.io/projected/702d9df8-2d3a-4163-a6b3-7585fc10bcfe-kube-api-access-vq7cj\") pod \"test-curl-7709-68f56f\" (UID: \"702d9df8-2d3a-4163-a6b3-7585fc10bcfe\") " pod="opendatahub/test-curl-7709-68f56f"
Sep 10 14:38:53.207773 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:53.207741    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq7cj\" (UniqueName: \"kubernetes.io/projected/702d9df8-2d3a-4163-a6b3-7585fc10bcfe-kube-api-access-vq7cj\") pod \"test-curl-7709-68f56f\" (UID: \"702d9df8-2d3a-4163-a6b3-7585fc10bcfe\") " pod="opendatahub/test-curl-7709-68f56f"
Sep 10 14:38:53.219509 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:53.219481    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq7cj\" (UniqueName: \"kubernetes.io/projected/702d9df8-2d3a-4163-a6b3-7585fc10bcfe-kube-api-access-vq7cj\") pod \"test-curl-7709-68f56f\" (UID: \"702d9df8-2d3a-4163-a6b3-7585fc10bcfe\") " pod="opendatahub/test-curl-7709-68f56f"
Sep 10 14:38:53.323590 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:53.323553    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7709-68f56f"
Sep 10 14:38:53.336071 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:38:53.336049    2547 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7709-61d100" container="test-curl-7709-61d100"
Sep 10 14:38:53.592351 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:53.592326    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7709-68f56f"]
Sep 10 14:38:53.593897 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:38:53.593872    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod702d9df8_2d3a_4163_a6b3_7585fc10bcfe.slice/crio-b07c7a341a0910d93e46f287c4d7134cc6fcea8678de0173b099734b4018e565 WatchSource:0}: Error finding container b07c7a341a0910d93e46f287c4d7134cc6fcea8678de0173b099734b4018e565: Status 404 returned error can't find the container with id b07c7a341a0910d93e46f287c4d7134cc6fcea8678de0173b099734b4018e565
Sep 10 14:38:54.342475 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:54.342439    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7709-68f56f" event={"ID":"702d9df8-2d3a-4163-a6b3-7585fc10bcfe","Type":"ContainerStarted","Data":"0b51ec2d081386d99044017cc7f85478b28e3918c243f221b8e8aa072d28e53a"}
Sep 10 14:38:54.342475 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:54.342480    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7709-68f56f" event={"ID":"702d9df8-2d3a-4163-a6b3-7585fc10bcfe","Type":"ContainerStarted","Data":"b07c7a341a0910d93e46f287c4d7134cc6fcea8678de0173b099734b4018e565"}
Sep 10 14:38:54.343010 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:54.342611    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7709-61d100" podUID="be7a5c08-5572-4b03-ae5a-8a93937db52d" containerName="test-curl-7709-61d100" containerID="cri-o://fde2c105aa9403798338997514c150adc2b14f04988d535cfab5e7e6e8ec8531" gracePeriod=2
Sep 10 14:38:54.356417 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:54.356367    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7709-68f56f" podStartSLOduration=2.356347958 podStartE2EDuration="2.356347958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:38:54.35468082 +0000 UTC m=+1146.396555300" watchObservedRunningTime="2026-09-10 14:38:54.356347958 +0000 UTC m=+1146.398222440"
Sep 10 14:38:54.356535 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:38:54.356491    2547 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7709-61d100\" is forbidden: User \"system:node:ip-10-0-130-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-249.ec2.internal' and this object" podUID="be7a5c08-5572-4b03-ae5a-8a93937db52d" pod="opendatahub/test-curl-7709-61d100"
Sep 10 14:38:54.903521 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:54.903487    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7709-68f56f"]
Sep 10 14:38:54.909449 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:54.909423    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7709-68f56f"]
Sep 10 14:38:55.347638 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:38:55.347608    2547 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7709-68f56f" container="test-curl-7709-68f56f"
Sep 10 14:38:55.647746 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.647642    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"]
Sep 10 14:38:55.648667 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.648640    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="702d9df8-2d3a-4163-a6b3-7585fc10bcfe" containerName="test-curl-7709-68f56f"
Sep 10 14:38:55.648822 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.648810    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="702d9df8-2d3a-4163-a6b3-7585fc10bcfe" containerName="test-curl-7709-68f56f"
Sep 10 14:38:55.649100 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.649087    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="702d9df8-2d3a-4163-a6b3-7585fc10bcfe" containerName="test-curl-7709-68f56f"
Sep 10 14:38:55.653338 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.653318    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.656130 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.656108    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-e7415a-openshift-default-dockercfg-79494\""
Sep 10 14:38:55.661439 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.661414    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"]
Sep 10 14:38:55.832235 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.832204    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-workload-certs\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.832402 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.832244    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istio-data\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.832402 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.832274    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istiod-ca-cert\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.832402 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.832339    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istio-token\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.832402 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.832380    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istio-envoy\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.832557 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.832451    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-credential-socket\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.832557 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.832478    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-workload-socket\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.832557 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.832544    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s45bs\" (UniqueName: \"kubernetes.io/projected/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-kube-api-access-s45bs\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.832652 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.832585    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istio-podinfo\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.933541 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.933508    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s45bs\" (UniqueName: \"kubernetes.io/projected/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-kube-api-access-s45bs\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.933725 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.933560    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istio-podinfo\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.933725 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.933607    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-workload-certs\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.933725 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.933636    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istio-data\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.933725 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.933664    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istiod-ca-cert\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.933725 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.933691    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istio-token\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.933980 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.933855    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istio-envoy\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.933980 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.933911    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-credential-socket\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.933980 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.933945    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-workload-socket\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.934182 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.934052    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-workload-certs\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.934182 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.934130    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istio-data\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.934282 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.934199    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-credential-socket\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.934361 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.934338    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-workload-socket\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.934512 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.934490    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istiod-ca-cert\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.936049 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.936022    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istio-envoy\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.936150 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.936082    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istio-podinfo\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.941069 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.941043    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istio-token\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.941211 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.941193    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s45bs\" (UniqueName: \"kubernetes.io/projected/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-kube-api-access-s45bs\") pod \"e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") " pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:55.964867 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:55.964842    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:56.243063 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:56.243033    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"]
Sep 10 14:38:56.244755 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:38:56.244719    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87cdc6a3_8bc6_4be6_b793_1d1ac852427a.slice/crio-2a6e9ce03edf8303b81682a1b30a08d22463eca18139d503d84a89223ced706f WatchSource:0}: Error finding container 2a6e9ce03edf8303b81682a1b30a08d22463eca18139d503d84a89223ced706f: Status 404 returned error can't find the container with id 2a6e9ce03edf8303b81682a1b30a08d22463eca18139d503d84a89223ced706f
Sep 10 14:38:56.247053 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:56.247018    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:38:56.247162 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:56.247094    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:38:56.247162 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:56.247138    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:38:56.357675 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:56.357638    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps" event={"ID":"87cdc6a3-8bc6-4be6-b793-1d1ac852427a","Type":"ContainerStarted","Data":"3b22ae9b03f8f051159c23bf0b6ba9e5fc89591a1ae37af156fc5ab28e51ea9d"}
Sep 10 14:38:56.358071 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:56.357718    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps" event={"ID":"87cdc6a3-8bc6-4be6-b793-1d1ac852427a","Type":"ContainerStarted","Data":"2a6e9ce03edf8303b81682a1b30a08d22463eca18139d503d84a89223ced706f"}
Sep 10 14:38:56.358071 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:56.357903    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7709-68f56f" podUID="702d9df8-2d3a-4163-a6b3-7585fc10bcfe" containerName="test-curl-7709-68f56f" containerID="cri-o://0b51ec2d081386d99044017cc7f85478b28e3918c243f221b8e8aa072d28e53a" gracePeriod=2
Sep 10 14:38:56.378262 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:56.378181    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps" podStartSLOduration=1.37816149 podStartE2EDuration="1.37816149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:38:56.376959838 +0000 UTC m=+1148.418834321" watchObservedRunningTime="2026-09-10 14:38:56.37816149 +0000 UTC m=+1148.420035972"
Sep 10 14:38:56.378903 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:38:56.378874    2547 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7709-68f56f\" is forbidden: User \"system:node:ip-10-0-130-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-249.ec2.internal' and this object" podUID="702d9df8-2d3a-4163-a6b3-7585fc10bcfe" pod="opendatahub/test-curl-7709-68f56f"
Sep 10 14:38:56.380663 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:38:56.380631    2547 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7709-68f56f\" is forbidden: User \"system:node:ip-10-0-130-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-249.ec2.internal' and this object" podUID="702d9df8-2d3a-4163-a6b3-7585fc10bcfe" pod="opendatahub/test-curl-7709-68f56f"
Sep 10 14:38:56.479870 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:56.479840    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7709-61d100"
Sep 10 14:38:56.638971 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:56.638888    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jd9gz\" (UniqueName: \"kubernetes.io/projected/be7a5c08-5572-4b03-ae5a-8a93937db52d-kube-api-access-jd9gz\") pod \"be7a5c08-5572-4b03-ae5a-8a93937db52d\" (UID: \"be7a5c08-5572-4b03-ae5a-8a93937db52d\") "
Sep 10 14:38:56.641049 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:56.641013    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7a5c08-5572-4b03-ae5a-8a93937db52d-kube-api-access-jd9gz" (OuterVolumeSpecName: "kube-api-access-jd9gz") pod "be7a5c08-5572-4b03-ae5a-8a93937db52d" (UID: "be7a5c08-5572-4b03-ae5a-8a93937db52d"). InnerVolumeSpecName "kube-api-access-jd9gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:38:56.739461 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:56.739425    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd9gz\" (UniqueName: \"kubernetes.io/projected/be7a5c08-5572-4b03-ae5a-8a93937db52d-kube-api-access-jd9gz\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:38:56.965467 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:56.965437    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:57.364657 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:57.364562    2547 generic.go:356] "Generic (PLEG): container finished" podID="be7a5c08-5572-4b03-ae5a-8a93937db52d" containerID="fde2c105aa9403798338997514c150adc2b14f04988d535cfab5e7e6e8ec8531" exitCode=137
Sep 10 14:38:57.364657 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:57.364609    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7709-61d100"
Sep 10 14:38:57.364657 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:57.364629    2547 scope.go:117] "RemoveContainer" containerID="fde2c105aa9403798338997514c150adc2b14f04988d535cfab5e7e6e8ec8531"
Sep 10 14:38:57.373582 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:57.373567    2547 scope.go:117] "RemoveContainer" containerID="fde2c105aa9403798338997514c150adc2b14f04988d535cfab5e7e6e8ec8531"
Sep 10 14:38:57.373879 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:38:57.373858    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fde2c105aa9403798338997514c150adc2b14f04988d535cfab5e7e6e8ec8531\": container with ID starting with fde2c105aa9403798338997514c150adc2b14f04988d535cfab5e7e6e8ec8531 not found: ID does not exist" containerID="fde2c105aa9403798338997514c150adc2b14f04988d535cfab5e7e6e8ec8531"
Sep 10 14:38:57.373935 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:57.373887    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fde2c105aa9403798338997514c150adc2b14f04988d535cfab5e7e6e8ec8531"} err="failed to get container status \"fde2c105aa9403798338997514c150adc2b14f04988d535cfab5e7e6e8ec8531\": rpc error: code = NotFound desc = could not find container \"fde2c105aa9403798338997514c150adc2b14f04988d535cfab5e7e6e8ec8531\": container with ID starting with fde2c105aa9403798338997514c150adc2b14f04988d535cfab5e7e6e8ec8531 not found: ID does not exist"
Sep 10 14:38:57.374944 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:38:57.374924    2547 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7709-61d100\" is forbidden: User \"system:node:ip-10-0-130-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-249.ec2.internal' and this object" podUID="be7a5c08-5572-4b03-ae5a-8a93937db52d" pod="opendatahub/test-curl-7709-61d100"
Sep 10 14:38:57.971304 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:57.971274    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:57.973919 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:38:57.973890    2547 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7709-61d100\" is forbidden: User \"system:node:ip-10-0-130-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-249.ec2.internal' and this object" podUID="be7a5c08-5572-4b03-ae5a-8a93937db52d" pod="opendatahub/test-curl-7709-61d100"
Sep 10 14:38:58.373130 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:58.373105    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:58.374264 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:58.374242    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:38:58.376589 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:38:58.376557    2547 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7709-61d100\" is forbidden: User \"system:node:ip-10-0-130-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-249.ec2.internal' and this object" podUID="be7a5c08-5572-4b03-ae5a-8a93937db52d" pod="opendatahub/test-curl-7709-61d100"
Sep 10 14:38:58.422509 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:58.422478    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be7a5c08-5572-4b03-ae5a-8a93937db52d" path="/var/lib/kubelet/pods/be7a5c08-5572-4b03-ae5a-8a93937db52d/volumes"
Sep 10 14:38:58.423473 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:38:58.423442    2547 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7709-61d100\" is forbidden: User \"system:node:ip-10-0-130-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-249.ec2.internal' and this object" podUID="be7a5c08-5572-4b03-ae5a-8a93937db52d" pod="opendatahub/test-curl-7709-61d100"
Sep 10 14:38:58.550302 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:58.550271    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7709-68f56f"
Sep 10 14:38:58.552425 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:38:58.552393    2547 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7709-68f56f\" is forbidden: User \"system:node:ip-10-0-130-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-249.ec2.internal' and this object" podUID="702d9df8-2d3a-4163-a6b3-7585fc10bcfe" pod="opendatahub/test-curl-7709-68f56f"
Sep 10 14:38:58.553600 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:58.553582    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq7cj\" (UniqueName: \"kubernetes.io/projected/702d9df8-2d3a-4163-a6b3-7585fc10bcfe-kube-api-access-vq7cj\") pod \"702d9df8-2d3a-4163-a6b3-7585fc10bcfe\" (UID: \"702d9df8-2d3a-4163-a6b3-7585fc10bcfe\") "
Sep 10 14:38:58.555534 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:58.555513    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702d9df8-2d3a-4163-a6b3-7585fc10bcfe-kube-api-access-vq7cj" (OuterVolumeSpecName: "kube-api-access-vq7cj") pod "702d9df8-2d3a-4163-a6b3-7585fc10bcfe" (UID: "702d9df8-2d3a-4163-a6b3-7585fc10bcfe"). InnerVolumeSpecName "kube-api-access-vq7cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:38:58.658282 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:58.658191    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq7cj\" (UniqueName: \"kubernetes.io/projected/702d9df8-2d3a-4163-a6b3-7585fc10bcfe-kube-api-access-vq7cj\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:38:59.381059 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:59.381022    2547 generic.go:356] "Generic (PLEG): container finished" podID="702d9df8-2d3a-4163-a6b3-7585fc10bcfe" containerID="0b51ec2d081386d99044017cc7f85478b28e3918c243f221b8e8aa072d28e53a" exitCode=137
Sep 10 14:38:59.381059 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:59.381058    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7709-68f56f"
Sep 10 14:38:59.381537 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:59.381125    2547 scope.go:117] "RemoveContainer" containerID="0b51ec2d081386d99044017cc7f85478b28e3918c243f221b8e8aa072d28e53a"
Sep 10 14:38:59.383259 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:38:59.383230    2547 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7709-68f56f\" is forbidden: User \"system:node:ip-10-0-130-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-249.ec2.internal' and this object" podUID="702d9df8-2d3a-4163-a6b3-7585fc10bcfe" pod="opendatahub/test-curl-7709-68f56f"
Sep 10 14:38:59.390464 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:59.390449    2547 scope.go:117] "RemoveContainer" containerID="0b51ec2d081386d99044017cc7f85478b28e3918c243f221b8e8aa072d28e53a"
Sep 10 14:38:59.390716 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:38:59.390679    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b51ec2d081386d99044017cc7f85478b28e3918c243f221b8e8aa072d28e53a\": container with ID starting with 0b51ec2d081386d99044017cc7f85478b28e3918c243f221b8e8aa072d28e53a not found: ID does not exist" containerID="0b51ec2d081386d99044017cc7f85478b28e3918c243f221b8e8aa072d28e53a"
Sep 10 14:38:59.390784 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:38:59.390725    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b51ec2d081386d99044017cc7f85478b28e3918c243f221b8e8aa072d28e53a"} err="failed to get container status \"0b51ec2d081386d99044017cc7f85478b28e3918c243f221b8e8aa072d28e53a\": rpc error: code = NotFound desc = could not find container \"0b51ec2d081386d99044017cc7f85478b28e3918c243f221b8e8aa072d28e53a\": container with ID starting with 0b51ec2d081386d99044017cc7f85478b28e3918c243f221b8e8aa072d28e53a not found: ID does not exist"
Sep 10 14:38:59.391876 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:38:59.391853    2547 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7709-68f56f\" is forbidden: User \"system:node:ip-10-0-130-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-249.ec2.internal' and this object" podUID="702d9df8-2d3a-4163-a6b3-7585fc10bcfe" pod="opendatahub/test-curl-7709-68f56f"
Sep 10 14:39:00.421459 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:00.421422    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="702d9df8-2d3a-4163-a6b3-7585fc10bcfe" path="/var/lib/kubelet/pods/702d9df8-2d3a-4163-a6b3-7585fc10bcfe/volumes"
Sep 10 14:39:05.745598 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:05.745557    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:39:07.363893 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:07.363837    2547 patch_prober.go:28] interesting pod/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 14:39:07.364416 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:07.363915    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77" podUID="8159cea0-0584-4d7b-8532-a136531ec0ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 14:39:08.431270 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:08.431241    2547 generic.go:356] "Generic (PLEG): container finished" podID="8159cea0-0584-4d7b-8532-a136531ec0ed" containerID="d6ae9daf38a0c235a5484020cf22226b1ad959cd3b91b748bd85e3a78627b84c" exitCode=0
Sep 10 14:39:08.431677 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:08.431311    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77" event={"ID":"8159cea0-0584-4d7b-8532-a136531ec0ed","Type":"ContainerDied","Data":"d6ae9daf38a0c235a5484020cf22226b1ad959cd3b91b748bd85e3a78627b84c"}
Sep 10 14:39:08.431749 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:08.431730    2547 scope.go:117] "RemoveContainer" containerID="d6ae9daf38a0c235a5484020cf22226b1ad959cd3b91b748bd85e3a78627b84c"
Sep 10 14:39:08.432755 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:08.432727    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:39:08.432814 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:08.432798    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:39:08.432853 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:08.432830    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:39:09.439081 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:09.439044    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77" event={"ID":"8159cea0-0584-4d7b-8532-a136531ec0ed","Type":"ContainerStarted","Data":"32a308f5a39508b59ee763ce75e686919d574674090640ebc9858e576a351359"}
Sep 10 14:39:09.671682 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:09.671633    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:39:09.671907 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:09.671799    2547 patch_prober.go:28] interesting pod/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 10 14:39:09.671907 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:09.671847    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77" podUID="8159cea0-0584-4d7b-8532-a136531ec0ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 10 14:39:10.671854 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:10.671807    2547 patch_prober.go:28] interesting pod/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 10 14:39:10.672331 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:10.671881    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77" podUID="8159cea0-0584-4d7b-8532-a136531ec0ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 10 14:39:11.671926 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:11.671879    2547 patch_prober.go:28] interesting pod/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 10 14:39:11.672431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:11.671954    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77" podUID="8159cea0-0584-4d7b-8532-a136531ec0ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 10 14:39:12.671897 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:12.671857    2547 patch_prober.go:28] interesting pod/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 10 14:39:12.672170 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:12.672141    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77" podUID="8159cea0-0584-4d7b-8532-a136531ec0ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 10 14:39:13.671671 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:13.671630    2547 patch_prober.go:28] interesting pod/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 10 14:39:13.671985 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:13.671957    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77" podUID="8159cea0-0584-4d7b-8532-a136531ec0ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 10 14:39:14.672088 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:14.671535    2547 patch_prober.go:28] interesting pod/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 10 14:39:14.672088 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:14.671651    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77" podUID="8159cea0-0584-4d7b-8532-a136531ec0ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 10 14:39:15.675395 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:15.675360    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:39:15.675810 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:15.675654    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:39:15.676311 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:15.676292    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:39:16.840089 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:16.840045    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"]
Sep 10 14:39:16.843310 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:16.843284    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:16.845254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:16.845216    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"]
Sep 10 14:39:16.846503 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:16.846469    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2365fe-openshift-default-dockercfg-b5hfv\""
Sep 10 14:39:16.934853 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:16.934813    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istio-data\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:16.935024 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:16.934864    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istio-token\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:16.935024 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:16.934917    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-workload-socket\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:16.935024 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:16.934985    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istiod-ca-cert\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:16.935162 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:16.935028    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istio-podinfo\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:16.935162 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:16.935056    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-credential-socket\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:16.935162 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:16.935071    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-workload-certs\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:16.935162 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:16.935129    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmjx9\" (UniqueName: \"kubernetes.io/projected/1e93f79d-8a6f-467d-b41c-151e9160b8b3-kube-api-access-dmjx9\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:16.935289 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:16.935165    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istio-envoy\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:17.037324 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.036364    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istiod-ca-cert\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:17.037324 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.036400    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istio-podinfo\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:17.037324 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.036449    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-credential-socket\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:17.037324 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.036475    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-workload-certs\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:17.037324 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.036504    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmjx9\" (UniqueName: \"kubernetes.io/projected/1e93f79d-8a6f-467d-b41c-151e9160b8b3-kube-api-access-dmjx9\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:17.037324 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.036531    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istio-envoy\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:17.037324 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.036564    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istio-data\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:17.037324 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.036748    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istio-token\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:17.037324 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.036831    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-workload-socket\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:17.037324 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.036903    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-credential-socket\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:17.037324 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.036938    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-workload-certs\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:17.037324 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.037149    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istio-data\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:17.037324 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.037176    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-workload-socket\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:17.053252 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.037359    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istiod-ca-cert\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:17.053252 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.038896    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istio-envoy\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:17.053252 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.039040    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istio-podinfo\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:17.053252 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.045077    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmjx9\" (UniqueName: \"kubernetes.io/projected/1e93f79d-8a6f-467d-b41c-151e9160b8b3-kube-api-access-dmjx9\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:17.053252 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.045134    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istio-token\") pod \"e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") " pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:17.157442 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.157415    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:17.434780 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.434755    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"]
Sep 10 14:39:17.435735 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:39:17.435691    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e93f79d_8a6f_467d_b41c_151e9160b8b3.slice/crio-3034b8e1794543431d8d66cd7e7bc4a07cd982bd21912b6405fca036d7ecb97d WatchSource:0}: Error finding container 3034b8e1794543431d8d66cd7e7bc4a07cd982bd21912b6405fca036d7ecb97d: Status 404 returned error can't find the container with id 3034b8e1794543431d8d66cd7e7bc4a07cd982bd21912b6405fca036d7ecb97d
Sep 10 14:39:17.437859 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.437829    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:39:17.437927 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.437891    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:39:17.437927 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.437920    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:39:17.492063 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.491985    2547 generic.go:356] "Generic (PLEG): container finished" podID="87cdc6a3-8bc6-4be6-b793-1d1ac852427a" containerID="3b22ae9b03f8f051159c23bf0b6ba9e5fc89591a1ae37af156fc5ab28e51ea9d" exitCode=0
Sep 10 14:39:17.492137 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.492057    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps" event={"ID":"87cdc6a3-8bc6-4be6-b793-1d1ac852427a","Type":"ContainerDied","Data":"3b22ae9b03f8f051159c23bf0b6ba9e5fc89591a1ae37af156fc5ab28e51ea9d"}
Sep 10 14:39:17.492567 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.492544    2547 scope.go:117] "RemoveContainer" containerID="3b22ae9b03f8f051159c23bf0b6ba9e5fc89591a1ae37af156fc5ab28e51ea9d"
Sep 10 14:39:17.493763 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.493728    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:39:17.493872 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.493816    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:39:17.493936 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.493885    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:39:17.494149 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:17.494131    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw" event={"ID":"1e93f79d-8a6f-467d-b41c-151e9160b8b3","Type":"ContainerStarted","Data":"3034b8e1794543431d8d66cd7e7bc4a07cd982bd21912b6405fca036d7ecb97d"}
Sep 10 14:39:18.501728 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:18.501664    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps" event={"ID":"87cdc6a3-8bc6-4be6-b793-1d1ac852427a","Type":"ContainerStarted","Data":"4b38606555e2bf66f1cdb4070a89f1c7d7ffa23a554dee02eedae4a94a7f016e"}
Sep 10 14:39:18.503217 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:18.503191    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw" event={"ID":"1e93f79d-8a6f-467d-b41c-151e9160b8b3","Type":"ContainerStarted","Data":"c085004e758cf25c7479b81b10a3625f26b68cba07426bd5673f849eeaa62304"}
Sep 10 14:39:18.542448 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:18.542398    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw" podStartSLOduration=2.54238447 podStartE2EDuration="2.54238447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:39:18.541919984 +0000 UTC m=+1170.583794468" watchObservedRunningTime="2026-09-10 14:39:18.54238447 +0000 UTC m=+1170.584258950"
Sep 10 14:39:18.965964 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:18.965923    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:39:18.966178 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:18.966050    2547 patch_prober.go:28] interesting pod/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 10 14:39:18.966178 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:18.966093    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps" podUID="87cdc6a3-8bc6-4be6-b793-1d1ac852427a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 10 14:39:19.160006 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:19.159959    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:19.172248 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:19.172213    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:19.508036 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:19.507994    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:19.509049 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:19.509027    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:19.965444 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:19.965396    2547 patch_prober.go:28] interesting pod/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 10 14:39:19.965656 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:19.965468    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps" podUID="87cdc6a3-8bc6-4be6-b793-1d1ac852427a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 10 14:39:20.965344 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:20.965303    2547 patch_prober.go:28] interesting pod/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 10 14:39:20.965964 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:20.965930    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps" podUID="87cdc6a3-8bc6-4be6-b793-1d1ac852427a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 10 14:39:21.966152 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:21.966110    2547 patch_prober.go:28] interesting pod/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 10 14:39:21.966784 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:21.966749    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps" podUID="87cdc6a3-8bc6-4be6-b793-1d1ac852427a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 10 14:39:22.324849 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:22.324761    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:39:22.967314 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:22.967274    2547 patch_prober.go:28] interesting pod/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 10 14:39:22.967920 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:22.967888    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps" podUID="87cdc6a3-8bc6-4be6-b793-1d1ac852427a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 10 14:39:23.969712 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:23.969676    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:39:23.970176 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:23.970084    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:39:23.970805 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:23.970783    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:39:29.381270 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:29.381234    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8a6c27c-bmtft"]
Sep 10 14:39:29.384058 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:29.384041    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8a6c27c-bmtft"
Sep 10 14:39:29.386520 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:29.386495    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xnxsg\""
Sep 10 14:39:29.386644 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:29.386497    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 14:39:29.392448 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:29.392419    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8a6c27c-bmtft"]
Sep 10 14:39:29.457818 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:29.457777    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34533d51-12b3-486c-b20c-62843836a687-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e8a6c27c-bmtft\" (UID: \"34533d51-12b3-486c-b20c-62843836a687\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8a6c27c-bmtft"
Sep 10 14:39:29.559223 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:29.559187    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34533d51-12b3-486c-b20c-62843836a687-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e8a6c27c-bmtft\" (UID: \"34533d51-12b3-486c-b20c-62843836a687\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8a6c27c-bmtft"
Sep 10 14:39:29.560057 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:29.560038    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34533d51-12b3-486c-b20c-62843836a687-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e8a6c27c-bmtft\" (UID: \"34533d51-12b3-486c-b20c-62843836a687\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8a6c27c-bmtft"
Sep 10 14:39:29.695590 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:29.695527    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8a6c27c-bmtft"
Sep 10 14:39:29.985573 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:29.985542    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8a6c27c-bmtft"]
Sep 10 14:39:29.986918 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:39:29.986884    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34533d51_12b3_486c_b20c_62843836a687.slice/crio-8d452c9969271d8ea409c91a8e900ec77a70659dbf1d048193a2f20df8554b51 WatchSource:0}: Error finding container 8d452c9969271d8ea409c91a8e900ec77a70659dbf1d048193a2f20df8554b51: Status 404 returned error can't find the container with id 8d452c9969271d8ea409c91a8e900ec77a70659dbf1d048193a2f20df8554b51
Sep 10 14:39:30.575179 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:30.575145    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8a6c27c-bmtft" event={"ID":"34533d51-12b3-486c-b20c-62843836a687","Type":"ContainerStarted","Data":"8d452c9969271d8ea409c91a8e900ec77a70659dbf1d048193a2f20df8554b51"}
Sep 10 14:39:33.595912 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:33.595873    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8a6c27c-bmtft" event={"ID":"34533d51-12b3-486c-b20c-62843836a687","Type":"ContainerStarted","Data":"50c27e45597ad24a0e5544de0714d0743304897e55431f6259cd7d5ee5ddab30"}
Sep 10 14:39:33.611987 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:33.611936    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8a6c27c-bmtft" podStartSLOduration=1.918731872 podStartE2EDuration="4.611922033s" totalImagesPullingTime="2.693190161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:39:33.610762133 +0000 UTC m=+1185.652636615" watchObservedRunningTime="2026-09-10 14:39:33.611922033 +0000 UTC m=+1185.653796510"
Sep 10 14:39:34.603235 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:34.603197    2547 generic.go:356] "Generic (PLEG): container finished" podID="34533d51-12b3-486c-b20c-62843836a687" containerID="50c27e45597ad24a0e5544de0714d0743304897e55431f6259cd7d5ee5ddab30" exitCode=0
Sep 10 14:39:34.603610 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:34.603278    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8a6c27c-bmtft" event={"ID":"34533d51-12b3-486c-b20c-62843836a687","Type":"ContainerDied","Data":"50c27e45597ad24a0e5544de0714d0743304897e55431f6259cd7d5ee5ddab30"}
Sep 10 14:39:35.791420 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:35.791387    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8a6c27c-bmtft"
Sep 10 14:39:35.920945 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:35.920914    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34533d51-12b3-486c-b20c-62843836a687-maas-api-service-ca\") pod \"34533d51-12b3-486c-b20c-62843836a687\" (UID: \"34533d51-12b3-486c-b20c-62843836a687\") "
Sep 10 14:39:35.921285 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:35.921260    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34533d51-12b3-486c-b20c-62843836a687-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "34533d51-12b3-486c-b20c-62843836a687" (UID: "34533d51-12b3-486c-b20c-62843836a687"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:39:36.021892 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:36.021853    2547 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34533d51-12b3-486c-b20c-62843836a687-maas-api-service-ca\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:39:36.616471 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:36.616435    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8a6c27c-bmtft"
Sep 10 14:39:36.616628 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:36.616465    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8a6c27c-bmtft" event={"ID":"34533d51-12b3-486c-b20c-62843836a687","Type":"ContainerDied","Data":"8d452c9969271d8ea409c91a8e900ec77a70659dbf1d048193a2f20df8554b51"}
Sep 10 14:39:36.616628 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:36.616573    2547 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d452c9969271d8ea409c91a8e900ec77a70659dbf1d048193a2f20df8554b51"
Sep 10 14:39:37.624123 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:37.624087    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:39:39.635221 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:39.635138    2547 generic.go:356] "Generic (PLEG): container finished" podID="1e93f79d-8a6f-467d-b41c-151e9160b8b3" containerID="c085004e758cf25c7479b81b10a3625f26b68cba07426bd5673f849eeaa62304" exitCode=0
Sep 10 14:39:39.635221 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:39.635207    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw" event={"ID":"1e93f79d-8a6f-467d-b41c-151e9160b8b3","Type":"ContainerDied","Data":"c085004e758cf25c7479b81b10a3625f26b68cba07426bd5673f849eeaa62304"}
Sep 10 14:39:39.635651 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:39.635571    2547 scope.go:117] "RemoveContainer" containerID="c085004e758cf25c7479b81b10a3625f26b68cba07426bd5673f849eeaa62304"
Sep 10 14:39:39.636646 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:39.636620    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:39:39.636747 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:39.636683    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:39:39.636747 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:39.636737    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:39:40.632619 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:40.632580    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8a6c27c-bmtft"]
Sep 10 14:39:40.634723 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:40.634690    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e8a6c27c-bmtft"]
Sep 10 14:39:40.643133 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:40.643106    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw" event={"ID":"1e93f79d-8a6f-467d-b41c-151e9160b8b3","Type":"ContainerStarted","Data":"30d3f57c1bae4d834bb77039105161a9f3899a3349b3a9518edfd79a337f6aff"}
Sep 10 14:39:41.158143 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.158101    2547 patch_prober.go:28] interesting pod/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 10 14:39:41.158350 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.158168    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw" podUID="1e93f79d-8a6f-467d-b41c-151e9160b8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 10 14:39:41.158535 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.158515    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:41.760331 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.760295    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"]
Sep 10 14:39:41.761462 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.761438    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34533d51-12b3-486c-b20c-62843836a687" containerName="revoke-keys"
Sep 10 14:39:41.761594 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.761581    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="34533d51-12b3-486c-b20c-62843836a687" containerName="revoke-keys"
Sep 10 14:39:41.761942 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.761923    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="34533d51-12b3-486c-b20c-62843836a687" containerName="revoke-keys"
Sep 10 14:39:41.778677 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.777430    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"]
Sep 10 14:39:41.778677 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.777549    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:41.780943 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.780921    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-7e4309cc-gw-openshift-default-dockercfg-6cm64\""
Sep 10 14:39:41.897404 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.897371    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istio-token\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:41.897578 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.897417    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk42r\" (UniqueName: \"kubernetes.io/projected/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-kube-api-access-wk42r\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:41.897578 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.897451    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-workload-socket\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:41.897578 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.897498    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-workload-certs\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:41.897578 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.897538    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istio-podinfo\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:41.897578 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.897570    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istio-data\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:41.897848 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.897671    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-credential-socket\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:41.897848 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.897726    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istio-envoy\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:41.897848 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.897748    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istiod-ca-cert\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:41.998779 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.998736    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istio-podinfo\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:41.998779 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.998783    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istio-data\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:41.999033 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.998843    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-credential-socket\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:41.999033 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.998875    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istio-envoy\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:41.999033 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.998906    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istiod-ca-cert\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:41.999033 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.998966    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istio-token\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:41.999033 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.999014    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk42r\" (UniqueName: \"kubernetes.io/projected/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-kube-api-access-wk42r\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:41.999304 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.999056    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-workload-socket\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:41.999304 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.999086    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-workload-certs\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:41.999304 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.999228    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istio-data\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:41.999304 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.999259    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-credential-socket\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:41.999511 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.999430    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-workload-certs\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:41.999511 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.999467    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-workload-socket\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:41.999733 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:41.999686    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istiod-ca-cert\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:42.001312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:42.001282    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istio-envoy\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:42.001446 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:42.001422    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istio-podinfo\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:42.015195 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:42.015132    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istio-token\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:42.015278 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:42.015208    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk42r\" (UniqueName: \"kubernetes.io/projected/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-kube-api-access-wk42r\") pod \"e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") " pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:42.089920 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:42.089881    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:42.171752 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:42.171206    2547 patch_prober.go:28] interesting pod/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 10 14:39:42.171752 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:42.171268    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw" podUID="1e93f79d-8a6f-467d-b41c-151e9160b8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 10 14:39:42.367460 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:42.367368    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"]
Sep 10 14:39:42.379652 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:39:42.379617    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba57f95_0387_49e9_ba43_06a6c02fc3d3.slice/crio-531d28595e916f73ea32224a6c23c5e537bb5e61d52f1dfb07d054c373e69475 WatchSource:0}: Error finding container 531d28595e916f73ea32224a6c23c5e537bb5e61d52f1dfb07d054c373e69475: Status 404 returned error can't find the container with id 531d28595e916f73ea32224a6c23c5e537bb5e61d52f1dfb07d054c373e69475
Sep 10 14:39:42.382035 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:42.381998    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:39:42.382143 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:42.382088    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:39:42.382143 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:42.382131    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:39:42.421853 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:42.421827    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34533d51-12b3-486c-b20c-62843836a687" path="/var/lib/kubelet/pods/34533d51-12b3-486c-b20c-62843836a687/volumes"
Sep 10 14:39:42.660755 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:42.660711    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8" event={"ID":"2ba57f95-0387-49e9-ba43-06a6c02fc3d3","Type":"ContainerStarted","Data":"6649387f7690a64394db6155bdd33eab50c0d7460f12f0e939a7cabd0758bcfe"}
Sep 10 14:39:42.660945 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:42.660787    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8" event={"ID":"2ba57f95-0387-49e9-ba43-06a6c02fc3d3","Type":"ContainerStarted","Data":"531d28595e916f73ea32224a6c23c5e537bb5e61d52f1dfb07d054c373e69475"}
Sep 10 14:39:42.684685 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:42.684620    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8" podStartSLOduration=1.684600903 podStartE2EDuration="1.684600903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:39:42.681013229 +0000 UTC m=+1194.722887742" watchObservedRunningTime="2026-09-10 14:39:42.684600903 +0000 UTC m=+1194.726475377"
Sep 10 14:39:43.091150 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:43.091053    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:43.096465 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:43.096436    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:43.158611 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:43.158566    2547 patch_prober.go:28] interesting pod/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 10 14:39:43.158837 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:43.158638    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw" podUID="1e93f79d-8a6f-467d-b41c-151e9160b8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 10 14:39:43.669804 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:43.669765    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:43.673918 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:43.673895    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:39:44.157991 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:44.157887    2547 patch_prober.go:28] interesting pod/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 10 14:39:44.158566 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:44.158534    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw" podUID="1e93f79d-8a6f-467d-b41c-151e9160b8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 10 14:39:45.162723 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:45.159078    2547 patch_prober.go:28] interesting pod/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 10 14:39:45.162723 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:45.159176    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw" podUID="1e93f79d-8a6f-467d-b41c-151e9160b8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 10 14:39:46.161848 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:46.161817    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:46.162095 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:46.162075    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:46.162775 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:46.162756    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:39:48.393372 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:48.393346    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mg9d6_359d40e9-f428-4cc3-b4e4-991c081d664a/console-operator/1.log"
Sep 10 14:39:48.396125 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:48.396103    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mg9d6_359d40e9-f428-4cc3-b4e4-991c081d664a/console-operator/1.log"
Sep 10 14:39:50.016204 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.016164    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"]
Sep 10 14:39:50.020765 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.020742    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:39:50.023459 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.023435    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-61b16dfc-kserve-self-signed-certs\""
Sep 10 14:39:50.027754 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.027729    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"]
Sep 10 14:39:50.072189 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.072158    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb48298-206b-4bf1-90d4-4baa11f6c854-home\") pod \"e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx\" (UID: \"0cb48298-206b-4bf1-90d4-4baa11f6c854\") " pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:39:50.072189 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.072195    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb48298-206b-4bf1-90d4-4baa11f6c854-model-cache\") pod \"e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx\" (UID: \"0cb48298-206b-4bf1-90d4-4baa11f6c854\") " pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:39:50.072408 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.072227    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sj5d\" (UniqueName: \"kubernetes.io/projected/0cb48298-206b-4bf1-90d4-4baa11f6c854-kube-api-access-2sj5d\") pod \"e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx\" (UID: \"0cb48298-206b-4bf1-90d4-4baa11f6c854\") " pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:39:50.072408 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.072246    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb48298-206b-4bf1-90d4-4baa11f6c854-tls-certs\") pod \"e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx\" (UID: \"0cb48298-206b-4bf1-90d4-4baa11f6c854\") " pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:39:50.072408 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.072294    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb48298-206b-4bf1-90d4-4baa11f6c854-tmp-dir\") pod \"e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx\" (UID: \"0cb48298-206b-4bf1-90d4-4baa11f6c854\") " pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:39:50.072408 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.072356    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb48298-206b-4bf1-90d4-4baa11f6c854-dshm\") pod \"e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx\" (UID: \"0cb48298-206b-4bf1-90d4-4baa11f6c854\") " pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:39:50.173514 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.173478    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb48298-206b-4bf1-90d4-4baa11f6c854-model-cache\") pod \"e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx\" (UID: \"0cb48298-206b-4bf1-90d4-4baa11f6c854\") " pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:39:50.173514 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.173522    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sj5d\" (UniqueName: \"kubernetes.io/projected/0cb48298-206b-4bf1-90d4-4baa11f6c854-kube-api-access-2sj5d\") pod \"e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx\" (UID: \"0cb48298-206b-4bf1-90d4-4baa11f6c854\") " pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:39:50.173821 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.173539    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb48298-206b-4bf1-90d4-4baa11f6c854-tls-certs\") pod \"e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx\" (UID: \"0cb48298-206b-4bf1-90d4-4baa11f6c854\") " pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:39:50.173821 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.173651    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb48298-206b-4bf1-90d4-4baa11f6c854-tmp-dir\") pod \"e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx\" (UID: \"0cb48298-206b-4bf1-90d4-4baa11f6c854\") " pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:39:50.173821 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.173765    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb48298-206b-4bf1-90d4-4baa11f6c854-dshm\") pod \"e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx\" (UID: \"0cb48298-206b-4bf1-90d4-4baa11f6c854\") " pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:39:50.173995 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.173864    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb48298-206b-4bf1-90d4-4baa11f6c854-home\") pod \"e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx\" (UID: \"0cb48298-206b-4bf1-90d4-4baa11f6c854\") " pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:39:50.173995 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.173974    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb48298-206b-4bf1-90d4-4baa11f6c854-model-cache\") pod \"e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx\" (UID: \"0cb48298-206b-4bf1-90d4-4baa11f6c854\") " pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:39:50.174072 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.174043    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb48298-206b-4bf1-90d4-4baa11f6c854-tmp-dir\") pod \"e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx\" (UID: \"0cb48298-206b-4bf1-90d4-4baa11f6c854\") " pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:39:50.174141 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.174120    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb48298-206b-4bf1-90d4-4baa11f6c854-home\") pod \"e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx\" (UID: \"0cb48298-206b-4bf1-90d4-4baa11f6c854\") " pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:39:50.175884 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.175865    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb48298-206b-4bf1-90d4-4baa11f6c854-dshm\") pod \"e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx\" (UID: \"0cb48298-206b-4bf1-90d4-4baa11f6c854\") " pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:39:50.176016 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.175998    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb48298-206b-4bf1-90d4-4baa11f6c854-tls-certs\") pod \"e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx\" (UID: \"0cb48298-206b-4bf1-90d4-4baa11f6c854\") " pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:39:50.181123 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.181100    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sj5d\" (UniqueName: \"kubernetes.io/projected/0cb48298-206b-4bf1-90d4-4baa11f6c854-kube-api-access-2sj5d\") pod \"e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx\" (UID: \"0cb48298-206b-4bf1-90d4-4baa11f6c854\") " pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:39:50.332339 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.332241    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:39:50.601467 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.601440    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"]
Sep 10 14:39:50.602837 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:39:50.602813    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cb48298_206b_4bf1_90d4_4baa11f6c854.slice/crio-03e3d53d86425410ab265390b4c09ce24c455503ad43f2d9b00cdb4324225c95 WatchSource:0}: Error finding container 03e3d53d86425410ab265390b4c09ce24c455503ad43f2d9b00cdb4324225c95: Status 404 returned error can't find the container with id 03e3d53d86425410ab265390b4c09ce24c455503ad43f2d9b00cdb4324225c95
Sep 10 14:39:50.718298 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.718266    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx" event={"ID":"0cb48298-206b-4bf1-90d4-4baa11f6c854","Type":"ContainerStarted","Data":"660d4a4bf57384c2ceac1ad674a510b7ab4c2b04ecb5642386ddd9841ec1f6d0"}
Sep 10 14:39:50.718444 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.718309    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx" event={"ID":"0cb48298-206b-4bf1-90d4-4baa11f6c854","Type":"ContainerStarted","Data":"03e3d53d86425410ab265390b4c09ce24c455503ad43f2d9b00cdb4324225c95"}
Sep 10 14:39:50.738092 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:50.738043    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx" podStartSLOduration=1.738027226 podStartE2EDuration="1.738027226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:39:50.735527437 +0000 UTC m=+1202.777401920" watchObservedRunningTime="2026-09-10 14:39:50.738027226 +0000 UTC m=+1202.779901733"
Sep 10 14:39:52.945547 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:39:52.945510    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:40:00.333065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:00.333019    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:40:00.333065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:00.333071    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:40:00.345449 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:00.345423    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:40:00.793780 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:00.793739    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:40:03.800737 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:03.800678    2547 generic.go:356] "Generic (PLEG): container finished" podID="2ba57f95-0387-49e9-ba43-06a6c02fc3d3" containerID="6649387f7690a64394db6155bdd33eab50c0d7460f12f0e939a7cabd0758bcfe" exitCode=0
Sep 10 14:40:03.801142 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:03.800751    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8" event={"ID":"2ba57f95-0387-49e9-ba43-06a6c02fc3d3","Type":"ContainerDied","Data":"6649387f7690a64394db6155bdd33eab50c0d7460f12f0e939a7cabd0758bcfe"}
Sep 10 14:40:03.801185 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:03.801158    2547 scope.go:117] "RemoveContainer" containerID="6649387f7690a64394db6155bdd33eab50c0d7460f12f0e939a7cabd0758bcfe"
Sep 10 14:40:03.802346 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:03.802312    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:40:03.802453 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:03.802389    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:40:03.802453 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:03.802424    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:40:04.807654 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:04.807617    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8" event={"ID":"2ba57f95-0387-49e9-ba43-06a6c02fc3d3","Type":"ContainerStarted","Data":"79d45f973b10415307977b3ddc777d4b01ddd29339ff3bb0c75497ff283a8ab3"}
Sep 10 14:40:05.090858 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:05.090747    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:40:05.091063 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:05.091035    2547 patch_prober.go:28] interesting pod/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 10 14:40:05.091136 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:05.091097    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8" podUID="2ba57f95-0387-49e9-ba43-06a6c02fc3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 10 14:40:06.091030 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:06.090988    2547 patch_prober.go:28] interesting pod/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 10 14:40:06.091423 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:06.091051    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8" podUID="2ba57f95-0387-49e9-ba43-06a6c02fc3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 10 14:40:07.090628 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.090591    2547 patch_prober.go:28] interesting pod/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 10 14:40:07.090919 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.090652    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8" podUID="2ba57f95-0387-49e9-ba43-06a6c02fc3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 10 14:40:07.716507 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.716472    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"]
Sep 10 14:40:07.719458 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.719436    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:07.723133 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.723112    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-e7415a\"/\"openshift-service-ca.crt\""
Sep 10 14:40:07.723366 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.723345    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-e7415a\"/\"autores-model-e7415a-kserve-self-signed-certs\""
Sep 10 14:40:07.723455 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.723349    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-e7415a\"/\"kube-root-ca.crt\""
Sep 10 14:40:07.723455 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.723349    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-e7415a\"/\"default-dockercfg-s8pwl\""
Sep 10 14:40:07.731851 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.731818    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"]
Sep 10 14:40:07.869211 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.869160    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/162c423a-2c9e-4bdf-bea6-51ab402a5f94-dshm\") pod \"autores-model-e7415a-kserve-7cf48579-48954\" (UID: \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\") " pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:07.869434 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.869230    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/162c423a-2c9e-4bdf-bea6-51ab402a5f94-tls-certs\") pod \"autores-model-e7415a-kserve-7cf48579-48954\" (UID: \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\") " pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:07.869434 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.869275    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/162c423a-2c9e-4bdf-bea6-51ab402a5f94-model-cache\") pod \"autores-model-e7415a-kserve-7cf48579-48954\" (UID: \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\") " pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:07.869434 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.869331    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx77z\" (UniqueName: \"kubernetes.io/projected/162c423a-2c9e-4bdf-bea6-51ab402a5f94-kube-api-access-tx77z\") pod \"autores-model-e7415a-kserve-7cf48579-48954\" (UID: \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\") " pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:07.869434 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.869368    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/162c423a-2c9e-4bdf-bea6-51ab402a5f94-home\") pod \"autores-model-e7415a-kserve-7cf48579-48954\" (UID: \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\") " pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:07.869434 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.869413    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/162c423a-2c9e-4bdf-bea6-51ab402a5f94-tmp-dir\") pod \"autores-model-e7415a-kserve-7cf48579-48954\" (UID: \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\") " pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:07.970914 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.970811    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/162c423a-2c9e-4bdf-bea6-51ab402a5f94-dshm\") pod \"autores-model-e7415a-kserve-7cf48579-48954\" (UID: \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\") " pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:07.971179 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.971156    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/162c423a-2c9e-4bdf-bea6-51ab402a5f94-tls-certs\") pod \"autores-model-e7415a-kserve-7cf48579-48954\" (UID: \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\") " pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:07.971332 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.971317    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/162c423a-2c9e-4bdf-bea6-51ab402a5f94-model-cache\") pod \"autores-model-e7415a-kserve-7cf48579-48954\" (UID: \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\") " pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:07.971483 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.971465    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx77z\" (UniqueName: \"kubernetes.io/projected/162c423a-2c9e-4bdf-bea6-51ab402a5f94-kube-api-access-tx77z\") pod \"autores-model-e7415a-kserve-7cf48579-48954\" (UID: \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\") " pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:07.971612 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.971597    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/162c423a-2c9e-4bdf-bea6-51ab402a5f94-home\") pod \"autores-model-e7415a-kserve-7cf48579-48954\" (UID: \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\") " pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:07.971766 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.971748    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/162c423a-2c9e-4bdf-bea6-51ab402a5f94-tmp-dir\") pod \"autores-model-e7415a-kserve-7cf48579-48954\" (UID: \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\") " pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:07.972251 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.972229    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/162c423a-2c9e-4bdf-bea6-51ab402a5f94-tmp-dir\") pod \"autores-model-e7415a-kserve-7cf48579-48954\" (UID: \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\") " pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:07.972459 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.972412    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/162c423a-2c9e-4bdf-bea6-51ab402a5f94-model-cache\") pod \"autores-model-e7415a-kserve-7cf48579-48954\" (UID: \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\") " pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:07.972628 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.972603    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/162c423a-2c9e-4bdf-bea6-51ab402a5f94-home\") pod \"autores-model-e7415a-kserve-7cf48579-48954\" (UID: \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\") " pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:07.973832 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.973809    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/162c423a-2c9e-4bdf-bea6-51ab402a5f94-dshm\") pod \"autores-model-e7415a-kserve-7cf48579-48954\" (UID: \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\") " pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:07.974361 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.974343    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/162c423a-2c9e-4bdf-bea6-51ab402a5f94-tls-certs\") pod \"autores-model-e7415a-kserve-7cf48579-48954\" (UID: \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\") " pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:07.979336 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:07.979310    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx77z\" (UniqueName: \"kubernetes.io/projected/162c423a-2c9e-4bdf-bea6-51ab402a5f94-kube-api-access-tx77z\") pod \"autores-model-e7415a-kserve-7cf48579-48954\" (UID: \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\") " pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:08.030185 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:08.030147    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:08.090492 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:08.090455    2547 patch_prober.go:28] interesting pod/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 10 14:40:08.090667 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:08.090520    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8" podUID="2ba57f95-0387-49e9-ba43-06a6c02fc3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 10 14:40:08.327756 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:08.327721    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"]
Sep 10 14:40:08.328348 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:40:08.328321    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod162c423a_2c9e_4bdf_bea6_51ab402a5f94.slice/crio-1503e38f0e02d00030661ecc73143e4ef308b7a808e9a3f1d366101aea919436 WatchSource:0}: Error finding container 1503e38f0e02d00030661ecc73143e4ef308b7a808e9a3f1d366101aea919436: Status 404 returned error can't find the container with id 1503e38f0e02d00030661ecc73143e4ef308b7a808e9a3f1d366101aea919436
Sep 10 14:40:08.831670 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:08.831630    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954" event={"ID":"162c423a-2c9e-4bdf-bea6-51ab402a5f94","Type":"ContainerStarted","Data":"ccfa860bb8d03112e6c521615183f5221eb7fcdeec7e2936624fee5f24857999"}
Sep 10 14:40:08.832035 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:08.831678    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954" event={"ID":"162c423a-2c9e-4bdf-bea6-51ab402a5f94","Type":"ContainerStarted","Data":"1503e38f0e02d00030661ecc73143e4ef308b7a808e9a3f1d366101aea919436"}
Sep 10 14:40:08.860461 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:08.860397    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954" podStartSLOduration=1.860375785 podStartE2EDuration="1.860375785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:40:08.847655455 +0000 UTC m=+1220.889529938" watchObservedRunningTime="2026-09-10 14:40:08.860375785 +0000 UTC m=+1220.902250267"
Sep 10 14:40:09.090910 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:09.090818    2547 patch_prober.go:28] interesting pod/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 10 14:40:09.090910 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:09.090894    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8" podUID="2ba57f95-0387-49e9-ba43-06a6c02fc3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 10 14:40:10.094460 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:10.094427    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:40:10.094850 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:10.094675    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:40:10.095500 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:10.095477    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:40:16.030148 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:16.030105    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:40:16.108438 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:16.108404    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"]
Sep 10 14:40:16.108658 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:16.108632    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8" podUID="2ba57f95-0387-49e9-ba43-06a6c02fc3d3" containerName="istio-proxy" containerID="cri-o://79d45f973b10415307977b3ddc777d4b01ddd29339ff3bb0c75497ff283a8ab3" gracePeriod=30
Sep 10 14:40:18.031062 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:18.031025    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:18.031468 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:18.031214    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:18.044127 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:18.044098    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:18.902772 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:18.902739    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:21.362896 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.362871    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:40:21.493228 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.493199    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istio-token\") pod \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") "
Sep 10 14:40:21.493446 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.493246    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istio-envoy\") pod \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") "
Sep 10 14:40:21.493446 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.493271    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istiod-ca-cert\") pod \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") "
Sep 10 14:40:21.493446 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.493385    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wk42r\" (UniqueName: \"kubernetes.io/projected/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-kube-api-access-wk42r\") pod \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") "
Sep 10 14:40:21.493446 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.493434    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istio-data\") pod \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") "
Sep 10 14:40:21.493676 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.493483    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-credential-socket\") pod \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") "
Sep 10 14:40:21.493676 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.493522    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-workload-certs\") pod \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") "
Sep 10 14:40:21.493676 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.493629    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-workload-socket\") pod \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") "
Sep 10 14:40:21.493676 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.493653    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2ba57f95-0387-49e9-ba43-06a6c02fc3d3" (UID: "2ba57f95-0387-49e9-ba43-06a6c02fc3d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:40:21.493927 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.493737    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istio-data" (OuterVolumeSpecName: "istio-data") pod "2ba57f95-0387-49e9-ba43-06a6c02fc3d3" (UID: "2ba57f95-0387-49e9-ba43-06a6c02fc3d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:40:21.493927 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.493756    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2ba57f95-0387-49e9-ba43-06a6c02fc3d3" (UID: "2ba57f95-0387-49e9-ba43-06a6c02fc3d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:40:21.493927 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.493659    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istio-podinfo\") pod \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\" (UID: \"2ba57f95-0387-49e9-ba43-06a6c02fc3d3\") "
Sep 10 14:40:21.493927 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.493920    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2ba57f95-0387-49e9-ba43-06a6c02fc3d3" (UID: "2ba57f95-0387-49e9-ba43-06a6c02fc3d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:40:21.494113 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.493971    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2ba57f95-0387-49e9-ba43-06a6c02fc3d3" (UID: "2ba57f95-0387-49e9-ba43-06a6c02fc3d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:40:21.494194 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.494175    2547 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istiod-ca-cert\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:40:21.494232 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.494200    2547 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istio-data\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:40:21.494232 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.494215    2547 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-credential-socket\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:40:21.494232 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.494228    2547 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-workload-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:40:21.494347 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.494241    2547 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-workload-socket\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:40:21.495578 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.495553    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2ba57f95-0387-49e9-ba43-06a6c02fc3d3" (UID: "2ba57f95-0387-49e9-ba43-06a6c02fc3d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:40:21.495739 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.495720    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2ba57f95-0387-49e9-ba43-06a6c02fc3d3" (UID: "2ba57f95-0387-49e9-ba43-06a6c02fc3d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 14:40:21.495910 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.495885    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istio-token" (OuterVolumeSpecName: "istio-token") pod "2ba57f95-0387-49e9-ba43-06a6c02fc3d3" (UID: "2ba57f95-0387-49e9-ba43-06a6c02fc3d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:40:21.495998 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.495910    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-kube-api-access-wk42r" (OuterVolumeSpecName: "kube-api-access-wk42r") pod "2ba57f95-0387-49e9-ba43-06a6c02fc3d3" (UID: "2ba57f95-0387-49e9-ba43-06a6c02fc3d3"). InnerVolumeSpecName "kube-api-access-wk42r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:40:21.595222 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.595187    2547 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istio-podinfo\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:40:21.595222 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.595216    2547 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istio-token\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:40:21.595222 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.595225    2547 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-istio-envoy\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:40:21.595455 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.595234    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk42r\" (UniqueName: \"kubernetes.io/projected/2ba57f95-0387-49e9-ba43-06a6c02fc3d3-kube-api-access-wk42r\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:40:21.909956 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.909925    2547 generic.go:356] "Generic (PLEG): container finished" podID="2ba57f95-0387-49e9-ba43-06a6c02fc3d3" containerID="79d45f973b10415307977b3ddc777d4b01ddd29339ff3bb0c75497ff283a8ab3" exitCode=0
Sep 10 14:40:21.910125 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.909966    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8" event={"ID":"2ba57f95-0387-49e9-ba43-06a6c02fc3d3","Type":"ContainerDied","Data":"79d45f973b10415307977b3ddc777d4b01ddd29339ff3bb0c75497ff283a8ab3"}
Sep 10 14:40:21.910125 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.909973    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"
Sep 10 14:40:21.910125 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.910009    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8" event={"ID":"2ba57f95-0387-49e9-ba43-06a6c02fc3d3","Type":"ContainerDied","Data":"531d28595e916f73ea32224a6c23c5e537bb5e61d52f1dfb07d054c373e69475"}
Sep 10 14:40:21.910125 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.910032    2547 scope.go:117] "RemoveContainer" containerID="79d45f973b10415307977b3ddc777d4b01ddd29339ff3bb0c75497ff283a8ab3"
Sep 10 14:40:21.920140 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.920120    2547 scope.go:117] "RemoveContainer" containerID="6649387f7690a64394db6155bdd33eab50c0d7460f12f0e939a7cabd0758bcfe"
Sep 10 14:40:21.928310 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.928291    2547 scope.go:117] "RemoveContainer" containerID="79d45f973b10415307977b3ddc777d4b01ddd29339ff3bb0c75497ff283a8ab3"
Sep 10 14:40:21.928573 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:40:21.928553    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79d45f973b10415307977b3ddc777d4b01ddd29339ff3bb0c75497ff283a8ab3\": container with ID starting with 79d45f973b10415307977b3ddc777d4b01ddd29339ff3bb0c75497ff283a8ab3 not found: ID does not exist" containerID="79d45f973b10415307977b3ddc777d4b01ddd29339ff3bb0c75497ff283a8ab3"
Sep 10 14:40:21.928617 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.928580    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d45f973b10415307977b3ddc777d4b01ddd29339ff3bb0c75497ff283a8ab3"} err="failed to get container status \"79d45f973b10415307977b3ddc777d4b01ddd29339ff3bb0c75497ff283a8ab3\": rpc error: code = NotFound desc = could not find container \"79d45f973b10415307977b3ddc777d4b01ddd29339ff3bb0c75497ff283a8ab3\": container with ID starting with 79d45f973b10415307977b3ddc777d4b01ddd29339ff3bb0c75497ff283a8ab3 not found: ID does not exist"
Sep 10 14:40:21.928617 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.928596    2547 scope.go:117] "RemoveContainer" containerID="6649387f7690a64394db6155bdd33eab50c0d7460f12f0e939a7cabd0758bcfe"
Sep 10 14:40:21.928852 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:40:21.928832    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6649387f7690a64394db6155bdd33eab50c0d7460f12f0e939a7cabd0758bcfe\": container with ID starting with 6649387f7690a64394db6155bdd33eab50c0d7460f12f0e939a7cabd0758bcfe not found: ID does not exist" containerID="6649387f7690a64394db6155bdd33eab50c0d7460f12f0e939a7cabd0758bcfe"
Sep 10 14:40:21.928967 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.928857    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6649387f7690a64394db6155bdd33eab50c0d7460f12f0e939a7cabd0758bcfe"} err="failed to get container status \"6649387f7690a64394db6155bdd33eab50c0d7460f12f0e939a7cabd0758bcfe\": rpc error: code = NotFound desc = could not find container \"6649387f7690a64394db6155bdd33eab50c0d7460f12f0e939a7cabd0758bcfe\": container with ID starting with 6649387f7690a64394db6155bdd33eab50c0d7460f12f0e939a7cabd0758bcfe not found: ID does not exist"
Sep 10 14:40:21.932082 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.932058    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"]
Sep 10 14:40:21.936459 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:21.936439    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-7e4309cc-gw-openshift-default-78998b5d48-v2wx8"]
Sep 10 14:40:22.421902 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:22.421875    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ba57f95-0387-49e9-ba43-06a6c02fc3d3" path="/var/lib/kubelet/pods/2ba57f95-0387-49e9-ba43-06a6c02fc3d3/volumes"
Sep 10 14:40:24.607670 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.607639    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"]
Sep 10 14:40:24.608041 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.608013    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ba57f95-0387-49e9-ba43-06a6c02fc3d3" containerName="istio-proxy"
Sep 10 14:40:24.608041 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.608025    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ba57f95-0387-49e9-ba43-06a6c02fc3d3" containerName="istio-proxy"
Sep 10 14:40:24.608128 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.608095    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ba57f95-0387-49e9-ba43-06a6c02fc3d3" containerName="istio-proxy"
Sep 10 14:40:24.608128 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.608103    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ba57f95-0387-49e9-ba43-06a6c02fc3d3" containerName="istio-proxy"
Sep 10 14:40:24.612510 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.612490    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:40:24.614908 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.614880    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c892fe\"/\"kube-root-ca.crt\""
Sep 10 14:40:24.615965 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.615941    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c892fe\"/\"default-dockercfg-smf85\""
Sep 10 14:40:24.616075 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.616042    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c892fe\"/\"test-model-c892fe-kserve-self-signed-certs\""
Sep 10 14:40:24.616153 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.615956    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c892fe\"/\"openshift-service-ca.crt\""
Sep 10 14:40:24.618204 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.618183    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-tmp-dir\") pod \"test-model-c892fe-kserve-68cf795699-xbd95\" (UID: \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\") " pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:40:24.618329 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.618212    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-model-cache\") pod \"test-model-c892fe-kserve-68cf795699-xbd95\" (UID: \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\") " pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:40:24.618329 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.618230    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-tls-certs\") pod \"test-model-c892fe-kserve-68cf795699-xbd95\" (UID: \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\") " pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:40:24.618329 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.618258    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-dshm\") pod \"test-model-c892fe-kserve-68cf795699-xbd95\" (UID: \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\") " pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:40:24.618329 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.618326    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-home\") pod \"test-model-c892fe-kserve-68cf795699-xbd95\" (UID: \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\") " pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:40:24.618566 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.618396    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v97dg\" (UniqueName: \"kubernetes.io/projected/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-kube-api-access-v97dg\") pod \"test-model-c892fe-kserve-68cf795699-xbd95\" (UID: \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\") " pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:40:24.619240 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.619221    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"]
Sep 10 14:40:24.719810 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.719760    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-tmp-dir\") pod \"test-model-c892fe-kserve-68cf795699-xbd95\" (UID: \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\") " pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:40:24.719810 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.719812    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-model-cache\") pod \"test-model-c892fe-kserve-68cf795699-xbd95\" (UID: \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\") " pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:40:24.720042 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.719936    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-tls-certs\") pod \"test-model-c892fe-kserve-68cf795699-xbd95\" (UID: \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\") " pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:40:24.720042 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.719973    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-dshm\") pod \"test-model-c892fe-kserve-68cf795699-xbd95\" (UID: \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\") " pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:40:24.720156 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.720089    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-home\") pod \"test-model-c892fe-kserve-68cf795699-xbd95\" (UID: \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\") " pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:40:24.720216 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.720147    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-tmp-dir\") pod \"test-model-c892fe-kserve-68cf795699-xbd95\" (UID: \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\") " pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:40:24.720276 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.720211    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-model-cache\") pod \"test-model-c892fe-kserve-68cf795699-xbd95\" (UID: \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\") " pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:40:24.720276 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.720242    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v97dg\" (UniqueName: \"kubernetes.io/projected/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-kube-api-access-v97dg\") pod \"test-model-c892fe-kserve-68cf795699-xbd95\" (UID: \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\") " pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:40:24.720417 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.720399    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-home\") pod \"test-model-c892fe-kserve-68cf795699-xbd95\" (UID: \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\") " pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:40:24.722106 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.722088    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-dshm\") pod \"test-model-c892fe-kserve-68cf795699-xbd95\" (UID: \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\") " pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:40:24.722359 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.722341    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-tls-certs\") pod \"test-model-c892fe-kserve-68cf795699-xbd95\" (UID: \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\") " pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:40:24.728141 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.728118    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v97dg\" (UniqueName: \"kubernetes.io/projected/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-kube-api-access-v97dg\") pod \"test-model-c892fe-kserve-68cf795699-xbd95\" (UID: \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\") " pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:40:24.922810 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:24.922770    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:40:25.193279 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:25.193249    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"]
Sep 10 14:40:25.194148 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:40:25.194122    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d5094f1_230b_4ca1_a6a9_e99a92105b4f.slice/crio-a5c6eb2c6742e382d8e01ff65fbb58f633d80485585318bba980327d526d65e1 WatchSource:0}: Error finding container a5c6eb2c6742e382d8e01ff65fbb58f633d80485585318bba980327d526d65e1: Status 404 returned error can't find the container with id a5c6eb2c6742e382d8e01ff65fbb58f633d80485585318bba980327d526d65e1
Sep 10 14:40:25.934032 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:25.933993    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95" event={"ID":"8d5094f1-230b-4ca1-a6a9-e99a92105b4f","Type":"ContainerStarted","Data":"3efe60404fbb54054804ad27762f9ee208c5bfbf237f44b96bd485c31ccdeae4"}
Sep 10 14:40:25.934426 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:25.934041    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95" event={"ID":"8d5094f1-230b-4ca1-a6a9-e99a92105b4f","Type":"ContainerStarted","Data":"a5c6eb2c6742e382d8e01ff65fbb58f633d80485585318bba980327d526d65e1"}
Sep 10 14:40:25.951171 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:25.951121    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95" podStartSLOduration=1.951105751 podStartE2EDuration="1.951105751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:40:25.94975682 +0000 UTC m=+1237.991631300" watchObservedRunningTime="2026-09-10 14:40:25.951105751 +0000 UTC m=+1237.992980239"
Sep 10 14:40:34.923828 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:34.923781    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:40:34.923828 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:34.923838    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:40:34.935954 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:34.935923    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:40:34.995404 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:34.995359    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:40:35.330886 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:35.330802    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:40:40.405398 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.405359    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"]
Sep 10 14:40:40.405846 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.405774    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ba57f95-0387-49e9-ba43-06a6c02fc3d3" containerName="istio-proxy"
Sep 10 14:40:40.405846 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.405786    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ba57f95-0387-49e9-ba43-06a6c02fc3d3" containerName="istio-proxy"
Sep 10 14:40:40.409146 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.409129    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"
Sep 10 14:40:40.411407 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.411386    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-61b16dfc-kserve-self-signed-certs\""
Sep 10 14:40:40.424462 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.424437    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"]
Sep 10 14:40:40.460206 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.460173    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902bc4c0-5df2-4cb0-815b-ed68543ae202-model-cache\") pod \"e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b\" (UID: \"902bc4c0-5df2-4cb0-815b-ed68543ae202\") " pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"
Sep 10 14:40:40.460374 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.460214    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902bc4c0-5df2-4cb0-815b-ed68543ae202-tmp-dir\") pod \"e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b\" (UID: \"902bc4c0-5df2-4cb0-815b-ed68543ae202\") " pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"
Sep 10 14:40:40.460374 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.460236    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902bc4c0-5df2-4cb0-815b-ed68543ae202-home\") pod \"e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b\" (UID: \"902bc4c0-5df2-4cb0-815b-ed68543ae202\") " pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"
Sep 10 14:40:40.460374 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.460276    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902bc4c0-5df2-4cb0-815b-ed68543ae202-dshm\") pod \"e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b\" (UID: \"902bc4c0-5df2-4cb0-815b-ed68543ae202\") " pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"
Sep 10 14:40:40.460374 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.460317    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902bc4c0-5df2-4cb0-815b-ed68543ae202-tls-certs\") pod \"e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b\" (UID: \"902bc4c0-5df2-4cb0-815b-ed68543ae202\") " pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"
Sep 10 14:40:40.460374 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.460347    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbrsq\" (UniqueName: \"kubernetes.io/projected/902bc4c0-5df2-4cb0-815b-ed68543ae202-kube-api-access-xbrsq\") pod \"e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b\" (UID: \"902bc4c0-5df2-4cb0-815b-ed68543ae202\") " pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"
Sep 10 14:40:40.561744 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.561673    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902bc4c0-5df2-4cb0-815b-ed68543ae202-tmp-dir\") pod \"e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b\" (UID: \"902bc4c0-5df2-4cb0-815b-ed68543ae202\") " pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"
Sep 10 14:40:40.561939 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.561755    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902bc4c0-5df2-4cb0-815b-ed68543ae202-home\") pod \"e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b\" (UID: \"902bc4c0-5df2-4cb0-815b-ed68543ae202\") " pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"
Sep 10 14:40:40.561939 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.561773    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902bc4c0-5df2-4cb0-815b-ed68543ae202-dshm\") pod \"e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b\" (UID: \"902bc4c0-5df2-4cb0-815b-ed68543ae202\") " pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"
Sep 10 14:40:40.561939 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.561796    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902bc4c0-5df2-4cb0-815b-ed68543ae202-tls-certs\") pod \"e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b\" (UID: \"902bc4c0-5df2-4cb0-815b-ed68543ae202\") " pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"
Sep 10 14:40:40.561939 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.561816    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbrsq\" (UniqueName: \"kubernetes.io/projected/902bc4c0-5df2-4cb0-815b-ed68543ae202-kube-api-access-xbrsq\") pod \"e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b\" (UID: \"902bc4c0-5df2-4cb0-815b-ed68543ae202\") " pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"
Sep 10 14:40:40.561939 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.561898    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902bc4c0-5df2-4cb0-815b-ed68543ae202-model-cache\") pod \"e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b\" (UID: \"902bc4c0-5df2-4cb0-815b-ed68543ae202\") " pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"
Sep 10 14:40:40.562203 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.562132    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902bc4c0-5df2-4cb0-815b-ed68543ae202-tmp-dir\") pod \"e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b\" (UID: \"902bc4c0-5df2-4cb0-815b-ed68543ae202\") " pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"
Sep 10 14:40:40.562273 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.562206    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902bc4c0-5df2-4cb0-815b-ed68543ae202-home\") pod \"e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b\" (UID: \"902bc4c0-5df2-4cb0-815b-ed68543ae202\") " pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"
Sep 10 14:40:40.562335 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.562316    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902bc4c0-5df2-4cb0-815b-ed68543ae202-model-cache\") pod \"e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b\" (UID: \"902bc4c0-5df2-4cb0-815b-ed68543ae202\") " pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"
Sep 10 14:40:40.564142 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.564118    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902bc4c0-5df2-4cb0-815b-ed68543ae202-dshm\") pod \"e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b\" (UID: \"902bc4c0-5df2-4cb0-815b-ed68543ae202\") " pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"
Sep 10 14:40:40.564396 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.564380    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902bc4c0-5df2-4cb0-815b-ed68543ae202-tls-certs\") pod \"e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b\" (UID: \"902bc4c0-5df2-4cb0-815b-ed68543ae202\") " pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"
Sep 10 14:40:40.570176 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.570157    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbrsq\" (UniqueName: \"kubernetes.io/projected/902bc4c0-5df2-4cb0-815b-ed68543ae202-kube-api-access-xbrsq\") pod \"e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b\" (UID: \"902bc4c0-5df2-4cb0-815b-ed68543ae202\") " pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"
Sep 10 14:40:40.720422 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.720385    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"
Sep 10 14:40:40.994066 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:40.993959    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"]
Sep 10 14:40:40.997639 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:40:40.997612    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod902bc4c0_5df2_4cb0_815b_ed68543ae202.slice/crio-2f770c3a5cb409eb1eb61f0aeae1df17aeb2149ce8dd4f214498e11ceffa6d5c WatchSource:0}: Error finding container 2f770c3a5cb409eb1eb61f0aeae1df17aeb2149ce8dd4f214498e11ceffa6d5c: Status 404 returned error can't find the container with id 2f770c3a5cb409eb1eb61f0aeae1df17aeb2149ce8dd4f214498e11ceffa6d5c
Sep 10 14:40:41.023756 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:41.023728    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b" event={"ID":"902bc4c0-5df2-4cb0-815b-ed68543ae202","Type":"ContainerStarted","Data":"2f770c3a5cb409eb1eb61f0aeae1df17aeb2149ce8dd4f214498e11ceffa6d5c"}
Sep 10 14:40:41.048183 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:41.047492    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e7415a-s8k4r"]
Sep 10 14:40:41.055168 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:41.055142    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e7415a-s8k4r"
Sep 10 14:40:41.056260 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:41.056234    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e7415a-s8k4r"]
Sep 10 14:40:41.058114 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:41.058095    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 14:40:41.058262 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:41.058234    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xnxsg\""
Sep 10 14:40:41.167306 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:41.167271    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7d9f7a-4b03-4631-b0d0-8730b01d73b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-e7415a-s8k4r\" (UID: \"8a7d9f7a-4b03-4631-b0d0-8730b01d73b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e7415a-s8k4r"
Sep 10 14:40:41.268762 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:41.268669    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7d9f7a-4b03-4631-b0d0-8730b01d73b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-e7415a-s8k4r\" (UID: \"8a7d9f7a-4b03-4631-b0d0-8730b01d73b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e7415a-s8k4r"
Sep 10 14:40:41.269285 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:41.269264    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7d9f7a-4b03-4631-b0d0-8730b01d73b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-e7415a-s8k4r\" (UID: \"8a7d9f7a-4b03-4631-b0d0-8730b01d73b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e7415a-s8k4r"
Sep 10 14:40:41.369290 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:41.369249    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e7415a-s8k4r"
Sep 10 14:40:41.639189 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:41.639159    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e7415a-s8k4r"]
Sep 10 14:40:41.640992 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:40:41.640965    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7d9f7a_4b03_4631_b0d0_8730b01d73b7.slice/crio-7e534b86f4e4385dfee370b6196b384c37be1d36c9997b4faf65fb57dfa99666 WatchSource:0}: Error finding container 7e534b86f4e4385dfee370b6196b384c37be1d36c9997b4faf65fb57dfa99666: Status 404 returned error can't find the container with id 7e534b86f4e4385dfee370b6196b384c37be1d36c9997b4faf65fb57dfa99666
Sep 10 14:40:42.033364 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:42.033328    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b" event={"ID":"902bc4c0-5df2-4cb0-815b-ed68543ae202","Type":"ContainerStarted","Data":"4d86261296422c424bf422c714ddbeb944d61aa55aca04205d1ae41af95bfcdb"}
Sep 10 14:40:42.034936 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:42.034906    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e7415a-s8k4r" event={"ID":"8a7d9f7a-4b03-4631-b0d0-8730b01d73b7","Type":"ContainerStarted","Data":"d2c1393b630b45e96d89fc2c271974987f7ed4317f68e5369af55ac6a0d270f1"}
Sep 10 14:40:42.035069 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:42.034954    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e7415a-s8k4r" event={"ID":"8a7d9f7a-4b03-4631-b0d0-8730b01d73b7","Type":"ContainerStarted","Data":"7e534b86f4e4385dfee370b6196b384c37be1d36c9997b4faf65fb57dfa99666"}
Sep 10 14:40:42.050882 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:42.050826    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b" podStartSLOduration=2.050811297 podStartE2EDuration="2.050811297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:40:42.049048319 +0000 UTC m=+1254.090922801" watchObservedRunningTime="2026-09-10 14:40:42.050811297 +0000 UTC m=+1254.092685777"
Sep 10 14:40:42.063172 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:42.063127    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e7415a-s8k4r" podStartSLOduration=1.063113301 podStartE2EDuration="1.063113301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:40:42.062549896 +0000 UTC m=+1254.104424378" watchObservedRunningTime="2026-09-10 14:40:42.063113301 +0000 UTC m=+1254.104987783"
Sep 10 14:40:43.042722 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:43.042668    2547 generic.go:356] "Generic (PLEG): container finished" podID="8a7d9f7a-4b03-4631-b0d0-8730b01d73b7" containerID="d2c1393b630b45e96d89fc2c271974987f7ed4317f68e5369af55ac6a0d270f1" exitCode=0
Sep 10 14:40:43.043148 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:43.042730    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e7415a-s8k4r" event={"ID":"8a7d9f7a-4b03-4631-b0d0-8730b01d73b7","Type":"ContainerDied","Data":"d2c1393b630b45e96d89fc2c271974987f7ed4317f68e5369af55ac6a0d270f1"}
Sep 10 14:40:43.361617 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:43.361535    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"]
Sep 10 14:40:44.181649 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:44.181622    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e7415a-s8k4r"
Sep 10 14:40:44.296205 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:44.296175    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7d9f7a-4b03-4631-b0d0-8730b01d73b7-maas-api-service-ca\") pod \"8a7d9f7a-4b03-4631-b0d0-8730b01d73b7\" (UID: \"8a7d9f7a-4b03-4631-b0d0-8730b01d73b7\") "
Sep 10 14:40:44.296580 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:44.296559    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7d9f7a-4b03-4631-b0d0-8730b01d73b7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8a7d9f7a-4b03-4631-b0d0-8730b01d73b7" (UID: "8a7d9f7a-4b03-4631-b0d0-8730b01d73b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:40:44.397716 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:44.397614    2547 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7d9f7a-4b03-4631-b0d0-8730b01d73b7-maas-api-service-ca\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:40:44.942850 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:44.942812    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04dfc68a-sg82m"]
Sep 10 14:40:44.943221 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:44.943209    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a7d9f7a-4b03-4631-b0d0-8730b01d73b7" containerName="revoke-keys"
Sep 10 14:40:44.943265 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:44.943222    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a7d9f7a-4b03-4631-b0d0-8730b01d73b7" containerName="revoke-keys"
Sep 10 14:40:44.943300 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:44.943281    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a7d9f7a-4b03-4631-b0d0-8730b01d73b7" containerName="revoke-keys"
Sep 10 14:40:44.946170 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:44.946154    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04dfc68a-sg82m"
Sep 10 14:40:44.954606 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:44.954576    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04dfc68a-sg82m"]
Sep 10 14:40:45.002360 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:45.002314    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb327a23-885e-4956-800e-147630433bce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-04dfc68a-sg82m\" (UID: \"fb327a23-885e-4956-800e-147630433bce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04dfc68a-sg82m"
Sep 10 14:40:45.054711 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:45.054673    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e7415a-s8k4r"
Sep 10 14:40:45.054877 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:45.054673    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e7415a-s8k4r" event={"ID":"8a7d9f7a-4b03-4631-b0d0-8730b01d73b7","Type":"ContainerDied","Data":"7e534b86f4e4385dfee370b6196b384c37be1d36c9997b4faf65fb57dfa99666"}
Sep 10 14:40:45.054877 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:45.054815    2547 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e534b86f4e4385dfee370b6196b384c37be1d36c9997b4faf65fb57dfa99666"
Sep 10 14:40:45.103746 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:45.103683    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb327a23-885e-4956-800e-147630433bce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-04dfc68a-sg82m\" (UID: \"fb327a23-885e-4956-800e-147630433bce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04dfc68a-sg82m"
Sep 10 14:40:45.104368 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:45.104343    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb327a23-885e-4956-800e-147630433bce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-04dfc68a-sg82m\" (UID: \"fb327a23-885e-4956-800e-147630433bce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04dfc68a-sg82m"
Sep 10 14:40:45.257296 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:45.257212    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04dfc68a-sg82m"
Sep 10 14:40:45.531739 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:45.531713    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04dfc68a-sg82m"]
Sep 10 14:40:45.533651 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:40:45.533627    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb327a23_885e_4956_800e_147630433bce.slice/crio-94690529a8a443acac8eb2150c2ac6cfbbc73ed74f1448474723413e7cb8f544 WatchSource:0}: Error finding container 94690529a8a443acac8eb2150c2ac6cfbbc73ed74f1448474723413e7cb8f544: Status 404 returned error can't find the container with id 94690529a8a443acac8eb2150c2ac6cfbbc73ed74f1448474723413e7cb8f544
Sep 10 14:40:46.063647 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:46.063612    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04dfc68a-sg82m" event={"ID":"fb327a23-885e-4956-800e-147630433bce","Type":"ContainerStarted","Data":"b4a7fe77397bda021f219cfd669d0e41dff2bf92f773b3adde268e33daae0b4f"}
Sep 10 14:40:46.063820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:46.063671    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04dfc68a-sg82m" event={"ID":"fb327a23-885e-4956-800e-147630433bce","Type":"ContainerStarted","Data":"94690529a8a443acac8eb2150c2ac6cfbbc73ed74f1448474723413e7cb8f544"}
Sep 10 14:40:46.082346 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:46.082302    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04dfc68a-sg82m" podStartSLOduration=2.082288628 podStartE2EDuration="2.082288628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:40:46.080651566 +0000 UTC m=+1258.122526048" watchObservedRunningTime="2026-09-10 14:40:46.082288628 +0000 UTC m=+1258.124163108"
Sep 10 14:40:47.071214 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:47.071181    2547 generic.go:356] "Generic (PLEG): container finished" podID="fb327a23-885e-4956-800e-147630433bce" containerID="b4a7fe77397bda021f219cfd669d0e41dff2bf92f773b3adde268e33daae0b4f" exitCode=0
Sep 10 14:40:47.071587 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:47.071279    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04dfc68a-sg82m" event={"ID":"fb327a23-885e-4956-800e-147630433bce","Type":"ContainerDied","Data":"b4a7fe77397bda021f219cfd669d0e41dff2bf92f773b3adde268e33daae0b4f"}
Sep 10 14:40:47.943912 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:47.943872    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"]
Sep 10 14:40:48.208230 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:48.208171    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04dfc68a-sg82m"
Sep 10 14:40:48.335251 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:48.335221    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb327a23-885e-4956-800e-147630433bce-maas-api-service-ca\") pod \"fb327a23-885e-4956-800e-147630433bce\" (UID: \"fb327a23-885e-4956-800e-147630433bce\") "
Sep 10 14:40:48.335540 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:48.335517    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb327a23-885e-4956-800e-147630433bce-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fb327a23-885e-4956-800e-147630433bce" (UID: "fb327a23-885e-4956-800e-147630433bce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:40:48.436806 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:48.436727    2547 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb327a23-885e-4956-800e-147630433bce-maas-api-service-ca\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:40:49.084160 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:49.084135    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04dfc68a-sg82m"
Sep 10 14:40:49.084160 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:49.084150    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04dfc68a-sg82m" event={"ID":"fb327a23-885e-4956-800e-147630433bce","Type":"ContainerDied","Data":"94690529a8a443acac8eb2150c2ac6cfbbc73ed74f1448474723413e7cb8f544"}
Sep 10 14:40:49.084369 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:49.084220    2547 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94690529a8a443acac8eb2150c2ac6cfbbc73ed74f1448474723413e7cb8f544"
Sep 10 14:40:49.676882 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:49.676850    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"]
Sep 10 14:40:50.720806 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:50.720768    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"
Sep 10 14:40:51.474027 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:51.473995    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e7415a-s8k4r"]
Sep 10 14:40:51.478178 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:51.478147    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e7415a-s8k4r"]
Sep 10 14:40:51.657609 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:51.657570    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04dfc68a-sg82m"]
Sep 10 14:40:51.661158 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:51.661031    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04dfc68a-sg82m"]
Sep 10 14:40:52.421762 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:52.421724    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a7d9f7a-4b03-4631-b0d0-8730b01d73b7" path="/var/lib/kubelet/pods/8a7d9f7a-4b03-4631-b0d0-8730b01d73b7/volumes"
Sep 10 14:40:52.422112 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:52.422065    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb327a23-885e-4956-800e-147630433bce" path="/var/lib/kubelet/pods/fb327a23-885e-4956-800e-147630433bce/volumes"
Sep 10 14:40:54.621622 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:54.621585    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:40:58.384066 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:58.384027    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954" podUID="162c423a-2c9e-4bdf-bea6-51ab402a5f94" containerName="main" containerID="cri-o://ccfa860bb8d03112e6c521615183f5221eb7fcdeec7e2936624fee5f24857999" gracePeriod=45
Sep 10 14:40:58.632785 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:58.632753    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:58.737672 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:58.737632    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tx77z\" (UniqueName: \"kubernetes.io/projected/162c423a-2c9e-4bdf-bea6-51ab402a5f94-kube-api-access-tx77z\") pod \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\" (UID: \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\") "
Sep 10 14:40:58.737898 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:58.737680    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/162c423a-2c9e-4bdf-bea6-51ab402a5f94-model-cache\") pod \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\" (UID: \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\") "
Sep 10 14:40:58.737898 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:58.737824    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/162c423a-2c9e-4bdf-bea6-51ab402a5f94-tls-certs\") pod \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\" (UID: \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\") "
Sep 10 14:40:58.737898 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:58.737864    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/162c423a-2c9e-4bdf-bea6-51ab402a5f94-home\") pod \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\" (UID: \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\") "
Sep 10 14:40:58.738057 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:58.737911    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/162c423a-2c9e-4bdf-bea6-51ab402a5f94-tmp-dir\") pod \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\" (UID: \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\") "
Sep 10 14:40:58.738057 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:58.737946    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/162c423a-2c9e-4bdf-bea6-51ab402a5f94-dshm\") pod \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\" (UID: \"162c423a-2c9e-4bdf-bea6-51ab402a5f94\") "
Sep 10 14:40:58.738057 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:58.737967    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162c423a-2c9e-4bdf-bea6-51ab402a5f94-model-cache" (OuterVolumeSpecName: "model-cache") pod "162c423a-2c9e-4bdf-bea6-51ab402a5f94" (UID: "162c423a-2c9e-4bdf-bea6-51ab402a5f94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:40:58.738208 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:58.738179    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162c423a-2c9e-4bdf-bea6-51ab402a5f94-home" (OuterVolumeSpecName: "home") pod "162c423a-2c9e-4bdf-bea6-51ab402a5f94" (UID: "162c423a-2c9e-4bdf-bea6-51ab402a5f94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:40:58.738208 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:58.738195    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162c423a-2c9e-4bdf-bea6-51ab402a5f94-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "162c423a-2c9e-4bdf-bea6-51ab402a5f94" (UID: "162c423a-2c9e-4bdf-bea6-51ab402a5f94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:40:58.738303 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:58.738256    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/162c423a-2c9e-4bdf-bea6-51ab402a5f94-model-cache\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:40:58.739973 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:58.739928    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162c423a-2c9e-4bdf-bea6-51ab402a5f94-dshm" (OuterVolumeSpecName: "dshm") pod "162c423a-2c9e-4bdf-bea6-51ab402a5f94" (UID: "162c423a-2c9e-4bdf-bea6-51ab402a5f94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:40:58.740089 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:58.739983    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/162c423a-2c9e-4bdf-bea6-51ab402a5f94-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "162c423a-2c9e-4bdf-bea6-51ab402a5f94" (UID: "162c423a-2c9e-4bdf-bea6-51ab402a5f94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:40:58.740089 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:58.740005    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/162c423a-2c9e-4bdf-bea6-51ab402a5f94-kube-api-access-tx77z" (OuterVolumeSpecName: "kube-api-access-tx77z") pod "162c423a-2c9e-4bdf-bea6-51ab402a5f94" (UID: "162c423a-2c9e-4bdf-bea6-51ab402a5f94"). InnerVolumeSpecName "kube-api-access-tx77z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:40:58.839416 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:58.839377    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/162c423a-2c9e-4bdf-bea6-51ab402a5f94-tls-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:40:58.839416 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:58.839410    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/162c423a-2c9e-4bdf-bea6-51ab402a5f94-home\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:40:58.839416 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:58.839418    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/162c423a-2c9e-4bdf-bea6-51ab402a5f94-tmp-dir\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:40:58.839416 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:58.839426    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/162c423a-2c9e-4bdf-bea6-51ab402a5f94-dshm\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:40:58.839680 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:58.839435    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx77z\" (UniqueName: \"kubernetes.io/projected/162c423a-2c9e-4bdf-bea6-51ab402a5f94-kube-api-access-tx77z\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:40:59.144473 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:59.144389    2547 generic.go:356] "Generic (PLEG): container finished" podID="162c423a-2c9e-4bdf-bea6-51ab402a5f94" containerID="ccfa860bb8d03112e6c521615183f5221eb7fcdeec7e2936624fee5f24857999" exitCode=0
Sep 10 14:40:59.144473 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:59.144441    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"
Sep 10 14:40:59.144653 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:59.144475    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954" event={"ID":"162c423a-2c9e-4bdf-bea6-51ab402a5f94","Type":"ContainerDied","Data":"ccfa860bb8d03112e6c521615183f5221eb7fcdeec7e2936624fee5f24857999"}
Sep 10 14:40:59.144653 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:59.144527    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954" event={"ID":"162c423a-2c9e-4bdf-bea6-51ab402a5f94","Type":"ContainerDied","Data":"1503e38f0e02d00030661ecc73143e4ef308b7a808e9a3f1d366101aea919436"}
Sep 10 14:40:59.144653 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:59.144545    2547 scope.go:117] "RemoveContainer" containerID="ccfa860bb8d03112e6c521615183f5221eb7fcdeec7e2936624fee5f24857999"
Sep 10 14:40:59.153880 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:59.153863    2547 scope.go:117] "RemoveContainer" containerID="ccfa860bb8d03112e6c521615183f5221eb7fcdeec7e2936624fee5f24857999"
Sep 10 14:40:59.154144 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:40:59.154123    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccfa860bb8d03112e6c521615183f5221eb7fcdeec7e2936624fee5f24857999\": container with ID starting with ccfa860bb8d03112e6c521615183f5221eb7fcdeec7e2936624fee5f24857999 not found: ID does not exist" containerID="ccfa860bb8d03112e6c521615183f5221eb7fcdeec7e2936624fee5f24857999"
Sep 10 14:40:59.154233 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:59.154152    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccfa860bb8d03112e6c521615183f5221eb7fcdeec7e2936624fee5f24857999"} err="failed to get container status \"ccfa860bb8d03112e6c521615183f5221eb7fcdeec7e2936624fee5f24857999\": rpc error: code = NotFound desc = could not find container \"ccfa860bb8d03112e6c521615183f5221eb7fcdeec7e2936624fee5f24857999\": container with ID starting with ccfa860bb8d03112e6c521615183f5221eb7fcdeec7e2936624fee5f24857999 not found: ID does not exist"
Sep 10 14:40:59.165361 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:59.165339    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"]
Sep 10 14:40:59.168961 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:40:59.168940    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-e7415a/autores-model-e7415a-kserve-7cf48579-48954"]
Sep 10 14:41:00.422094 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:00.422065    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="162c423a-2c9e-4bdf-bea6-51ab402a5f94" path="/var/lib/kubelet/pods/162c423a-2c9e-4bdf-bea6-51ab402a5f94/volumes"
Sep 10 14:41:02.909796 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:02.909759    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"]
Sep 10 14:41:02.910648 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:02.910610    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps" podUID="87cdc6a3-8bc6-4be6-b793-1d1ac852427a" containerName="istio-proxy" containerID="cri-o://4b38606555e2bf66f1cdb4070a89f1c7d7ffa23a554dee02eedae4a94a7f016e" gracePeriod=30
Sep 10 14:41:02.963839 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:02.963778    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b" podUID="902bc4c0-5df2-4cb0-815b-ed68543ae202" containerName="main" containerID="cri-o://4d86261296422c424bf422c714ddbeb944d61aa55aca04205d1ae41af95bfcdb" gracePeriod=45
Sep 10 14:41:03.171633 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.171538    2547 generic.go:356] "Generic (PLEG): container finished" podID="902bc4c0-5df2-4cb0-815b-ed68543ae202" containerID="4d86261296422c424bf422c714ddbeb944d61aa55aca04205d1ae41af95bfcdb" exitCode=0
Sep 10 14:41:03.171822 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.171665    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b" event={"ID":"902bc4c0-5df2-4cb0-815b-ed68543ae202","Type":"ContainerDied","Data":"4d86261296422c424bf422c714ddbeb944d61aa55aca04205d1ae41af95bfcdb"}
Sep 10 14:41:03.219823 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.219792    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"
Sep 10 14:41:03.380689 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.380654    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902bc4c0-5df2-4cb0-815b-ed68543ae202-tmp-dir\") pod \"902bc4c0-5df2-4cb0-815b-ed68543ae202\" (UID: \"902bc4c0-5df2-4cb0-815b-ed68543ae202\") "
Sep 10 14:41:03.380689 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.380716    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902bc4c0-5df2-4cb0-815b-ed68543ae202-model-cache\") pod \"902bc4c0-5df2-4cb0-815b-ed68543ae202\" (UID: \"902bc4c0-5df2-4cb0-815b-ed68543ae202\") "
Sep 10 14:41:03.380971 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.380787    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902bc4c0-5df2-4cb0-815b-ed68543ae202-dshm\") pod \"902bc4c0-5df2-4cb0-815b-ed68543ae202\" (UID: \"902bc4c0-5df2-4cb0-815b-ed68543ae202\") "
Sep 10 14:41:03.380971 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.380806    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902bc4c0-5df2-4cb0-815b-ed68543ae202-home\") pod \"902bc4c0-5df2-4cb0-815b-ed68543ae202\" (UID: \"902bc4c0-5df2-4cb0-815b-ed68543ae202\") "
Sep 10 14:41:03.380971 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.380865    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbrsq\" (UniqueName: \"kubernetes.io/projected/902bc4c0-5df2-4cb0-815b-ed68543ae202-kube-api-access-xbrsq\") pod \"902bc4c0-5df2-4cb0-815b-ed68543ae202\" (UID: \"902bc4c0-5df2-4cb0-815b-ed68543ae202\") "
Sep 10 14:41:03.380971 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.380888    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902bc4c0-5df2-4cb0-815b-ed68543ae202-tls-certs\") pod \"902bc4c0-5df2-4cb0-815b-ed68543ae202\" (UID: \"902bc4c0-5df2-4cb0-815b-ed68543ae202\") "
Sep 10 14:41:03.381203 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.380982    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902bc4c0-5df2-4cb0-815b-ed68543ae202-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "902bc4c0-5df2-4cb0-815b-ed68543ae202" (UID: "902bc4c0-5df2-4cb0-815b-ed68543ae202"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:41:03.381203 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.381052    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902bc4c0-5df2-4cb0-815b-ed68543ae202-model-cache" (OuterVolumeSpecName: "model-cache") pod "902bc4c0-5df2-4cb0-815b-ed68543ae202" (UID: "902bc4c0-5df2-4cb0-815b-ed68543ae202"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:41:03.381203 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.381182    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902bc4c0-5df2-4cb0-815b-ed68543ae202-home" (OuterVolumeSpecName: "home") pod "902bc4c0-5df2-4cb0-815b-ed68543ae202" (UID: "902bc4c0-5df2-4cb0-815b-ed68543ae202"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:41:03.381368 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.381259    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902bc4c0-5df2-4cb0-815b-ed68543ae202-tmp-dir\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:41:03.381368 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.381283    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902bc4c0-5df2-4cb0-815b-ed68543ae202-model-cache\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:41:03.383099 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.383070    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902bc4c0-5df2-4cb0-815b-ed68543ae202-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "902bc4c0-5df2-4cb0-815b-ed68543ae202" (UID: "902bc4c0-5df2-4cb0-815b-ed68543ae202"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:41:03.383210 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.383121    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902bc4c0-5df2-4cb0-815b-ed68543ae202-kube-api-access-xbrsq" (OuterVolumeSpecName: "kube-api-access-xbrsq") pod "902bc4c0-5df2-4cb0-815b-ed68543ae202" (UID: "902bc4c0-5df2-4cb0-815b-ed68543ae202"). InnerVolumeSpecName "kube-api-access-xbrsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:41:03.383210 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.383144    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902bc4c0-5df2-4cb0-815b-ed68543ae202-dshm" (OuterVolumeSpecName: "dshm") pod "902bc4c0-5df2-4cb0-815b-ed68543ae202" (UID: "902bc4c0-5df2-4cb0-815b-ed68543ae202"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:41:03.481993 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.481900    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbrsq\" (UniqueName: \"kubernetes.io/projected/902bc4c0-5df2-4cb0-815b-ed68543ae202-kube-api-access-xbrsq\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:41:03.481993 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.481933    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902bc4c0-5df2-4cb0-815b-ed68543ae202-tls-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:41:03.481993 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.481944    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902bc4c0-5df2-4cb0-815b-ed68543ae202-dshm\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:41:03.481993 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.481951    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902bc4c0-5df2-4cb0-815b-ed68543ae202-home\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:41:03.628984 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.628948    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"]
Sep 10 14:41:03.629357 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.629343    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="162c423a-2c9e-4bdf-bea6-51ab402a5f94" containerName="main"
Sep 10 14:41:03.629398 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.629361    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="162c423a-2c9e-4bdf-bea6-51ab402a5f94" containerName="main"
Sep 10 14:41:03.629398 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.629389    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="902bc4c0-5df2-4cb0-815b-ed68543ae202" containerName="main"
Sep 10 14:41:03.629398 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.629395    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="902bc4c0-5df2-4cb0-815b-ed68543ae202" containerName="main"
Sep 10 14:41:03.629482 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.629406    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb327a23-885e-4956-800e-147630433bce" containerName="revoke-keys"
Sep 10 14:41:03.629482 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.629412    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb327a23-885e-4956-800e-147630433bce" containerName="revoke-keys"
Sep 10 14:41:03.629482 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.629471    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="162c423a-2c9e-4bdf-bea6-51ab402a5f94" containerName="main"
Sep 10 14:41:03.629482 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.629480    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="902bc4c0-5df2-4cb0-815b-ed68543ae202" containerName="main"
Sep 10 14:41:03.629603 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.629486    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb327a23-885e-4956-800e-147630433bce" containerName="revoke-keys"
Sep 10 14:41:03.632839 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.632815    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.635376 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.635355    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-b2dfe7-openshift-default-dockercfg-sghgr\""
Sep 10 14:41:03.644261 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.644238    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"]
Sep 10 14:41:03.785011 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.784928    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istio-token\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.785011 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.784971    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-credential-socket\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.785011 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.785002    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-workload-socket\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.785254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.785052    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-workload-certs\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.785254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.785099    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istiod-ca-cert\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.785254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.785152    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7nmc\" (UniqueName: \"kubernetes.io/projected/0f1b2b1f-ff80-4c73-ac93-edab890862c0-kube-api-access-r7nmc\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.785254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.785174    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istio-envoy\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.785254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.785198    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istio-podinfo\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.785254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.785216    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istio-data\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.886772 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.886732    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istio-token\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.886772 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.886774    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-credential-socket\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.886979 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.886814    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-workload-socket\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.886979 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.886841    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-workload-certs\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.886979 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.886867    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istiod-ca-cert\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.886979 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.886944    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7nmc\" (UniqueName: \"kubernetes.io/projected/0f1b2b1f-ff80-4c73-ac93-edab890862c0-kube-api-access-r7nmc\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.886979 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.886971    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istio-envoy\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.887220 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.887003    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istio-podinfo\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.887220 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.887123    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istio-data\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.887330 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.887233    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-credential-socket\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.887330 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.887269    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-workload-socket\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.887330 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.887298    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-workload-certs\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.887520 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.887497    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istio-data\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.887776 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.887759    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istiod-ca-cert\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.889266 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.889242    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istio-envoy\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.889388 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.889370    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istio-podinfo\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.894912 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.894891    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istio-token\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.895616 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.895597    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7nmc\" (UniqueName: \"kubernetes.io/projected/0f1b2b1f-ff80-4c73-ac93-edab890862c0-kube-api-access-r7nmc\") pod \"e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") " pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:03.944269 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:03.944232    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:04.179043 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:04.179013    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"
Sep 10 14:41:04.179043 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:04.179019    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b" event={"ID":"902bc4c0-5df2-4cb0-815b-ed68543ae202","Type":"ContainerDied","Data":"2f770c3a5cb409eb1eb61f0aeae1df17aeb2149ce8dd4f214498e11ceffa6d5c"}
Sep 10 14:41:04.179281 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:04.179073    2547 scope.go:117] "RemoveContainer" containerID="4d86261296422c424bf422c714ddbeb944d61aa55aca04205d1ae41af95bfcdb"
Sep 10 14:41:04.201627 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:04.201603    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"]
Sep 10 14:41:04.207894 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:04.207868    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-61b16dfc-kserve-7f7f6966d4-7vr9b"]
Sep 10 14:41:04.221555 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:04.221529    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"]
Sep 10 14:41:04.222424 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:41:04.222403    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1b2b1f_ff80_4c73_ac93_edab890862c0.slice/crio-0b3b0dd7126e0acda90c3a9f97bce187e8a3dc2aebd2725f3a4604c9eff940ec WatchSource:0}: Error finding container 0b3b0dd7126e0acda90c3a9f97bce187e8a3dc2aebd2725f3a4604c9eff940ec: Status 404 returned error can't find the container with id 0b3b0dd7126e0acda90c3a9f97bce187e8a3dc2aebd2725f3a4604c9eff940ec
Sep 10 14:41:04.225088 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:04.225060    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:41:04.225138 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:04.225122    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:41:04.225184 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:04.225147    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:41:04.422640 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:04.422604    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="902bc4c0-5df2-4cb0-815b-ed68543ae202" path="/var/lib/kubelet/pods/902bc4c0-5df2-4cb0-815b-ed68543ae202/volumes"
Sep 10 14:41:04.696147 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:04.696090    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx" podUID="0cb48298-206b-4bf1-90d4-4baa11f6c854" containerName="main" containerID="cri-o://660d4a4bf57384c2ceac1ad674a510b7ab4c2b04ecb5642386ddd9841ec1f6d0" gracePeriod=45
Sep 10 14:41:04.948484 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:04.948432    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:41:05.099109 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.099078    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb48298-206b-4bf1-90d4-4baa11f6c854-tls-certs\") pod \"0cb48298-206b-4bf1-90d4-4baa11f6c854\" (UID: \"0cb48298-206b-4bf1-90d4-4baa11f6c854\") "
Sep 10 14:41:05.099270 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.099125    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sj5d\" (UniqueName: \"kubernetes.io/projected/0cb48298-206b-4bf1-90d4-4baa11f6c854-kube-api-access-2sj5d\") pod \"0cb48298-206b-4bf1-90d4-4baa11f6c854\" (UID: \"0cb48298-206b-4bf1-90d4-4baa11f6c854\") "
Sep 10 14:41:05.099270 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.099162    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb48298-206b-4bf1-90d4-4baa11f6c854-home\") pod \"0cb48298-206b-4bf1-90d4-4baa11f6c854\" (UID: \"0cb48298-206b-4bf1-90d4-4baa11f6c854\") "
Sep 10 14:41:05.099270 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.099195    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb48298-206b-4bf1-90d4-4baa11f6c854-dshm\") pod \"0cb48298-206b-4bf1-90d4-4baa11f6c854\" (UID: \"0cb48298-206b-4bf1-90d4-4baa11f6c854\") "
Sep 10 14:41:05.099270 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.099228    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb48298-206b-4bf1-90d4-4baa11f6c854-tmp-dir\") pod \"0cb48298-206b-4bf1-90d4-4baa11f6c854\" (UID: \"0cb48298-206b-4bf1-90d4-4baa11f6c854\") "
Sep 10 14:41:05.099270 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.099273    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb48298-206b-4bf1-90d4-4baa11f6c854-model-cache\") pod \"0cb48298-206b-4bf1-90d4-4baa11f6c854\" (UID: \"0cb48298-206b-4bf1-90d4-4baa11f6c854\") "
Sep 10 14:41:05.099554 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.099459    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb48298-206b-4bf1-90d4-4baa11f6c854-home" (OuterVolumeSpecName: "home") pod "0cb48298-206b-4bf1-90d4-4baa11f6c854" (UID: "0cb48298-206b-4bf1-90d4-4baa11f6c854"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:41:05.099624 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.099587    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb48298-206b-4bf1-90d4-4baa11f6c854-home\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:41:05.099624 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.099565    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb48298-206b-4bf1-90d4-4baa11f6c854-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0cb48298-206b-4bf1-90d4-4baa11f6c854" (UID: "0cb48298-206b-4bf1-90d4-4baa11f6c854"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:41:05.099731 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.099680    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb48298-206b-4bf1-90d4-4baa11f6c854-model-cache" (OuterVolumeSpecName: "model-cache") pod "0cb48298-206b-4bf1-90d4-4baa11f6c854" (UID: "0cb48298-206b-4bf1-90d4-4baa11f6c854"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:41:05.101447 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.101419    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb48298-206b-4bf1-90d4-4baa11f6c854-kube-api-access-2sj5d" (OuterVolumeSpecName: "kube-api-access-2sj5d") pod "0cb48298-206b-4bf1-90d4-4baa11f6c854" (UID: "0cb48298-206b-4bf1-90d4-4baa11f6c854"). InnerVolumeSpecName "kube-api-access-2sj5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:41:05.101558 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.101507    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb48298-206b-4bf1-90d4-4baa11f6c854-dshm" (OuterVolumeSpecName: "dshm") pod "0cb48298-206b-4bf1-90d4-4baa11f6c854" (UID: "0cb48298-206b-4bf1-90d4-4baa11f6c854"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:41:05.101558 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.101503    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb48298-206b-4bf1-90d4-4baa11f6c854-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0cb48298-206b-4bf1-90d4-4baa11f6c854" (UID: "0cb48298-206b-4bf1-90d4-4baa11f6c854"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:41:05.186521 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.186479    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9" event={"ID":"0f1b2b1f-ff80-4c73-ac93-edab890862c0","Type":"ContainerStarted","Data":"e5e550b71347cf64774f0cf58565653f9df3e227c7ca23fa04b24c5ee611999e"}
Sep 10 14:41:05.186718 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.186545    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9" event={"ID":"0f1b2b1f-ff80-4c73-ac93-edab890862c0","Type":"ContainerStarted","Data":"0b3b0dd7126e0acda90c3a9f97bce187e8a3dc2aebd2725f3a4604c9eff940ec"}
Sep 10 14:41:05.187858 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.187835    2547 generic.go:356] "Generic (PLEG): container finished" podID="0cb48298-206b-4bf1-90d4-4baa11f6c854" containerID="660d4a4bf57384c2ceac1ad674a510b7ab4c2b04ecb5642386ddd9841ec1f6d0" exitCode=0
Sep 10 14:41:05.187976 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.187869    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"
Sep 10 14:41:05.187976 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.187922    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx" event={"ID":"0cb48298-206b-4bf1-90d4-4baa11f6c854","Type":"ContainerDied","Data":"660d4a4bf57384c2ceac1ad674a510b7ab4c2b04ecb5642386ddd9841ec1f6d0"}
Sep 10 14:41:05.187976 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.187961    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx" event={"ID":"0cb48298-206b-4bf1-90d4-4baa11f6c854","Type":"ContainerDied","Data":"03e3d53d86425410ab265390b4c09ce24c455503ad43f2d9b00cdb4324225c95"}
Sep 10 14:41:05.187976 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.187978    2547 scope.go:117] "RemoveContainer" containerID="660d4a4bf57384c2ceac1ad674a510b7ab4c2b04ecb5642386ddd9841ec1f6d0"
Sep 10 14:41:05.197655 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.197637    2547 scope.go:117] "RemoveContainer" containerID="660d4a4bf57384c2ceac1ad674a510b7ab4c2b04ecb5642386ddd9841ec1f6d0"
Sep 10 14:41:05.197936 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:41:05.197918    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"660d4a4bf57384c2ceac1ad674a510b7ab4c2b04ecb5642386ddd9841ec1f6d0\": container with ID starting with 660d4a4bf57384c2ceac1ad674a510b7ab4c2b04ecb5642386ddd9841ec1f6d0 not found: ID does not exist" containerID="660d4a4bf57384c2ceac1ad674a510b7ab4c2b04ecb5642386ddd9841ec1f6d0"
Sep 10 14:41:05.197992 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.197947    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"660d4a4bf57384c2ceac1ad674a510b7ab4c2b04ecb5642386ddd9841ec1f6d0"} err="failed to get container status \"660d4a4bf57384c2ceac1ad674a510b7ab4c2b04ecb5642386ddd9841ec1f6d0\": rpc error: code = NotFound desc = could not find container \"660d4a4bf57384c2ceac1ad674a510b7ab4c2b04ecb5642386ddd9841ec1f6d0\": container with ID starting with 660d4a4bf57384c2ceac1ad674a510b7ab4c2b04ecb5642386ddd9841ec1f6d0 not found: ID does not exist"
Sep 10 14:41:05.200416 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.200374    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb48298-206b-4bf1-90d4-4baa11f6c854-dshm\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:41:05.200416 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.200394    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb48298-206b-4bf1-90d4-4baa11f6c854-tmp-dir\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:41:05.200416 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.200403    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb48298-206b-4bf1-90d4-4baa11f6c854-model-cache\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:41:05.200416 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.200411    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb48298-206b-4bf1-90d4-4baa11f6c854-tls-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:41:05.200560 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.200418    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sj5d\" (UniqueName: \"kubernetes.io/projected/0cb48298-206b-4bf1-90d4-4baa11f6c854-kube-api-access-2sj5d\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:41:05.211066 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.211026    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9" podStartSLOduration=2.211014122 podStartE2EDuration="2.211014122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:41:05.209809288 +0000 UTC m=+1277.251683769" watchObservedRunningTime="2026-09-10 14:41:05.211014122 +0000 UTC m=+1277.252888602"
Sep 10 14:41:05.227406 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.227384    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"]
Sep 10 14:41:05.233147 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.233128    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-61b16dfc-kserve-bc65c758b-846bx"]
Sep 10 14:41:05.944875 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.944835    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:05.949779 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:05.949752    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:06.195879 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:06.195795    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:06.196706 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:06.196673    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:06.428628 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:06.428591    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cb48298-206b-4bf1-90d4-4baa11f6c854" path="/var/lib/kubelet/pods/0cb48298-206b-4bf1-90d4-4baa11f6c854/volumes"
Sep 10 14:41:08.172032 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.172008    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:41:08.209714 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.209618    2547 generic.go:356] "Generic (PLEG): container finished" podID="87cdc6a3-8bc6-4be6-b793-1d1ac852427a" containerID="4b38606555e2bf66f1cdb4070a89f1c7d7ffa23a554dee02eedae4a94a7f016e" exitCode=0
Sep 10 14:41:08.209714 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.209666    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"
Sep 10 14:41:08.209937 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.209709    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps" event={"ID":"87cdc6a3-8bc6-4be6-b793-1d1ac852427a","Type":"ContainerDied","Data":"4b38606555e2bf66f1cdb4070a89f1c7d7ffa23a554dee02eedae4a94a7f016e"}
Sep 10 14:41:08.209937 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.209774    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps" event={"ID":"87cdc6a3-8bc6-4be6-b793-1d1ac852427a","Type":"ContainerDied","Data":"2a6e9ce03edf8303b81682a1b30a08d22463eca18139d503d84a89223ced706f"}
Sep 10 14:41:08.209937 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.209805    2547 scope.go:117] "RemoveContainer" containerID="4b38606555e2bf66f1cdb4070a89f1c7d7ffa23a554dee02eedae4a94a7f016e"
Sep 10 14:41:08.218931 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.218909    2547 scope.go:117] "RemoveContainer" containerID="3b22ae9b03f8f051159c23bf0b6ba9e5fc89591a1ae37af156fc5ab28e51ea9d"
Sep 10 14:41:08.226978 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.226957    2547 scope.go:117] "RemoveContainer" containerID="4b38606555e2bf66f1cdb4070a89f1c7d7ffa23a554dee02eedae4a94a7f016e"
Sep 10 14:41:08.227253 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:41:08.227225    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b38606555e2bf66f1cdb4070a89f1c7d7ffa23a554dee02eedae4a94a7f016e\": container with ID starting with 4b38606555e2bf66f1cdb4070a89f1c7d7ffa23a554dee02eedae4a94a7f016e not found: ID does not exist" containerID="4b38606555e2bf66f1cdb4070a89f1c7d7ffa23a554dee02eedae4a94a7f016e"
Sep 10 14:41:08.227325 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.227265    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b38606555e2bf66f1cdb4070a89f1c7d7ffa23a554dee02eedae4a94a7f016e"} err="failed to get container status \"4b38606555e2bf66f1cdb4070a89f1c7d7ffa23a554dee02eedae4a94a7f016e\": rpc error: code = NotFound desc = could not find container \"4b38606555e2bf66f1cdb4070a89f1c7d7ffa23a554dee02eedae4a94a7f016e\": container with ID starting with 4b38606555e2bf66f1cdb4070a89f1c7d7ffa23a554dee02eedae4a94a7f016e not found: ID does not exist"
Sep 10 14:41:08.227325 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.227290    2547 scope.go:117] "RemoveContainer" containerID="3b22ae9b03f8f051159c23bf0b6ba9e5fc89591a1ae37af156fc5ab28e51ea9d"
Sep 10 14:41:08.227547 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:41:08.227530    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b22ae9b03f8f051159c23bf0b6ba9e5fc89591a1ae37af156fc5ab28e51ea9d\": container with ID starting with 3b22ae9b03f8f051159c23bf0b6ba9e5fc89591a1ae37af156fc5ab28e51ea9d not found: ID does not exist" containerID="3b22ae9b03f8f051159c23bf0b6ba9e5fc89591a1ae37af156fc5ab28e51ea9d"
Sep 10 14:41:08.227586 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.227555    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b22ae9b03f8f051159c23bf0b6ba9e5fc89591a1ae37af156fc5ab28e51ea9d"} err="failed to get container status \"3b22ae9b03f8f051159c23bf0b6ba9e5fc89591a1ae37af156fc5ab28e51ea9d\": rpc error: code = NotFound desc = could not find container \"3b22ae9b03f8f051159c23bf0b6ba9e5fc89591a1ae37af156fc5ab28e51ea9d\": container with ID starting with 3b22ae9b03f8f051159c23bf0b6ba9e5fc89591a1ae37af156fc5ab28e51ea9d not found: ID does not exist"
Sep 10 14:41:08.332277 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.332243    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-credential-socket\") pod \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") "
Sep 10 14:41:08.332496 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.332306    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-workload-socket\") pod \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") "
Sep 10 14:41:08.332496 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.332358    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istio-token\") pod \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") "
Sep 10 14:41:08.332496 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.332390    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istio-data\") pod \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") "
Sep 10 14:41:08.332496 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.332438    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istio-podinfo\") pod \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") "
Sep 10 14:41:08.332496 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.332479    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istio-envoy\") pod \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") "
Sep 10 14:41:08.332843 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.332522    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s45bs\" (UniqueName: \"kubernetes.io/projected/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-kube-api-access-s45bs\") pod \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") "
Sep 10 14:41:08.332843 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.332552    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-workload-certs\") pod \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") "
Sep 10 14:41:08.332843 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.332607    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "87cdc6a3-8bc6-4be6-b793-1d1ac852427a" (UID: "87cdc6a3-8bc6-4be6-b793-1d1ac852427a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:41:08.332843 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.332620    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istiod-ca-cert\") pod \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\" (UID: \"87cdc6a3-8bc6-4be6-b793-1d1ac852427a\") "
Sep 10 14:41:08.333044 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.332920    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "87cdc6a3-8bc6-4be6-b793-1d1ac852427a" (UID: "87cdc6a3-8bc6-4be6-b793-1d1ac852427a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:41:08.333044 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.332956    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istio-data" (OuterVolumeSpecName: "istio-data") pod "87cdc6a3-8bc6-4be6-b793-1d1ac852427a" (UID: "87cdc6a3-8bc6-4be6-b793-1d1ac852427a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:41:08.333469 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.333158    2547 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istiod-ca-cert\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:41:08.333469 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.333172    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "87cdc6a3-8bc6-4be6-b793-1d1ac852427a" (UID: "87cdc6a3-8bc6-4be6-b793-1d1ac852427a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:41:08.333469 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.333182    2547 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-workload-socket\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:41:08.333469 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.333223    2547 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istio-data\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:41:08.333469 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.333351    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "87cdc6a3-8bc6-4be6-b793-1d1ac852427a" (UID: "87cdc6a3-8bc6-4be6-b793-1d1ac852427a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:41:08.335493 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.335457    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istio-token" (OuterVolumeSpecName: "istio-token") pod "87cdc6a3-8bc6-4be6-b793-1d1ac852427a" (UID: "87cdc6a3-8bc6-4be6-b793-1d1ac852427a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:41:08.335736 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.335715    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-kube-api-access-s45bs" (OuterVolumeSpecName: "kube-api-access-s45bs") pod "87cdc6a3-8bc6-4be6-b793-1d1ac852427a" (UID: "87cdc6a3-8bc6-4be6-b793-1d1ac852427a"). InnerVolumeSpecName "kube-api-access-s45bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:41:08.336033 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.336016    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "87cdc6a3-8bc6-4be6-b793-1d1ac852427a" (UID: "87cdc6a3-8bc6-4be6-b793-1d1ac852427a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:41:08.336339 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.336311    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "87cdc6a3-8bc6-4be6-b793-1d1ac852427a" (UID: "87cdc6a3-8bc6-4be6-b793-1d1ac852427a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 14:41:08.434368 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.434341    2547 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istio-token\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:41:08.434368 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.434369    2547 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istio-podinfo\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:41:08.434527 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.434382    2547 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-istio-envoy\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:41:08.434527 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.434394    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s45bs\" (UniqueName: \"kubernetes.io/projected/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-kube-api-access-s45bs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:41:08.434527 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.434406    2547 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-workload-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:41:08.434527 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.434419    2547 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cdc6a3-8bc6-4be6-b793-1d1ac852427a-credential-socket\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:41:08.525718 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.525621    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"]
Sep 10 14:41:08.530046 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:08.530019    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-e7415a-openshift-default-7dcddfd4f6-vvsps"]
Sep 10 14:41:09.249491 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:09.249457    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:41:10.421858 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:10.421826    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87cdc6a3-8bc6-4be6-b793-1d1ac852427a" path="/var/lib/kubelet/pods/87cdc6a3-8bc6-4be6-b793-1d1ac852427a/volumes"
Sep 10 14:41:11.210906 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.210865    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"]
Sep 10 14:41:11.211250 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.211238    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cb48298-206b-4bf1-90d4-4baa11f6c854" containerName="main"
Sep 10 14:41:11.211291 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.211252    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cb48298-206b-4bf1-90d4-4baa11f6c854" containerName="main"
Sep 10 14:41:11.211291 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.211273    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87cdc6a3-8bc6-4be6-b793-1d1ac852427a" containerName="istio-proxy"
Sep 10 14:41:11.211291 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.211279    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="87cdc6a3-8bc6-4be6-b793-1d1ac852427a" containerName="istio-proxy"
Sep 10 14:41:11.211291 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.211286    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87cdc6a3-8bc6-4be6-b793-1d1ac852427a" containerName="istio-proxy"
Sep 10 14:41:11.211291 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.211292    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="87cdc6a3-8bc6-4be6-b793-1d1ac852427a" containerName="istio-proxy"
Sep 10 14:41:11.211454 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.211345    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="87cdc6a3-8bc6-4be6-b793-1d1ac852427a" containerName="istio-proxy"
Sep 10 14:41:11.211454 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.211351    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="87cdc6a3-8bc6-4be6-b793-1d1ac852427a" containerName="istio-proxy"
Sep 10 14:41:11.211454 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.211361    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cb48298-206b-4bf1-90d4-4baa11f6c854" containerName="main"
Sep 10 14:41:11.216041 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.216024    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:41:11.220151 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.220119    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2365fe\"/\"openshift-service-ca.crt\""
Sep 10 14:41:11.220277 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.220189    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2365fe\"/\"test-model-2365fe-kserve-self-signed-certs\""
Sep 10 14:41:11.220277 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.220132    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2365fe\"/\"default-dockercfg-t82g2\""
Sep 10 14:41:11.220277 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.220245    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2365fe\"/\"kube-root-ca.crt\""
Sep 10 14:41:11.226931 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.226901    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"]
Sep 10 14:41:11.257016 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.256988    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bf48933-cbd3-44b1-beae-422b145bb9c9-tls-certs\") pod \"test-model-2365fe-kserve-5b46dddcdc-kj4k4\" (UID: \"0bf48933-cbd3-44b1-beae-422b145bb9c9\") " pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:41:11.257145 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.257023    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9pdl\" (UniqueName: \"kubernetes.io/projected/0bf48933-cbd3-44b1-beae-422b145bb9c9-kube-api-access-r9pdl\") pod \"test-model-2365fe-kserve-5b46dddcdc-kj4k4\" (UID: \"0bf48933-cbd3-44b1-beae-422b145bb9c9\") " pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:41:11.257145 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.257046    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bf48933-cbd3-44b1-beae-422b145bb9c9-home\") pod \"test-model-2365fe-kserve-5b46dddcdc-kj4k4\" (UID: \"0bf48933-cbd3-44b1-beae-422b145bb9c9\") " pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:41:11.257222 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.257140    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bf48933-cbd3-44b1-beae-422b145bb9c9-tmp-dir\") pod \"test-model-2365fe-kserve-5b46dddcdc-kj4k4\" (UID: \"0bf48933-cbd3-44b1-beae-422b145bb9c9\") " pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:41:11.257222 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.257188    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bf48933-cbd3-44b1-beae-422b145bb9c9-dshm\") pod \"test-model-2365fe-kserve-5b46dddcdc-kj4k4\" (UID: \"0bf48933-cbd3-44b1-beae-422b145bb9c9\") " pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:41:11.257222 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.257213    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bf48933-cbd3-44b1-beae-422b145bb9c9-model-cache\") pod \"test-model-2365fe-kserve-5b46dddcdc-kj4k4\" (UID: \"0bf48933-cbd3-44b1-beae-422b145bb9c9\") " pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:41:11.358059 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.358023    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bf48933-cbd3-44b1-beae-422b145bb9c9-tls-certs\") pod \"test-model-2365fe-kserve-5b46dddcdc-kj4k4\" (UID: \"0bf48933-cbd3-44b1-beae-422b145bb9c9\") " pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:41:11.358188 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.358064    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9pdl\" (UniqueName: \"kubernetes.io/projected/0bf48933-cbd3-44b1-beae-422b145bb9c9-kube-api-access-r9pdl\") pod \"test-model-2365fe-kserve-5b46dddcdc-kj4k4\" (UID: \"0bf48933-cbd3-44b1-beae-422b145bb9c9\") " pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:41:11.358246 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.358184    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bf48933-cbd3-44b1-beae-422b145bb9c9-home\") pod \"test-model-2365fe-kserve-5b46dddcdc-kj4k4\" (UID: \"0bf48933-cbd3-44b1-beae-422b145bb9c9\") " pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:41:11.358280 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.358250    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bf48933-cbd3-44b1-beae-422b145bb9c9-tmp-dir\") pod \"test-model-2365fe-kserve-5b46dddcdc-kj4k4\" (UID: \"0bf48933-cbd3-44b1-beae-422b145bb9c9\") " pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:41:11.358327 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.358312    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bf48933-cbd3-44b1-beae-422b145bb9c9-dshm\") pod \"test-model-2365fe-kserve-5b46dddcdc-kj4k4\" (UID: \"0bf48933-cbd3-44b1-beae-422b145bb9c9\") " pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:41:11.358360 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.358339    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bf48933-cbd3-44b1-beae-422b145bb9c9-model-cache\") pod \"test-model-2365fe-kserve-5b46dddcdc-kj4k4\" (UID: \"0bf48933-cbd3-44b1-beae-422b145bb9c9\") " pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:41:11.358621 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.358594    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bf48933-cbd3-44b1-beae-422b145bb9c9-home\") pod \"test-model-2365fe-kserve-5b46dddcdc-kj4k4\" (UID: \"0bf48933-cbd3-44b1-beae-422b145bb9c9\") " pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:41:11.358753 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.358682    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bf48933-cbd3-44b1-beae-422b145bb9c9-tmp-dir\") pod \"test-model-2365fe-kserve-5b46dddcdc-kj4k4\" (UID: \"0bf48933-cbd3-44b1-beae-422b145bb9c9\") " pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:41:11.358753 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.358738    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bf48933-cbd3-44b1-beae-422b145bb9c9-model-cache\") pod \"test-model-2365fe-kserve-5b46dddcdc-kj4k4\" (UID: \"0bf48933-cbd3-44b1-beae-422b145bb9c9\") " pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:41:11.360414 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.360389    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bf48933-cbd3-44b1-beae-422b145bb9c9-dshm\") pod \"test-model-2365fe-kserve-5b46dddcdc-kj4k4\" (UID: \"0bf48933-cbd3-44b1-beae-422b145bb9c9\") " pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:41:11.360597 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.360579    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bf48933-cbd3-44b1-beae-422b145bb9c9-tls-certs\") pod \"test-model-2365fe-kserve-5b46dddcdc-kj4k4\" (UID: \"0bf48933-cbd3-44b1-beae-422b145bb9c9\") " pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:41:11.365479 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.365459    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9pdl\" (UniqueName: \"kubernetes.io/projected/0bf48933-cbd3-44b1-beae-422b145bb9c9-kube-api-access-r9pdl\") pod \"test-model-2365fe-kserve-5b46dddcdc-kj4k4\" (UID: \"0bf48933-cbd3-44b1-beae-422b145bb9c9\") " pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:41:11.532525 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.532434    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:41:11.805023 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:11.804975    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"]
Sep 10 14:41:11.807561 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:41:11.807517    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bf48933_cbd3_44b1_beae_422b145bb9c9.slice/crio-18c1eee5208e38d4738a2abd8da2a9b3d6b6ad6350f276e0f9b405404ef1f931 WatchSource:0}: Error finding container 18c1eee5208e38d4738a2abd8da2a9b3d6b6ad6350f276e0f9b405404ef1f931: Status 404 returned error can't find the container with id 18c1eee5208e38d4738a2abd8da2a9b3d6b6ad6350f276e0f9b405404ef1f931
Sep 10 14:41:12.239950 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:12.239913    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4" event={"ID":"0bf48933-cbd3-44b1-beae-422b145bb9c9","Type":"ContainerStarted","Data":"963f11aa432c74046a2b86fb5cf10c6fcdf70b033110a6065fa938166f44b411"}
Sep 10 14:41:12.240114 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:12.239962    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4" event={"ID":"0bf48933-cbd3-44b1-beae-422b145bb9c9","Type":"ContainerStarted","Data":"18c1eee5208e38d4738a2abd8da2a9b3d6b6ad6350f276e0f9b405404ef1f931"}
Sep 10 14:41:12.258333 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:12.258288    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4" podStartSLOduration=1.258273947 podStartE2EDuration="1.258273947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:41:12.255087022 +0000 UTC m=+1284.296961502" watchObservedRunningTime="2026-09-10 14:41:12.258273947 +0000 UTC m=+1284.300148427"
Sep 10 14:41:20.336484 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:20.336446    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b46e5fd4-ss9pr"]
Sep 10 14:41:20.339816 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:20.339798    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b46e5fd4-ss9pr"
Sep 10 14:41:20.342419 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:20.342393    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xnxsg\""
Sep 10 14:41:20.342516 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:20.342431    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 14:41:20.348252 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:20.348229    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b46e5fd4-ss9pr"]
Sep 10 14:41:20.432624 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:20.432588    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db380ef6-2f11-425b-b58a-c6409ab662c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b46e5fd4-ss9pr\" (UID: \"db380ef6-2f11-425b-b58a-c6409ab662c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b46e5fd4-ss9pr"
Sep 10 14:41:20.533529 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:20.533493    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db380ef6-2f11-425b-b58a-c6409ab662c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b46e5fd4-ss9pr\" (UID: \"db380ef6-2f11-425b-b58a-c6409ab662c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b46e5fd4-ss9pr"
Sep 10 14:41:20.534174 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:20.534151    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db380ef6-2f11-425b-b58a-c6409ab662c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b46e5fd4-ss9pr\" (UID: \"db380ef6-2f11-425b-b58a-c6409ab662c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b46e5fd4-ss9pr"
Sep 10 14:41:20.651470 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:20.651368    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b46e5fd4-ss9pr"
Sep 10 14:41:20.931950 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:20.931925    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b46e5fd4-ss9pr"]
Sep 10 14:41:20.933557 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:41:20.933529    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb380ef6_2f11_425b_b58a_c6409ab662c2.slice/crio-bf75dc2779e1f503505618b01ca16129e5114981b6b9a93bae516c07dc6a83e6 WatchSource:0}: Error finding container bf75dc2779e1f503505618b01ca16129e5114981b6b9a93bae516c07dc6a83e6: Status 404 returned error can't find the container with id bf75dc2779e1f503505618b01ca16129e5114981b6b9a93bae516c07dc6a83e6
Sep 10 14:41:21.292613 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:21.292521    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b46e5fd4-ss9pr" event={"ID":"db380ef6-2f11-425b-b58a-c6409ab662c2","Type":"ContainerStarted","Data":"bf75dc2779e1f503505618b01ca16129e5114981b6b9a93bae516c07dc6a83e6"}
Sep 10 14:41:21.532606 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:21.532560    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:41:21.533134 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:21.532644    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:41:21.545151 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:21.545073    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:41:22.303009 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:22.302913    2547 generic.go:356] "Generic (PLEG): container finished" podID="db380ef6-2f11-425b-b58a-c6409ab662c2" containerID="7913c7568eed7dbd8e1098b40534135c3e07d310ba254930999c2a80e917ddfb" exitCode=0
Sep 10 14:41:22.303009 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:22.302976    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b46e5fd4-ss9pr" event={"ID":"db380ef6-2f11-425b-b58a-c6409ab662c2","Type":"ContainerDied","Data":"7913c7568eed7dbd8e1098b40534135c3e07d310ba254930999c2a80e917ddfb"}
Sep 10 14:41:22.314286 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:22.314258    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:41:23.478135 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:23.478104    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b46e5fd4-ss9pr"
Sep 10 14:41:23.574148 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:23.574062    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db380ef6-2f11-425b-b58a-c6409ab662c2-maas-api-service-ca\") pod \"db380ef6-2f11-425b-b58a-c6409ab662c2\" (UID: \"db380ef6-2f11-425b-b58a-c6409ab662c2\") "
Sep 10 14:41:23.574433 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:23.574410    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db380ef6-2f11-425b-b58a-c6409ab662c2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "db380ef6-2f11-425b-b58a-c6409ab662c2" (UID: "db380ef6-2f11-425b-b58a-c6409ab662c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:41:23.675357 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:23.675317    2547 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db380ef6-2f11-425b-b58a-c6409ab662c2-maas-api-service-ca\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:41:24.316182 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:24.316150    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b46e5fd4-ss9pr"
Sep 10 14:41:24.316364 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:24.316156    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b46e5fd4-ss9pr" event={"ID":"db380ef6-2f11-425b-b58a-c6409ab662c2","Type":"ContainerDied","Data":"bf75dc2779e1f503505618b01ca16129e5114981b6b9a93bae516c07dc6a83e6"}
Sep 10 14:41:24.316364 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:24.316279    2547 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf75dc2779e1f503505618b01ca16129e5114981b6b9a93bae516c07dc6a83e6"
Sep 10 14:41:27.787621 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:27.787575    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b46e5fd4-ss9pr"]
Sep 10 14:41:27.789704 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:27.789676    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b46e5fd4-ss9pr"]
Sep 10 14:41:28.240817 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:28.240774    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:41:28.420841 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:28.420812    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db380ef6-2f11-425b-b58a-c6409ab662c2" path="/var/lib/kubelet/pods/db380ef6-2f11-425b-b58a-c6409ab662c2/volumes"
Sep 10 14:41:29.349996 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:29.349971    2547 generic.go:356] "Generic (PLEG): container finished" podID="0f1b2b1f-ff80-4c73-ac93-edab890862c0" containerID="e5e550b71347cf64774f0cf58565653f9df3e227c7ca23fa04b24c5ee611999e" exitCode=0
Sep 10 14:41:29.350327 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:29.350046    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9" event={"ID":"0f1b2b1f-ff80-4c73-ac93-edab890862c0","Type":"ContainerDied","Data":"e5e550b71347cf64774f0cf58565653f9df3e227c7ca23fa04b24c5ee611999e"}
Sep 10 14:41:29.350425 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:29.350411    2547 scope.go:117] "RemoveContainer" containerID="e5e550b71347cf64774f0cf58565653f9df3e227c7ca23fa04b24c5ee611999e"
Sep 10 14:41:29.351456 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:29.351427    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:41:29.351510 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:29.351497    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:41:29.351549 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:29.351535    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:41:30.358301 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:30.358269    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9" event={"ID":"0f1b2b1f-ff80-4c73-ac93-edab890862c0","Type":"ContainerStarted","Data":"5a82c4b7a3d23de3b6481a636b17969f692d8672b9dd2f266a6511d9e6f3270f"}
Sep 10 14:41:30.944773 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:30.944729    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:30.944991 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:30.944968    2547 patch_prober.go:28] interesting pod/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 10 14:41:30.945044 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:30.945017    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9" podUID="0f1b2b1f-ff80-4c73-ac93-edab890862c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 10 14:41:31.945463 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:31.945429    2547 patch_prober.go:28] interesting pod/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 10 14:41:31.945881 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:31.945489    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9" podUID="0f1b2b1f-ff80-4c73-ac93-edab890862c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 10 14:41:32.945530 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:32.945487    2547 patch_prober.go:28] interesting pod/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 10 14:41:32.945953 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:32.945548    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9" podUID="0f1b2b1f-ff80-4c73-ac93-edab890862c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 10 14:41:33.944624 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:33.944580    2547 patch_prober.go:28] interesting pod/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 10 14:41:33.944826 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:33.944645    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9" podUID="0f1b2b1f-ff80-4c73-ac93-edab890862c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 10 14:41:34.944811 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:34.944771    2547 patch_prober.go:28] interesting pod/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 10 14:41:34.945196 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:34.944847    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9" podUID="0f1b2b1f-ff80-4c73-ac93-edab890862c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 10 14:41:35.945501 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:35.945462    2547 patch_prober.go:28] interesting pod/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 10 14:41:35.945912 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:35.945533    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9" podUID="0f1b2b1f-ff80-4c73-ac93-edab890862c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 10 14:41:36.196442 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:36.196351    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:36.949517 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:36.949484    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:36.956882 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:36.956846    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:41:43.524764 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:43.524724    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:41:57.718316 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.718278    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"]
Sep 10 14:41:57.718734 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.718670    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db380ef6-2f11-425b-b58a-c6409ab662c2" containerName="revoke-keys"
Sep 10 14:41:57.718734 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.718681    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="db380ef6-2f11-425b-b58a-c6409ab662c2" containerName="revoke-keys"
Sep 10 14:41:57.718824 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.718777    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="db380ef6-2f11-425b-b58a-c6409ab662c2" containerName="revoke-keys"
Sep 10 14:41:57.723287 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.723268    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"
Sep 10 14:41:57.725920 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.725895    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b2dfe7\"/\"kube-root-ca.crt\""
Sep 10 14:41:57.726058 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.725949    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b2dfe7\"/\"openshift-service-ca.crt\""
Sep 10 14:41:57.727372 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.727301    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b2dfe7\"/\"default-dockercfg-cj85v\""
Sep 10 14:41:57.727372 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.727338    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b2dfe7\"/\"explicit-model-b2dfe7-kserve-self-signed-certs\""
Sep 10 14:41:57.730898 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.730869    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"]
Sep 10 14:41:57.895927 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.895895    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d048c259-8537-4740-b548-5cc40285da44-tmp-dir\") pod \"explicit-model-b2dfe7-kserve-787cf94bfc-qv4df\" (UID: \"d048c259-8537-4740-b548-5cc40285da44\") " pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"
Sep 10 14:41:57.896080 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.895948    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d048c259-8537-4740-b548-5cc40285da44-home\") pod \"explicit-model-b2dfe7-kserve-787cf94bfc-qv4df\" (UID: \"d048c259-8537-4740-b548-5cc40285da44\") " pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"
Sep 10 14:41:57.896080 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.895973    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d048c259-8537-4740-b548-5cc40285da44-tls-certs\") pod \"explicit-model-b2dfe7-kserve-787cf94bfc-qv4df\" (UID: \"d048c259-8537-4740-b548-5cc40285da44\") " pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"
Sep 10 14:41:57.896080 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.895989    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d048c259-8537-4740-b548-5cc40285da44-dshm\") pod \"explicit-model-b2dfe7-kserve-787cf94bfc-qv4df\" (UID: \"d048c259-8537-4740-b548-5cc40285da44\") " pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"
Sep 10 14:41:57.896080 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.896006    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d048c259-8537-4740-b548-5cc40285da44-model-cache\") pod \"explicit-model-b2dfe7-kserve-787cf94bfc-qv4df\" (UID: \"d048c259-8537-4740-b548-5cc40285da44\") " pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"
Sep 10 14:41:57.896080 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.896035    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fhtj\" (UniqueName: \"kubernetes.io/projected/d048c259-8537-4740-b548-5cc40285da44-kube-api-access-4fhtj\") pod \"explicit-model-b2dfe7-kserve-787cf94bfc-qv4df\" (UID: \"d048c259-8537-4740-b548-5cc40285da44\") " pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"
Sep 10 14:41:57.997084 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.996995    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d048c259-8537-4740-b548-5cc40285da44-tmp-dir\") pod \"explicit-model-b2dfe7-kserve-787cf94bfc-qv4df\" (UID: \"d048c259-8537-4740-b548-5cc40285da44\") " pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"
Sep 10 14:41:57.997084 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.997052    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d048c259-8537-4740-b548-5cc40285da44-home\") pod \"explicit-model-b2dfe7-kserve-787cf94bfc-qv4df\" (UID: \"d048c259-8537-4740-b548-5cc40285da44\") " pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"
Sep 10 14:41:57.997300 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.997163    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d048c259-8537-4740-b548-5cc40285da44-tls-certs\") pod \"explicit-model-b2dfe7-kserve-787cf94bfc-qv4df\" (UID: \"d048c259-8537-4740-b548-5cc40285da44\") " pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"
Sep 10 14:41:57.997300 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.997190    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d048c259-8537-4740-b548-5cc40285da44-dshm\") pod \"explicit-model-b2dfe7-kserve-787cf94bfc-qv4df\" (UID: \"d048c259-8537-4740-b548-5cc40285da44\") " pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"
Sep 10 14:41:57.997300 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.997217    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d048c259-8537-4740-b548-5cc40285da44-model-cache\") pod \"explicit-model-b2dfe7-kserve-787cf94bfc-qv4df\" (UID: \"d048c259-8537-4740-b548-5cc40285da44\") " pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"
Sep 10 14:41:57.997300 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.997249    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fhtj\" (UniqueName: \"kubernetes.io/projected/d048c259-8537-4740-b548-5cc40285da44-kube-api-access-4fhtj\") pod \"explicit-model-b2dfe7-kserve-787cf94bfc-qv4df\" (UID: \"d048c259-8537-4740-b548-5cc40285da44\") " pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"
Sep 10 14:41:57.997515 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.997441    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d048c259-8537-4740-b548-5cc40285da44-tmp-dir\") pod \"explicit-model-b2dfe7-kserve-787cf94bfc-qv4df\" (UID: \"d048c259-8537-4740-b548-5cc40285da44\") " pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"
Sep 10 14:41:57.997515 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.997478    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d048c259-8537-4740-b548-5cc40285da44-home\") pod \"explicit-model-b2dfe7-kserve-787cf94bfc-qv4df\" (UID: \"d048c259-8537-4740-b548-5cc40285da44\") " pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"
Sep 10 14:41:57.998021 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.997598    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d048c259-8537-4740-b548-5cc40285da44-model-cache\") pod \"explicit-model-b2dfe7-kserve-787cf94bfc-qv4df\" (UID: \"d048c259-8537-4740-b548-5cc40285da44\") " pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"
Sep 10 14:41:57.999409 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.999392    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d048c259-8537-4740-b548-5cc40285da44-dshm\") pod \"explicit-model-b2dfe7-kserve-787cf94bfc-qv4df\" (UID: \"d048c259-8537-4740-b548-5cc40285da44\") " pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"
Sep 10 14:41:57.999671 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:57.999650    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d048c259-8537-4740-b548-5cc40285da44-tls-certs\") pod \"explicit-model-b2dfe7-kserve-787cf94bfc-qv4df\" (UID: \"d048c259-8537-4740-b548-5cc40285da44\") " pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"
Sep 10 14:41:58.006295 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:58.006264    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fhtj\" (UniqueName: \"kubernetes.io/projected/d048c259-8537-4740-b548-5cc40285da44-kube-api-access-4fhtj\") pod \"explicit-model-b2dfe7-kserve-787cf94bfc-qv4df\" (UID: \"d048c259-8537-4740-b548-5cc40285da44\") " pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"
Sep 10 14:41:58.034501 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:58.034462    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"
Sep 10 14:41:58.305915 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:58.305888    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"]
Sep 10 14:41:58.307496 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:41:58.307457    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd048c259_8537_4740_b548_5cc40285da44.slice/crio-7ae87ed6f82bc8a88496a4375141b4cf8602814a5e08ffde40e3dd6c1806d504 WatchSource:0}: Error finding container 7ae87ed6f82bc8a88496a4375141b4cf8602814a5e08ffde40e3dd6c1806d504: Status 404 returned error can't find the container with id 7ae87ed6f82bc8a88496a4375141b4cf8602814a5e08ffde40e3dd6c1806d504
Sep 10 14:41:58.421941 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:58.421913    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:41:58.524779 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:58.524742    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df" event={"ID":"d048c259-8537-4740-b548-5cc40285da44","Type":"ContainerStarted","Data":"5d94d73e8e74a765b681ef5d452b05375ab78a41f6e6d235c315c74da9908491"}
Sep 10 14:41:58.524779 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:58.524787    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df" event={"ID":"d048c259-8537-4740-b548-5cc40285da44","Type":"ContainerStarted","Data":"7ae87ed6f82bc8a88496a4375141b4cf8602814a5e08ffde40e3dd6c1806d504"}
Sep 10 14:41:58.547462 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:41:58.547412    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df" podStartSLOduration=1.54739688 podStartE2EDuration="1.54739688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:41:58.544982478 +0000 UTC m=+1330.586856959" watchObservedRunningTime="2026-09-10 14:41:58.54739688 +0000 UTC m=+1330.589271353"
Sep 10 14:42:00.800037 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:00.800003    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"]
Sep 10 14:42:08.035131 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:08.035092    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"
Sep 10 14:42:11.541741 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:11.541644    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:42:15.821004 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:15.820936    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df" podUID="d048c259-8537-4740-b548-5cc40285da44" containerName="main" containerID="cri-o://5d94d73e8e74a765b681ef5d452b05375ab78a41f6e6d235c315c74da9908491" gracePeriod=45
Sep 10 14:42:16.066867 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.066839    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"
Sep 10 14:42:16.169337 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.169303    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fhtj\" (UniqueName: \"kubernetes.io/projected/d048c259-8537-4740-b548-5cc40285da44-kube-api-access-4fhtj\") pod \"d048c259-8537-4740-b548-5cc40285da44\" (UID: \"d048c259-8537-4740-b548-5cc40285da44\") "
Sep 10 14:42:16.169522 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.169363    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d048c259-8537-4740-b548-5cc40285da44-tmp-dir\") pod \"d048c259-8537-4740-b548-5cc40285da44\" (UID: \"d048c259-8537-4740-b548-5cc40285da44\") "
Sep 10 14:42:16.169522 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.169391    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d048c259-8537-4740-b548-5cc40285da44-model-cache\") pod \"d048c259-8537-4740-b548-5cc40285da44\" (UID: \"d048c259-8537-4740-b548-5cc40285da44\") "
Sep 10 14:42:16.169522 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.169407    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d048c259-8537-4740-b548-5cc40285da44-home\") pod \"d048c259-8537-4740-b548-5cc40285da44\" (UID: \"d048c259-8537-4740-b548-5cc40285da44\") "
Sep 10 14:42:16.169522 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.169433    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d048c259-8537-4740-b548-5cc40285da44-dshm\") pod \"d048c259-8537-4740-b548-5cc40285da44\" (UID: \"d048c259-8537-4740-b548-5cc40285da44\") "
Sep 10 14:42:16.169522 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.169492    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d048c259-8537-4740-b548-5cc40285da44-tls-certs\") pod \"d048c259-8537-4740-b548-5cc40285da44\" (UID: \"d048c259-8537-4740-b548-5cc40285da44\") "
Sep 10 14:42:16.169823 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.169665    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d048c259-8537-4740-b548-5cc40285da44-model-cache" (OuterVolumeSpecName: "model-cache") pod "d048c259-8537-4740-b548-5cc40285da44" (UID: "d048c259-8537-4740-b548-5cc40285da44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:16.169823 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.169680    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d048c259-8537-4740-b548-5cc40285da44-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d048c259-8537-4740-b548-5cc40285da44" (UID: "d048c259-8537-4740-b548-5cc40285da44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:16.169823 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.169710    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d048c259-8537-4740-b548-5cc40285da44-home" (OuterVolumeSpecName: "home") pod "d048c259-8537-4740-b548-5cc40285da44" (UID: "d048c259-8537-4740-b548-5cc40285da44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:16.169930 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.169832    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d048c259-8537-4740-b548-5cc40285da44-tmp-dir\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:16.169930 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.169854    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d048c259-8537-4740-b548-5cc40285da44-model-cache\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:16.169930 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.169866    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d048c259-8537-4740-b548-5cc40285da44-home\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:16.171622 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.171591    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d048c259-8537-4740-b548-5cc40285da44-dshm" (OuterVolumeSpecName: "dshm") pod "d048c259-8537-4740-b548-5cc40285da44" (UID: "d048c259-8537-4740-b548-5cc40285da44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:16.171767 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.171650    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d048c259-8537-4740-b548-5cc40285da44-kube-api-access-4fhtj" (OuterVolumeSpecName: "kube-api-access-4fhtj") pod "d048c259-8537-4740-b548-5cc40285da44" (UID: "d048c259-8537-4740-b548-5cc40285da44"). InnerVolumeSpecName "kube-api-access-4fhtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:42:16.171767 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.171658    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d048c259-8537-4740-b548-5cc40285da44-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d048c259-8537-4740-b548-5cc40285da44" (UID: "d048c259-8537-4740-b548-5cc40285da44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:42:16.270667 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.270629    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d048c259-8537-4740-b548-5cc40285da44-dshm\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:16.270667 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.270660    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d048c259-8537-4740-b548-5cc40285da44-tls-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:16.270667 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.270670    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fhtj\" (UniqueName: \"kubernetes.io/projected/d048c259-8537-4740-b548-5cc40285da44-kube-api-access-4fhtj\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:16.631224 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.631191    2547 generic.go:356] "Generic (PLEG): container finished" podID="d048c259-8537-4740-b548-5cc40285da44" containerID="5d94d73e8e74a765b681ef5d452b05375ab78a41f6e6d235c315c74da9908491" exitCode=0
Sep 10 14:42:16.631416 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.631230    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"
Sep 10 14:42:16.631416 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.631278    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df" event={"ID":"d048c259-8537-4740-b548-5cc40285da44","Type":"ContainerDied","Data":"5d94d73e8e74a765b681ef5d452b05375ab78a41f6e6d235c315c74da9908491"}
Sep 10 14:42:16.631416 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.631323    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df" event={"ID":"d048c259-8537-4740-b548-5cc40285da44","Type":"ContainerDied","Data":"7ae87ed6f82bc8a88496a4375141b4cf8602814a5e08ffde40e3dd6c1806d504"}
Sep 10 14:42:16.631416 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.631340    2547 scope.go:117] "RemoveContainer" containerID="5d94d73e8e74a765b681ef5d452b05375ab78a41f6e6d235c315c74da9908491"
Sep 10 14:42:16.640715 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.640672    2547 scope.go:117] "RemoveContainer" containerID="5d94d73e8e74a765b681ef5d452b05375ab78a41f6e6d235c315c74da9908491"
Sep 10 14:42:16.640993 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:42:16.640972    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d94d73e8e74a765b681ef5d452b05375ab78a41f6e6d235c315c74da9908491\": container with ID starting with 5d94d73e8e74a765b681ef5d452b05375ab78a41f6e6d235c315c74da9908491 not found: ID does not exist" containerID="5d94d73e8e74a765b681ef5d452b05375ab78a41f6e6d235c315c74da9908491"
Sep 10 14:42:16.641064 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.641001    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d94d73e8e74a765b681ef5d452b05375ab78a41f6e6d235c315c74da9908491"} err="failed to get container status \"5d94d73e8e74a765b681ef5d452b05375ab78a41f6e6d235c315c74da9908491\": rpc error: code = NotFound desc = could not find container \"5d94d73e8e74a765b681ef5d452b05375ab78a41f6e6d235c315c74da9908491\": container with ID starting with 5d94d73e8e74a765b681ef5d452b05375ab78a41f6e6d235c315c74da9908491 not found: ID does not exist"
Sep 10 14:42:16.649940 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.649918    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"]
Sep 10 14:42:16.653156 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:16.653131    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-b2dfe7/explicit-model-b2dfe7-kserve-787cf94bfc-qv4df"]
Sep 10 14:42:18.332683 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:18.332647    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"]
Sep 10 14:42:18.333076 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:18.332935    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9" podUID="0f1b2b1f-ff80-4c73-ac93-edab890862c0" containerName="istio-proxy" containerID="cri-o://5a82c4b7a3d23de3b6481a636b17969f692d8672b9dd2f266a6511d9e6f3270f" gracePeriod=30
Sep 10 14:42:18.423408 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:18.423370    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d048c259-8537-4740-b548-5cc40285da44" path="/var/lib/kubelet/pods/d048c259-8537-4740-b548-5cc40285da44/volumes"
Sep 10 14:42:21.952756 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:21.952725    2547 patch_prober.go:28] interesting pod/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 14:42:21.953104 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:21.952779    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9" podUID="0f1b2b1f-ff80-4c73-ac93-edab890862c0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 14:42:23.601820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.601759    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:42:23.671163 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.671124    2547 generic.go:356] "Generic (PLEG): container finished" podID="0f1b2b1f-ff80-4c73-ac93-edab890862c0" containerID="5a82c4b7a3d23de3b6481a636b17969f692d8672b9dd2f266a6511d9e6f3270f" exitCode=0
Sep 10 14:42:23.671313 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.671175    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"
Sep 10 14:42:23.671313 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.671210    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9" event={"ID":"0f1b2b1f-ff80-4c73-ac93-edab890862c0","Type":"ContainerDied","Data":"5a82c4b7a3d23de3b6481a636b17969f692d8672b9dd2f266a6511d9e6f3270f"}
Sep 10 14:42:23.671313 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.671267    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9" event={"ID":"0f1b2b1f-ff80-4c73-ac93-edab890862c0","Type":"ContainerDied","Data":"0b3b0dd7126e0acda90c3a9f97bce187e8a3dc2aebd2725f3a4604c9eff940ec"}
Sep 10 14:42:23.671313 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.671291    2547 scope.go:117] "RemoveContainer" containerID="5a82c4b7a3d23de3b6481a636b17969f692d8672b9dd2f266a6511d9e6f3270f"
Sep 10 14:42:23.680372 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.680360    2547 scope.go:117] "RemoveContainer" containerID="e5e550b71347cf64774f0cf58565653f9df3e227c7ca23fa04b24c5ee611999e"
Sep 10 14:42:23.688295 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.688268    2547 scope.go:117] "RemoveContainer" containerID="5a82c4b7a3d23de3b6481a636b17969f692d8672b9dd2f266a6511d9e6f3270f"
Sep 10 14:42:23.688578 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:42:23.688560    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a82c4b7a3d23de3b6481a636b17969f692d8672b9dd2f266a6511d9e6f3270f\": container with ID starting with 5a82c4b7a3d23de3b6481a636b17969f692d8672b9dd2f266a6511d9e6f3270f not found: ID does not exist" containerID="5a82c4b7a3d23de3b6481a636b17969f692d8672b9dd2f266a6511d9e6f3270f"
Sep 10 14:42:23.688627 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.688588    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a82c4b7a3d23de3b6481a636b17969f692d8672b9dd2f266a6511d9e6f3270f"} err="failed to get container status \"5a82c4b7a3d23de3b6481a636b17969f692d8672b9dd2f266a6511d9e6f3270f\": rpc error: code = NotFound desc = could not find container \"5a82c4b7a3d23de3b6481a636b17969f692d8672b9dd2f266a6511d9e6f3270f\": container with ID starting with 5a82c4b7a3d23de3b6481a636b17969f692d8672b9dd2f266a6511d9e6f3270f not found: ID does not exist"
Sep 10 14:42:23.688627 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.688611    2547 scope.go:117] "RemoveContainer" containerID="e5e550b71347cf64774f0cf58565653f9df3e227c7ca23fa04b24c5ee611999e"
Sep 10 14:42:23.688909 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:42:23.688889    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5e550b71347cf64774f0cf58565653f9df3e227c7ca23fa04b24c5ee611999e\": container with ID starting with e5e550b71347cf64774f0cf58565653f9df3e227c7ca23fa04b24c5ee611999e not found: ID does not exist" containerID="e5e550b71347cf64774f0cf58565653f9df3e227c7ca23fa04b24c5ee611999e"
Sep 10 14:42:23.688972 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.688914    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5e550b71347cf64774f0cf58565653f9df3e227c7ca23fa04b24c5ee611999e"} err="failed to get container status \"e5e550b71347cf64774f0cf58565653f9df3e227c7ca23fa04b24c5ee611999e\": rpc error: code = NotFound desc = could not find container \"e5e550b71347cf64774f0cf58565653f9df3e227c7ca23fa04b24c5ee611999e\": container with ID starting with e5e550b71347cf64774f0cf58565653f9df3e227c7ca23fa04b24c5ee611999e not found: ID does not exist"
Sep 10 14:42:23.740965 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.740938    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istio-data\") pod \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") "
Sep 10 14:42:23.741118 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.740991    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-credential-socket\") pod \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") "
Sep 10 14:42:23.741118 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.741040    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istio-podinfo\") pod \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") "
Sep 10 14:42:23.741118 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.741072    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istiod-ca-cert\") pod \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") "
Sep 10 14:42:23.741118 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.741091    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istio-envoy\") pod \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") "
Sep 10 14:42:23.741118 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.741108    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istio-token\") pod \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") "
Sep 10 14:42:23.741390 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.741124    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-workload-socket\") pod \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") "
Sep 10 14:42:23.741390 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.741147    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7nmc\" (UniqueName: \"kubernetes.io/projected/0f1b2b1f-ff80-4c73-ac93-edab890862c0-kube-api-access-r7nmc\") pod \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") "
Sep 10 14:42:23.741390 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.741197    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-workload-certs\") pod \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\" (UID: \"0f1b2b1f-ff80-4c73-ac93-edab890862c0\") "
Sep 10 14:42:23.741390 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.741252    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istio-data" (OuterVolumeSpecName: "istio-data") pod "0f1b2b1f-ff80-4c73-ac93-edab890862c0" (UID: "0f1b2b1f-ff80-4c73-ac93-edab890862c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:23.741601 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.741395    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0f1b2b1f-ff80-4c73-ac93-edab890862c0" (UID: "0f1b2b1f-ff80-4c73-ac93-edab890862c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:23.741601 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.741478    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0f1b2b1f-ff80-4c73-ac93-edab890862c0" (UID: "0f1b2b1f-ff80-4c73-ac93-edab890862c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:42:23.741601 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.741587    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0f1b2b1f-ff80-4c73-ac93-edab890862c0" (UID: "0f1b2b1f-ff80-4c73-ac93-edab890862c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:23.741792 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.741613    2547 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istio-data\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:23.741792 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.741629    2547 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-credential-socket\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:23.741792 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.741652    2547 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istiod-ca-cert\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:23.741792 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.741750    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0f1b2b1f-ff80-4c73-ac93-edab890862c0" (UID: "0f1b2b1f-ff80-4c73-ac93-edab890862c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:23.743231 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.743202    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0f1b2b1f-ff80-4c73-ac93-edab890862c0" (UID: "0f1b2b1f-ff80-4c73-ac93-edab890862c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 14:42:23.743384 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.743360    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1b2b1f-ff80-4c73-ac93-edab890862c0-kube-api-access-r7nmc" (OuterVolumeSpecName: "kube-api-access-r7nmc") pod "0f1b2b1f-ff80-4c73-ac93-edab890862c0" (UID: "0f1b2b1f-ff80-4c73-ac93-edab890862c0"). InnerVolumeSpecName "kube-api-access-r7nmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:42:23.743644 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.743620    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0f1b2b1f-ff80-4c73-ac93-edab890862c0" (UID: "0f1b2b1f-ff80-4c73-ac93-edab890862c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:23.743644 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.743633    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istio-token" (OuterVolumeSpecName: "istio-token") pod "0f1b2b1f-ff80-4c73-ac93-edab890862c0" (UID: "0f1b2b1f-ff80-4c73-ac93-edab890862c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:42:23.842526 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.842487    2547 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istio-podinfo\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:23.842526 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.842520    2547 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istio-envoy\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:23.842526 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.842529    2547 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f1b2b1f-ff80-4c73-ac93-edab890862c0-istio-token\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:23.842526 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.842537    2547 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-workload-socket\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:23.842825 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.842546    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7nmc\" (UniqueName: \"kubernetes.io/projected/0f1b2b1f-ff80-4c73-ac93-edab890862c0-kube-api-access-r7nmc\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:23.842825 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.842557    2547 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1b2b1f-ff80-4c73-ac93-edab890862c0-workload-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:23.994898 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.994864    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"]
Sep 10 14:42:23.998904 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:23.998876    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-b2dfe7-openshift-default-75c79cbf4d-lllk9"]
Sep 10 14:42:24.117776 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:24.117730    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:42:24.422080 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:24.422043    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f1b2b1f-ff80-4c73-ac93-edab890862c0" path="/var/lib/kubelet/pods/0f1b2b1f-ff80-4c73-ac93-edab890862c0/volumes"
Sep 10 14:42:32.275426 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.275388    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"]
Sep 10 14:42:32.275830 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.275766    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f1b2b1f-ff80-4c73-ac93-edab890862c0" containerName="istio-proxy"
Sep 10 14:42:32.275830 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.275779    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f1b2b1f-ff80-4c73-ac93-edab890862c0" containerName="istio-proxy"
Sep 10 14:42:32.275830 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.275790    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f1b2b1f-ff80-4c73-ac93-edab890862c0" containerName="istio-proxy"
Sep 10 14:42:32.275830 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.275795    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f1b2b1f-ff80-4c73-ac93-edab890862c0" containerName="istio-proxy"
Sep 10 14:42:32.275830 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.275804    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d048c259-8537-4740-b548-5cc40285da44" containerName="main"
Sep 10 14:42:32.275830 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.275810    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="d048c259-8537-4740-b548-5cc40285da44" containerName="main"
Sep 10 14:42:32.276025 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.275878    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f1b2b1f-ff80-4c73-ac93-edab890862c0" containerName="istio-proxy"
Sep 10 14:42:32.276025 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.275886    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="d048c259-8537-4740-b548-5cc40285da44" containerName="main"
Sep 10 14:42:32.276025 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.275895    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f1b2b1f-ff80-4c73-ac93-edab890862c0" containerName="istio-proxy"
Sep 10 14:42:32.280393 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.280376    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"
Sep 10 14:42:32.283050 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.283023    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-1ad4f6\"/\"default-dockercfg-fnhgr\""
Sep 10 14:42:32.284072 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.284055    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-1ad4f6\"/\"noten-model-1ad4f6-kserve-self-signed-certs\""
Sep 10 14:42:32.284135 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.284069    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-1ad4f6\"/\"openshift-service-ca.crt\""
Sep 10 14:42:32.284176 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.284069    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-1ad4f6\"/\"kube-root-ca.crt\""
Sep 10 14:42:32.296419 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.296385    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"]
Sep 10 14:42:32.421176 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.421146    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae44b666-4a54-4e51-8d23-d67d784d91e7-home\") pod \"noten-model-1ad4f6-kserve-648f5bd46c-h94vt\" (UID: \"ae44b666-4a54-4e51-8d23-d67d784d91e7\") " pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"
Sep 10 14:42:32.421377 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.421189    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62b7g\" (UniqueName: \"kubernetes.io/projected/ae44b666-4a54-4e51-8d23-d67d784d91e7-kube-api-access-62b7g\") pod \"noten-model-1ad4f6-kserve-648f5bd46c-h94vt\" (UID: \"ae44b666-4a54-4e51-8d23-d67d784d91e7\") " pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"
Sep 10 14:42:32.421377 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.421302    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae44b666-4a54-4e51-8d23-d67d784d91e7-dshm\") pod \"noten-model-1ad4f6-kserve-648f5bd46c-h94vt\" (UID: \"ae44b666-4a54-4e51-8d23-d67d784d91e7\") " pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"
Sep 10 14:42:32.421377 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.421372    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae44b666-4a54-4e51-8d23-d67d784d91e7-tls-certs\") pod \"noten-model-1ad4f6-kserve-648f5bd46c-h94vt\" (UID: \"ae44b666-4a54-4e51-8d23-d67d784d91e7\") " pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"
Sep 10 14:42:32.421544 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.421434    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae44b666-4a54-4e51-8d23-d67d784d91e7-tmp-dir\") pod \"noten-model-1ad4f6-kserve-648f5bd46c-h94vt\" (UID: \"ae44b666-4a54-4e51-8d23-d67d784d91e7\") " pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"
Sep 10 14:42:32.421544 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.421474    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae44b666-4a54-4e51-8d23-d67d784d91e7-model-cache\") pod \"noten-model-1ad4f6-kserve-648f5bd46c-h94vt\" (UID: \"ae44b666-4a54-4e51-8d23-d67d784d91e7\") " pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"
Sep 10 14:42:32.522859 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.522822    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62b7g\" (UniqueName: \"kubernetes.io/projected/ae44b666-4a54-4e51-8d23-d67d784d91e7-kube-api-access-62b7g\") pod \"noten-model-1ad4f6-kserve-648f5bd46c-h94vt\" (UID: \"ae44b666-4a54-4e51-8d23-d67d784d91e7\") " pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"
Sep 10 14:42:32.523053 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.522884    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae44b666-4a54-4e51-8d23-d67d784d91e7-dshm\") pod \"noten-model-1ad4f6-kserve-648f5bd46c-h94vt\" (UID: \"ae44b666-4a54-4e51-8d23-d67d784d91e7\") " pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"
Sep 10 14:42:32.523053 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.522923    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae44b666-4a54-4e51-8d23-d67d784d91e7-tls-certs\") pod \"noten-model-1ad4f6-kserve-648f5bd46c-h94vt\" (UID: \"ae44b666-4a54-4e51-8d23-d67d784d91e7\") " pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"
Sep 10 14:42:32.523188 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.523144    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae44b666-4a54-4e51-8d23-d67d784d91e7-tmp-dir\") pod \"noten-model-1ad4f6-kserve-648f5bd46c-h94vt\" (UID: \"ae44b666-4a54-4e51-8d23-d67d784d91e7\") " pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"
Sep 10 14:42:32.523258 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.523202    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae44b666-4a54-4e51-8d23-d67d784d91e7-model-cache\") pod \"noten-model-1ad4f6-kserve-648f5bd46c-h94vt\" (UID: \"ae44b666-4a54-4e51-8d23-d67d784d91e7\") " pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"
Sep 10 14:42:32.523313 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.523257    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae44b666-4a54-4e51-8d23-d67d784d91e7-home\") pod \"noten-model-1ad4f6-kserve-648f5bd46c-h94vt\" (UID: \"ae44b666-4a54-4e51-8d23-d67d784d91e7\") " pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"
Sep 10 14:42:32.523505 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.523453    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae44b666-4a54-4e51-8d23-d67d784d91e7-tmp-dir\") pod \"noten-model-1ad4f6-kserve-648f5bd46c-h94vt\" (UID: \"ae44b666-4a54-4e51-8d23-d67d784d91e7\") " pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"
Sep 10 14:42:32.523587 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.523511    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae44b666-4a54-4e51-8d23-d67d784d91e7-home\") pod \"noten-model-1ad4f6-kserve-648f5bd46c-h94vt\" (UID: \"ae44b666-4a54-4e51-8d23-d67d784d91e7\") " pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"
Sep 10 14:42:32.523633 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.523590    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae44b666-4a54-4e51-8d23-d67d784d91e7-model-cache\") pod \"noten-model-1ad4f6-kserve-648f5bd46c-h94vt\" (UID: \"ae44b666-4a54-4e51-8d23-d67d784d91e7\") " pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"
Sep 10 14:42:32.525184 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.525160    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae44b666-4a54-4e51-8d23-d67d784d91e7-dshm\") pod \"noten-model-1ad4f6-kserve-648f5bd46c-h94vt\" (UID: \"ae44b666-4a54-4e51-8d23-d67d784d91e7\") " pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"
Sep 10 14:42:32.525355 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.525338    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae44b666-4a54-4e51-8d23-d67d784d91e7-tls-certs\") pod \"noten-model-1ad4f6-kserve-648f5bd46c-h94vt\" (UID: \"ae44b666-4a54-4e51-8d23-d67d784d91e7\") " pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"
Sep 10 14:42:32.530541 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.530485    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62b7g\" (UniqueName: \"kubernetes.io/projected/ae44b666-4a54-4e51-8d23-d67d784d91e7-kube-api-access-62b7g\") pod \"noten-model-1ad4f6-kserve-648f5bd46c-h94vt\" (UID: \"ae44b666-4a54-4e51-8d23-d67d784d91e7\") " pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"
Sep 10 14:42:32.592272 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.592236    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"
Sep 10 14:42:32.863219 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:32.863192    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"]
Sep 10 14:42:32.864784 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:42:32.864748    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae44b666_4a54_4e51_8d23_d67d784d91e7.slice/crio-a95957817c52de8099e2798e21dd9e15363831104f443faecb6cafcb4419c5f5 WatchSource:0}: Error finding container a95957817c52de8099e2798e21dd9e15363831104f443faecb6cafcb4419c5f5: Status 404 returned error can't find the container with id a95957817c52de8099e2798e21dd9e15363831104f443faecb6cafcb4419c5f5
Sep 10 14:42:33.732793 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:33.732753    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt" event={"ID":"ae44b666-4a54-4e51-8d23-d67d784d91e7","Type":"ContainerStarted","Data":"b297cbc33d8f1f142a85e7fbd13406c46b9383e57dd32d298e0036f2866d693b"}
Sep 10 14:42:33.733184 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:33.732801    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt" event={"ID":"ae44b666-4a54-4e51-8d23-d67d784d91e7","Type":"ContainerStarted","Data":"a95957817c52de8099e2798e21dd9e15363831104f443faecb6cafcb4419c5f5"}
Sep 10 14:42:33.749120 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:33.748972    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt" podStartSLOduration=1.748957323 podStartE2EDuration="1.748957323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:42:33.748604112 +0000 UTC m=+1365.790478594" watchObservedRunningTime="2026-09-10 14:42:33.748957323 +0000 UTC m=+1365.790831857"
Sep 10 14:42:38.172892 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:38.172859    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"]
Sep 10 14:42:41.275540 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:41.275456    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"]
Sep 10 14:42:42.246130 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:42.246092    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"]
Sep 10 14:42:42.376116 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:42.376081    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c892fe-288kk"]
Sep 10 14:42:42.379707 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:42.379683    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c892fe-288kk"
Sep 10 14:42:42.382510 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:42.382485    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 14:42:42.382648 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:42.382485    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xnxsg\""
Sep 10 14:42:42.388241 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:42.388218    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c892fe-288kk"]
Sep 10 14:42:42.424976 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:42.424764    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd61096f-6641-45b3-a006-5be8c77e510b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c892fe-288kk\" (UID: \"fd61096f-6641-45b3-a006-5be8c77e510b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c892fe-288kk"
Sep 10 14:42:42.525630 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:42.525539    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd61096f-6641-45b3-a006-5be8c77e510b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c892fe-288kk\" (UID: \"fd61096f-6641-45b3-a006-5be8c77e510b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c892fe-288kk"
Sep 10 14:42:42.526181 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:42.526151    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd61096f-6641-45b3-a006-5be8c77e510b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c892fe-288kk\" (UID: \"fd61096f-6641-45b3-a006-5be8c77e510b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c892fe-288kk"
Sep 10 14:42:42.592759 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:42.592689    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"
Sep 10 14:42:42.691463 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:42.691424    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c892fe-288kk"
Sep 10 14:42:42.964141 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:42.964113    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c892fe-288kk"]
Sep 10 14:42:42.965945 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:42:42.965913    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd61096f_6641_45b3_a006_5be8c77e510b.slice/crio-4d895582bc7b9e4cf0ba4d7c2a167bdaba65fd65606f0c5800020672ad0cf289 WatchSource:0}: Error finding container 4d895582bc7b9e4cf0ba4d7c2a167bdaba65fd65606f0c5800020672ad0cf289: Status 404 returned error can't find the container with id 4d895582bc7b9e4cf0ba4d7c2a167bdaba65fd65606f0c5800020672ad0cf289
Sep 10 14:42:43.788572 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:43.788535    2547 generic.go:356] "Generic (PLEG): container finished" podID="fd61096f-6641-45b3-a006-5be8c77e510b" containerID="450161f09c1d8d5ef4f35be1c4e2315251ed83aeb266eec913358efaeb8f2450" exitCode=0
Sep 10 14:42:43.788971 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:43.788627    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c892fe-288kk" event={"ID":"fd61096f-6641-45b3-a006-5be8c77e510b","Type":"ContainerDied","Data":"450161f09c1d8d5ef4f35be1c4e2315251ed83aeb266eec913358efaeb8f2450"}
Sep 10 14:42:43.788971 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:43.788687    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c892fe-288kk" event={"ID":"fd61096f-6641-45b3-a006-5be8c77e510b","Type":"ContainerStarted","Data":"4d895582bc7b9e4cf0ba4d7c2a167bdaba65fd65606f0c5800020672ad0cf289"}
Sep 10 14:42:44.935471 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:44.935446    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c892fe-288kk"
Sep 10 14:42:44.945421 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:44.945399    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd61096f-6641-45b3-a006-5be8c77e510b-maas-api-service-ca\") pod \"fd61096f-6641-45b3-a006-5be8c77e510b\" (UID: \"fd61096f-6641-45b3-a006-5be8c77e510b\") "
Sep 10 14:42:44.945802 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:44.945780    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd61096f-6641-45b3-a006-5be8c77e510b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fd61096f-6641-45b3-a006-5be8c77e510b" (UID: "fd61096f-6641-45b3-a006-5be8c77e510b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:42:45.046649 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:45.046559    2547 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd61096f-6641-45b3-a006-5be8c77e510b-maas-api-service-ca\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:45.802178 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:45.802150    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c892fe-288kk"
Sep 10 14:42:45.802354 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:45.802187    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c892fe-288kk" event={"ID":"fd61096f-6641-45b3-a006-5be8c77e510b","Type":"ContainerDied","Data":"4d895582bc7b9e4cf0ba4d7c2a167bdaba65fd65606f0c5800020672ad0cf289"}
Sep 10 14:42:45.802354 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:45.802242    2547 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d895582bc7b9e4cf0ba4d7c2a167bdaba65fd65606f0c5800020672ad0cf289"
Sep 10 14:42:48.628495 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:48.628456    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:42:53.144297 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.144255    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c892fe-288kk"]
Sep 10 14:42:53.151460 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.151434    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c892fe-288kk"]
Sep 10 14:42:53.192711 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.192660    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt" podUID="ae44b666-4a54-4e51-8d23-d67d784d91e7" containerName="main" containerID="cri-o://b297cbc33d8f1f142a85e7fbd13406c46b9383e57dd32d298e0036f2866d693b" gracePeriod=45
Sep 10 14:42:53.445506 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.445482    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"
Sep 10 14:42:53.522412 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.522381    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62b7g\" (UniqueName: \"kubernetes.io/projected/ae44b666-4a54-4e51-8d23-d67d784d91e7-kube-api-access-62b7g\") pod \"ae44b666-4a54-4e51-8d23-d67d784d91e7\" (UID: \"ae44b666-4a54-4e51-8d23-d67d784d91e7\") "
Sep 10 14:42:53.522593 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.522423    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae44b666-4a54-4e51-8d23-d67d784d91e7-home\") pod \"ae44b666-4a54-4e51-8d23-d67d784d91e7\" (UID: \"ae44b666-4a54-4e51-8d23-d67d784d91e7\") "
Sep 10 14:42:53.522593 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.522448    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae44b666-4a54-4e51-8d23-d67d784d91e7-model-cache\") pod \"ae44b666-4a54-4e51-8d23-d67d784d91e7\" (UID: \"ae44b666-4a54-4e51-8d23-d67d784d91e7\") "
Sep 10 14:42:53.522593 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.522468    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae44b666-4a54-4e51-8d23-d67d784d91e7-tls-certs\") pod \"ae44b666-4a54-4e51-8d23-d67d784d91e7\" (UID: \"ae44b666-4a54-4e51-8d23-d67d784d91e7\") "
Sep 10 14:42:53.522593 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.522509    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae44b666-4a54-4e51-8d23-d67d784d91e7-dshm\") pod \"ae44b666-4a54-4e51-8d23-d67d784d91e7\" (UID: \"ae44b666-4a54-4e51-8d23-d67d784d91e7\") "
Sep 10 14:42:53.522593 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.522532    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae44b666-4a54-4e51-8d23-d67d784d91e7-tmp-dir\") pod \"ae44b666-4a54-4e51-8d23-d67d784d91e7\" (UID: \"ae44b666-4a54-4e51-8d23-d67d784d91e7\") "
Sep 10 14:42:53.522933 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.522764    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae44b666-4a54-4e51-8d23-d67d784d91e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae44b666-4a54-4e51-8d23-d67d784d91e7" (UID: "ae44b666-4a54-4e51-8d23-d67d784d91e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:53.522933 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.522780    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae44b666-4a54-4e51-8d23-d67d784d91e7-home" (OuterVolumeSpecName: "home") pod "ae44b666-4a54-4e51-8d23-d67d784d91e7" (UID: "ae44b666-4a54-4e51-8d23-d67d784d91e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:53.522933 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.522884    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae44b666-4a54-4e51-8d23-d67d784d91e7-home\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:53.522933 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.522902    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae44b666-4a54-4e51-8d23-d67d784d91e7-model-cache\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:53.522933 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.522896    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae44b666-4a54-4e51-8d23-d67d784d91e7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae44b666-4a54-4e51-8d23-d67d784d91e7" (UID: "ae44b666-4a54-4e51-8d23-d67d784d91e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:53.524752 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.524727    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae44b666-4a54-4e51-8d23-d67d784d91e7-dshm" (OuterVolumeSpecName: "dshm") pod "ae44b666-4a54-4e51-8d23-d67d784d91e7" (UID: "ae44b666-4a54-4e51-8d23-d67d784d91e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:53.524752 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.524742    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae44b666-4a54-4e51-8d23-d67d784d91e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae44b666-4a54-4e51-8d23-d67d784d91e7" (UID: "ae44b666-4a54-4e51-8d23-d67d784d91e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:42:53.524900 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.524824    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae44b666-4a54-4e51-8d23-d67d784d91e7-kube-api-access-62b7g" (OuterVolumeSpecName: "kube-api-access-62b7g") pod "ae44b666-4a54-4e51-8d23-d67d784d91e7" (UID: "ae44b666-4a54-4e51-8d23-d67d784d91e7"). InnerVolumeSpecName "kube-api-access-62b7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:42:53.624120 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.624085    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae44b666-4a54-4e51-8d23-d67d784d91e7-tls-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:53.624120 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.624112    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae44b666-4a54-4e51-8d23-d67d784d91e7-dshm\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:53.624345 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.624142    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae44b666-4a54-4e51-8d23-d67d784d91e7-tmp-dir\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:53.624345 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.624155    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62b7g\" (UniqueName: \"kubernetes.io/projected/ae44b666-4a54-4e51-8d23-d67d784d91e7-kube-api-access-62b7g\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:53.849874 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.849789    2547 generic.go:356] "Generic (PLEG): container finished" podID="ae44b666-4a54-4e51-8d23-d67d784d91e7" containerID="b297cbc33d8f1f142a85e7fbd13406c46b9383e57dd32d298e0036f2866d693b" exitCode=0
Sep 10 14:42:53.849874 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.849834    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"
Sep 10 14:42:53.850058 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.849876    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt" event={"ID":"ae44b666-4a54-4e51-8d23-d67d784d91e7","Type":"ContainerDied","Data":"b297cbc33d8f1f142a85e7fbd13406c46b9383e57dd32d298e0036f2866d693b"}
Sep 10 14:42:53.850058 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.849920    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt" event={"ID":"ae44b666-4a54-4e51-8d23-d67d784d91e7","Type":"ContainerDied","Data":"a95957817c52de8099e2798e21dd9e15363831104f443faecb6cafcb4419c5f5"}
Sep 10 14:42:53.850058 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.849945    2547 scope.go:117] "RemoveContainer" containerID="b297cbc33d8f1f142a85e7fbd13406c46b9383e57dd32d298e0036f2866d693b"
Sep 10 14:42:53.859357 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.859338    2547 scope.go:117] "RemoveContainer" containerID="b297cbc33d8f1f142a85e7fbd13406c46b9383e57dd32d298e0036f2866d693b"
Sep 10 14:42:53.859625 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:42:53.859607    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b297cbc33d8f1f142a85e7fbd13406c46b9383e57dd32d298e0036f2866d693b\": container with ID starting with b297cbc33d8f1f142a85e7fbd13406c46b9383e57dd32d298e0036f2866d693b not found: ID does not exist" containerID="b297cbc33d8f1f142a85e7fbd13406c46b9383e57dd32d298e0036f2866d693b"
Sep 10 14:42:53.859673 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.859635    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b297cbc33d8f1f142a85e7fbd13406c46b9383e57dd32d298e0036f2866d693b"} err="failed to get container status \"b297cbc33d8f1f142a85e7fbd13406c46b9383e57dd32d298e0036f2866d693b\": rpc error: code = NotFound desc = could not find container \"b297cbc33d8f1f142a85e7fbd13406c46b9383e57dd32d298e0036f2866d693b\": container with ID starting with b297cbc33d8f1f142a85e7fbd13406c46b9383e57dd32d298e0036f2866d693b not found: ID does not exist"
Sep 10 14:42:53.872075 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.872054    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"]
Sep 10 14:42:53.875560 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:53.875539    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-1ad4f6/noten-model-1ad4f6-kserve-648f5bd46c-h94vt"]
Sep 10 14:42:54.421854 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:54.421823    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae44b666-4a54-4e51-8d23-d67d784d91e7" path="/var/lib/kubelet/pods/ae44b666-4a54-4e51-8d23-d67d784d91e7/volumes"
Sep 10 14:42:54.422219 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:54.422149    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd61096f-6641-45b3-a006-5be8c77e510b" path="/var/lib/kubelet/pods/fd61096f-6641-45b3-a006-5be8c77e510b/volumes"
Sep 10 14:42:56.299472 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.299426    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95" podUID="8d5094f1-230b-4ca1-a6a9-e99a92105b4f" containerName="main" containerID="cri-o://3efe60404fbb54054804ad27762f9ee208c5bfbf237f44b96bd485c31ccdeae4" gracePeriod=45
Sep 10 14:42:56.549791 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.549729    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:42:56.650622 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.650591    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-model-cache\") pod \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\" (UID: \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\") "
Sep 10 14:42:56.650803 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.650631    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v97dg\" (UniqueName: \"kubernetes.io/projected/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-kube-api-access-v97dg\") pod \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\" (UID: \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\") "
Sep 10 14:42:56.650803 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.650692    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-tls-certs\") pod \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\" (UID: \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\") "
Sep 10 14:42:56.650803 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.650756    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-tmp-dir\") pod \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\" (UID: \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\") "
Sep 10 14:42:56.650803 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.650779    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-home\") pod \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\" (UID: \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\") "
Sep 10 14:42:56.650803 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.650796    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-dshm\") pod \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\" (UID: \"8d5094f1-230b-4ca1-a6a9-e99a92105b4f\") "
Sep 10 14:42:56.651036 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.650920    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-model-cache" (OuterVolumeSpecName: "model-cache") pod "8d5094f1-230b-4ca1-a6a9-e99a92105b4f" (UID: "8d5094f1-230b-4ca1-a6a9-e99a92105b4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:56.651036 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.651029    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8d5094f1-230b-4ca1-a6a9-e99a92105b4f" (UID: "8d5094f1-230b-4ca1-a6a9-e99a92105b4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:56.651142 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.651060    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-home" (OuterVolumeSpecName: "home") pod "8d5094f1-230b-4ca1-a6a9-e99a92105b4f" (UID: "8d5094f1-230b-4ca1-a6a9-e99a92105b4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:56.651206 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.651140    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-tmp-dir\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:56.651206 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.651157    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-home\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:56.651206 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.651169    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-model-cache\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:56.652947 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.652918    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d5094f1-230b-4ca1-a6a9-e99a92105b4f" (UID: "8d5094f1-230b-4ca1-a6a9-e99a92105b4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:42:56.653058 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.652966    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-dshm" (OuterVolumeSpecName: "dshm") pod "8d5094f1-230b-4ca1-a6a9-e99a92105b4f" (UID: "8d5094f1-230b-4ca1-a6a9-e99a92105b4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:56.653058 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.652975    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-kube-api-access-v97dg" (OuterVolumeSpecName: "kube-api-access-v97dg") pod "8d5094f1-230b-4ca1-a6a9-e99a92105b4f" (UID: "8d5094f1-230b-4ca1-a6a9-e99a92105b4f"). InnerVolumeSpecName "kube-api-access-v97dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:42:56.751936 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.751896    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-tls-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:56.751936 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.751929    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-dshm\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:56.751936 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.751938    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v97dg\" (UniqueName: \"kubernetes.io/projected/8d5094f1-230b-4ca1-a6a9-e99a92105b4f-kube-api-access-v97dg\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:56.868557 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.868481    2547 generic.go:356] "Generic (PLEG): container finished" podID="8d5094f1-230b-4ca1-a6a9-e99a92105b4f" containerID="3efe60404fbb54054804ad27762f9ee208c5bfbf237f44b96bd485c31ccdeae4" exitCode=0
Sep 10 14:42:56.868557 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.868521    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"
Sep 10 14:42:56.868557 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.868529    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95" event={"ID":"8d5094f1-230b-4ca1-a6a9-e99a92105b4f","Type":"ContainerDied","Data":"3efe60404fbb54054804ad27762f9ee208c5bfbf237f44b96bd485c31ccdeae4"}
Sep 10 14:42:56.868834 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.868561    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95" event={"ID":"8d5094f1-230b-4ca1-a6a9-e99a92105b4f","Type":"ContainerDied","Data":"a5c6eb2c6742e382d8e01ff65fbb58f633d80485585318bba980327d526d65e1"}
Sep 10 14:42:56.868834 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.868578    2547 scope.go:117] "RemoveContainer" containerID="3efe60404fbb54054804ad27762f9ee208c5bfbf237f44b96bd485c31ccdeae4"
Sep 10 14:42:56.878248 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.878229    2547 scope.go:117] "RemoveContainer" containerID="3efe60404fbb54054804ad27762f9ee208c5bfbf237f44b96bd485c31ccdeae4"
Sep 10 14:42:56.878520 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:42:56.878502    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3efe60404fbb54054804ad27762f9ee208c5bfbf237f44b96bd485c31ccdeae4\": container with ID starting with 3efe60404fbb54054804ad27762f9ee208c5bfbf237f44b96bd485c31ccdeae4 not found: ID does not exist" containerID="3efe60404fbb54054804ad27762f9ee208c5bfbf237f44b96bd485c31ccdeae4"
Sep 10 14:42:56.878572 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.878529    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3efe60404fbb54054804ad27762f9ee208c5bfbf237f44b96bd485c31ccdeae4"} err="failed to get container status \"3efe60404fbb54054804ad27762f9ee208c5bfbf237f44b96bd485c31ccdeae4\": rpc error: code = NotFound desc = could not find container \"3efe60404fbb54054804ad27762f9ee208c5bfbf237f44b96bd485c31ccdeae4\": container with ID starting with 3efe60404fbb54054804ad27762f9ee208c5bfbf237f44b96bd485c31ccdeae4 not found: ID does not exist"
Sep 10 14:42:56.889960 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.889936    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"]
Sep 10 14:42:56.893326 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:56.893304    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c892fe/test-model-c892fe-kserve-68cf795699-xbd95"]
Sep 10 14:42:57.265779 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.265736    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4" podUID="0bf48933-cbd3-44b1-beae-422b145bb9c9" containerName="main" containerID="cri-o://963f11aa432c74046a2b86fb5cf10c6fcdf70b033110a6065fa938166f44b411" gracePeriod=45
Sep 10 14:42:57.523786 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.523723    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:42:57.558607 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.558577    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9pdl\" (UniqueName: \"kubernetes.io/projected/0bf48933-cbd3-44b1-beae-422b145bb9c9-kube-api-access-r9pdl\") pod \"0bf48933-cbd3-44b1-beae-422b145bb9c9\" (UID: \"0bf48933-cbd3-44b1-beae-422b145bb9c9\") "
Sep 10 14:42:57.558607 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.558607    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bf48933-cbd3-44b1-beae-422b145bb9c9-dshm\") pod \"0bf48933-cbd3-44b1-beae-422b145bb9c9\" (UID: \"0bf48933-cbd3-44b1-beae-422b145bb9c9\") "
Sep 10 14:42:57.558834 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.558637    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bf48933-cbd3-44b1-beae-422b145bb9c9-tmp-dir\") pod \"0bf48933-cbd3-44b1-beae-422b145bb9c9\" (UID: \"0bf48933-cbd3-44b1-beae-422b145bb9c9\") "
Sep 10 14:42:57.558834 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.558657    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bf48933-cbd3-44b1-beae-422b145bb9c9-model-cache\") pod \"0bf48933-cbd3-44b1-beae-422b145bb9c9\" (UID: \"0bf48933-cbd3-44b1-beae-422b145bb9c9\") "
Sep 10 14:42:57.558834 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.558676    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bf48933-cbd3-44b1-beae-422b145bb9c9-home\") pod \"0bf48933-cbd3-44b1-beae-422b145bb9c9\" (UID: \"0bf48933-cbd3-44b1-beae-422b145bb9c9\") "
Sep 10 14:42:57.558834 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.558748    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bf48933-cbd3-44b1-beae-422b145bb9c9-tls-certs\") pod \"0bf48933-cbd3-44b1-beae-422b145bb9c9\" (UID: \"0bf48933-cbd3-44b1-beae-422b145bb9c9\") "
Sep 10 14:42:57.559425 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.559218    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf48933-cbd3-44b1-beae-422b145bb9c9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0bf48933-cbd3-44b1-beae-422b145bb9c9" (UID: "0bf48933-cbd3-44b1-beae-422b145bb9c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:57.559425 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.559360    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf48933-cbd3-44b1-beae-422b145bb9c9-home" (OuterVolumeSpecName: "home") pod "0bf48933-cbd3-44b1-beae-422b145bb9c9" (UID: "0bf48933-cbd3-44b1-beae-422b145bb9c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:57.559599 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.559466    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf48933-cbd3-44b1-beae-422b145bb9c9-model-cache" (OuterVolumeSpecName: "model-cache") pod "0bf48933-cbd3-44b1-beae-422b145bb9c9" (UID: "0bf48933-cbd3-44b1-beae-422b145bb9c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:57.561598 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.559677    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bf48933-cbd3-44b1-beae-422b145bb9c9-tmp-dir\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:57.561598 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.559728    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bf48933-cbd3-44b1-beae-422b145bb9c9-home\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:57.563734 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.562168    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bf48933-cbd3-44b1-beae-422b145bb9c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0bf48933-cbd3-44b1-beae-422b145bb9c9" (UID: "0bf48933-cbd3-44b1-beae-422b145bb9c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:42:57.563734 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.562560    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf48933-cbd3-44b1-beae-422b145bb9c9-kube-api-access-r9pdl" (OuterVolumeSpecName: "kube-api-access-r9pdl") pod "0bf48933-cbd3-44b1-beae-422b145bb9c9" (UID: "0bf48933-cbd3-44b1-beae-422b145bb9c9"). InnerVolumeSpecName "kube-api-access-r9pdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:42:57.571483 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.571456    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf48933-cbd3-44b1-beae-422b145bb9c9-dshm" (OuterVolumeSpecName: "dshm") pod "0bf48933-cbd3-44b1-beae-422b145bb9c9" (UID: "0bf48933-cbd3-44b1-beae-422b145bb9c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:57.661055 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.661020    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9pdl\" (UniqueName: \"kubernetes.io/projected/0bf48933-cbd3-44b1-beae-422b145bb9c9-kube-api-access-r9pdl\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:57.661055 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.661052    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bf48933-cbd3-44b1-beae-422b145bb9c9-dshm\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:57.661271 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.661066    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bf48933-cbd3-44b1-beae-422b145bb9c9-model-cache\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:57.661271 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.661075    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bf48933-cbd3-44b1-beae-422b145bb9c9-tls-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:42:57.877729 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.877621    2547 generic.go:356] "Generic (PLEG): container finished" podID="0bf48933-cbd3-44b1-beae-422b145bb9c9" containerID="963f11aa432c74046a2b86fb5cf10c6fcdf70b033110a6065fa938166f44b411" exitCode=0
Sep 10 14:42:57.877729 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.877687    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"
Sep 10 14:42:57.877971 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.877724    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4" event={"ID":"0bf48933-cbd3-44b1-beae-422b145bb9c9","Type":"ContainerDied","Data":"963f11aa432c74046a2b86fb5cf10c6fcdf70b033110a6065fa938166f44b411"}
Sep 10 14:42:57.877971 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.877781    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4" event={"ID":"0bf48933-cbd3-44b1-beae-422b145bb9c9","Type":"ContainerDied","Data":"18c1eee5208e38d4738a2abd8da2a9b3d6b6ad6350f276e0f9b405404ef1f931"}
Sep 10 14:42:57.877971 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.877806    2547 scope.go:117] "RemoveContainer" containerID="963f11aa432c74046a2b86fb5cf10c6fcdf70b033110a6065fa938166f44b411"
Sep 10 14:42:57.886758 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.886740    2547 scope.go:117] "RemoveContainer" containerID="963f11aa432c74046a2b86fb5cf10c6fcdf70b033110a6065fa938166f44b411"
Sep 10 14:42:57.887046 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:42:57.887027    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963f11aa432c74046a2b86fb5cf10c6fcdf70b033110a6065fa938166f44b411\": container with ID starting with 963f11aa432c74046a2b86fb5cf10c6fcdf70b033110a6065fa938166f44b411 not found: ID does not exist" containerID="963f11aa432c74046a2b86fb5cf10c6fcdf70b033110a6065fa938166f44b411"
Sep 10 14:42:57.887111 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.887054    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963f11aa432c74046a2b86fb5cf10c6fcdf70b033110a6065fa938166f44b411"} err="failed to get container status \"963f11aa432c74046a2b86fb5cf10c6fcdf70b033110a6065fa938166f44b411\": rpc error: code = NotFound desc = could not find container \"963f11aa432c74046a2b86fb5cf10c6fcdf70b033110a6065fa938166f44b411\": container with ID starting with 963f11aa432c74046a2b86fb5cf10c6fcdf70b033110a6065fa938166f44b411 not found: ID does not exist"
Sep 10 14:42:57.898943 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.898913    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"]
Sep 10 14:42:57.900964 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:57.900943    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2365fe/test-model-2365fe-kserve-5b46dddcdc-kj4k4"]
Sep 10 14:42:58.421681 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:58.421647    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bf48933-cbd3-44b1-beae-422b145bb9c9" path="/var/lib/kubelet/pods/0bf48933-cbd3-44b1-beae-422b145bb9c9/volumes"
Sep 10 14:42:58.422061 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:58.422047    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d5094f1-230b-4ca1-a6a9-e99a92105b4f" path="/var/lib/kubelet/pods/8d5094f1-230b-4ca1-a6a9-e99a92105b4f/volumes"
Sep 10 14:42:58.735383 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:58.735297    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:42:59.761180 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:59.761149    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"]
Sep 10 14:42:59.761620 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:42:59.761347    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77" podUID="8159cea0-0584-4d7b-8532-a136531ec0ed" containerName="istio-proxy" containerID="cri-o://32a308f5a39508b59ee763ce75e686919d574674090640ebc9858e576a351359" gracePeriod=30
Sep 10 14:43:00.678100 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:00.678063    2547 patch_prober.go:28] interesting pod/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 14:43:00.678257 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:00.678123    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77" podUID="8159cea0-0584-4d7b-8532-a136531ec0ed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 14:43:04.921433 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:04.921401    2547 generic.go:356] "Generic (PLEG): container finished" podID="8159cea0-0584-4d7b-8532-a136531ec0ed" containerID="32a308f5a39508b59ee763ce75e686919d574674090640ebc9858e576a351359" exitCode=0
Sep 10 14:43:04.921815 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:04.921477    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77" event={"ID":"8159cea0-0584-4d7b-8532-a136531ec0ed","Type":"ContainerDied","Data":"32a308f5a39508b59ee763ce75e686919d574674090640ebc9858e576a351359"}
Sep 10 14:43:04.921815 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:04.921558    2547 scope.go:117] "RemoveContainer" containerID="d6ae9daf38a0c235a5484020cf22226b1ad959cd3b91b748bd85e3a78627b84c"
Sep 10 14:43:05.017956 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.017931    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:43:05.129076 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.128995    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gv9c\" (UniqueName: \"kubernetes.io/projected/8159cea0-0584-4d7b-8532-a136531ec0ed-kube-api-access-5gv9c\") pod \"8159cea0-0584-4d7b-8532-a136531ec0ed\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") "
Sep 10 14:43:05.129076 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.129066    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-workload-certs\") pod \"8159cea0-0584-4d7b-8532-a136531ec0ed\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") "
Sep 10 14:43:05.129312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.129095    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8159cea0-0584-4d7b-8532-a136531ec0ed-istio-token\") pod \"8159cea0-0584-4d7b-8532-a136531ec0ed\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") "
Sep 10 14:43:05.129312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.129132    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-istio-data\") pod \"8159cea0-0584-4d7b-8532-a136531ec0ed\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") "
Sep 10 14:43:05.129312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.129159    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8159cea0-0584-4d7b-8532-a136531ec0ed-istiod-ca-cert\") pod \"8159cea0-0584-4d7b-8532-a136531ec0ed\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") "
Sep 10 14:43:05.129312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.129210    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-credential-socket\") pod \"8159cea0-0584-4d7b-8532-a136531ec0ed\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") "
Sep 10 14:43:05.129312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.129249    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-workload-socket\") pod \"8159cea0-0584-4d7b-8532-a136531ec0ed\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") "
Sep 10 14:43:05.129312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.129289    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8159cea0-0584-4d7b-8532-a136531ec0ed-istio-podinfo\") pod \"8159cea0-0584-4d7b-8532-a136531ec0ed\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") "
Sep 10 14:43:05.129633 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.129316    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-istio-envoy\") pod \"8159cea0-0584-4d7b-8532-a136531ec0ed\" (UID: \"8159cea0-0584-4d7b-8532-a136531ec0ed\") "
Sep 10 14:43:05.129633 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.129416    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8159cea0-0584-4d7b-8532-a136531ec0ed" (UID: "8159cea0-0584-4d7b-8532-a136531ec0ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:43:05.129633 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.129518    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8159cea0-0584-4d7b-8532-a136531ec0ed" (UID: "8159cea0-0584-4d7b-8532-a136531ec0ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:43:05.129633 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.129568    2547 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-workload-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:43:05.129633 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.129589    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8159cea0-0584-4d7b-8532-a136531ec0ed-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8159cea0-0584-4d7b-8532-a136531ec0ed" (UID: "8159cea0-0584-4d7b-8532-a136531ec0ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:43:05.129633 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.129592    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8159cea0-0584-4d7b-8532-a136531ec0ed" (UID: "8159cea0-0584-4d7b-8532-a136531ec0ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:43:05.129980 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.129657    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-istio-data" (OuterVolumeSpecName: "istio-data") pod "8159cea0-0584-4d7b-8532-a136531ec0ed" (UID: "8159cea0-0584-4d7b-8532-a136531ec0ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:43:05.131359 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.131326    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8159cea0-0584-4d7b-8532-a136531ec0ed-istio-token" (OuterVolumeSpecName: "istio-token") pod "8159cea0-0584-4d7b-8532-a136531ec0ed" (UID: "8159cea0-0584-4d7b-8532-a136531ec0ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:43:05.131359 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.131346    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8159cea0-0584-4d7b-8532-a136531ec0ed" (UID: "8159cea0-0584-4d7b-8532-a136531ec0ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:43:05.131359 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.131330    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8159cea0-0584-4d7b-8532-a136531ec0ed-kube-api-access-5gv9c" (OuterVolumeSpecName: "kube-api-access-5gv9c") pod "8159cea0-0584-4d7b-8532-a136531ec0ed" (UID: "8159cea0-0584-4d7b-8532-a136531ec0ed"). InnerVolumeSpecName "kube-api-access-5gv9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:43:05.131568 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.131483    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8159cea0-0584-4d7b-8532-a136531ec0ed-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8159cea0-0584-4d7b-8532-a136531ec0ed" (UID: "8159cea0-0584-4d7b-8532-a136531ec0ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 14:43:05.230500 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.230447    2547 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8159cea0-0584-4d7b-8532-a136531ec0ed-istio-token\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:43:05.230500 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.230492    2547 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-istio-data\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:43:05.230500 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.230504    2547 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8159cea0-0584-4d7b-8532-a136531ec0ed-istiod-ca-cert\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:43:05.230781 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.230516    2547 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-credential-socket\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:43:05.230781 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.230530    2547 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-workload-socket\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:43:05.230781 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.230541    2547 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8159cea0-0584-4d7b-8532-a136531ec0ed-istio-podinfo\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:43:05.230781 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.230553    2547 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8159cea0-0584-4d7b-8532-a136531ec0ed-istio-envoy\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:43:05.230781 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.230564    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gv9c\" (UniqueName: \"kubernetes.io/projected/8159cea0-0584-4d7b-8532-a136531ec0ed-kube-api-access-5gv9c\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:43:05.928130 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.928088    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77" event={"ID":"8159cea0-0584-4d7b-8532-a136531ec0ed","Type":"ContainerDied","Data":"afdb4945ba9a37ce281caca2ec5c633ab4e38bbd0241e324bcac5524c5137981"}
Sep 10 14:43:05.928566 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.928142    2547 scope.go:117] "RemoveContainer" containerID="32a308f5a39508b59ee763ce75e686919d574674090640ebc9858e576a351359"
Sep 10 14:43:05.928566 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.928103    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"
Sep 10 14:43:05.953264 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.953230    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"]
Sep 10 14:43:05.959761 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:05.959736    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c892fe-openshift-default-76b977cf5b-dkz77"]
Sep 10 14:43:06.421292 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:06.421263    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8159cea0-0584-4d7b-8532-a136531ec0ed" path="/var/lib/kubelet/pods/8159cea0-0584-4d7b-8532-a136531ec0ed/volumes"
Sep 10 14:43:07.149443 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.149411    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql"]
Sep 10 14:43:07.149975 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.149956    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae44b666-4a54-4e51-8d23-d67d784d91e7" containerName="main"
Sep 10 14:43:07.150074 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.149977    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae44b666-4a54-4e51-8d23-d67d784d91e7" containerName="main"
Sep 10 14:43:07.150074 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.149996    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8159cea0-0584-4d7b-8532-a136531ec0ed" containerName="istio-proxy"
Sep 10 14:43:07.150074 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.150006    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="8159cea0-0584-4d7b-8532-a136531ec0ed" containerName="istio-proxy"
Sep 10 14:43:07.150074 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.150029    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d5094f1-230b-4ca1-a6a9-e99a92105b4f" containerName="main"
Sep 10 14:43:07.150074 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.150038    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d5094f1-230b-4ca1-a6a9-e99a92105b4f" containerName="main"
Sep 10 14:43:07.150074 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.150047    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd61096f-6641-45b3-a006-5be8c77e510b" containerName="revoke-keys"
Sep 10 14:43:07.150074 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.150055    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd61096f-6641-45b3-a006-5be8c77e510b" containerName="revoke-keys"
Sep 10 14:43:07.150074 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.150067    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8159cea0-0584-4d7b-8532-a136531ec0ed" containerName="istio-proxy"
Sep 10 14:43:07.150074 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.150075    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="8159cea0-0584-4d7b-8532-a136531ec0ed" containerName="istio-proxy"
Sep 10 14:43:07.150513 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.150086    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bf48933-cbd3-44b1-beae-422b145bb9c9" containerName="main"
Sep 10 14:43:07.150513 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.150097    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bf48933-cbd3-44b1-beae-422b145bb9c9" containerName="main"
Sep 10 14:43:07.150513 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.150178    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d5094f1-230b-4ca1-a6a9-e99a92105b4f" containerName="main"
Sep 10 14:43:07.150513 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.150195    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="8159cea0-0584-4d7b-8532-a136531ec0ed" containerName="istio-proxy"
Sep 10 14:43:07.150513 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.150205    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae44b666-4a54-4e51-8d23-d67d784d91e7" containerName="main"
Sep 10 14:43:07.150513 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.150215    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd61096f-6641-45b3-a006-5be8c77e510b" containerName="revoke-keys"
Sep 10 14:43:07.150513 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.150225    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bf48933-cbd3-44b1-beae-422b145bb9c9" containerName="main"
Sep 10 14:43:07.154941 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.154921    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql"
Sep 10 14:43:07.159096 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.158944    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 14:43:07.159096 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.158955    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-db3978e3-serving-cert\""
Sep 10 14:43:07.159096 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.158963    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 14:43:07.159096 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.159018    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 14:43:07.159096 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.159049    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9r2v9\""
Sep 10 14:43:07.164194 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.164174    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql"]
Sep 10 14:43:07.248815 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.248782    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55e03cc8-84b5-40a1-a72b-b66b112ecdc8-openshift-service-ca\") pod \"maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql\" (UID: \"55e03cc8-84b5-40a1-a72b-b66b112ecdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql"
Sep 10 14:43:07.248986 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.248858    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55e03cc8-84b5-40a1-a72b-b66b112ecdc8-metrics-certs\") pod \"maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql\" (UID: \"55e03cc8-84b5-40a1-a72b-b66b112ecdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql"
Sep 10 14:43:07.248986 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.248894    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5jwk\" (UniqueName: \"kubernetes.io/projected/55e03cc8-84b5-40a1-a72b-b66b112ecdc8-kube-api-access-x5jwk\") pod \"maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql\" (UID: \"55e03cc8-84b5-40a1-a72b-b66b112ecdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql"
Sep 10 14:43:07.248986 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.248933    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55e03cc8-84b5-40a1-a72b-b66b112ecdc8-maas-api-tls\") pod \"maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql\" (UID: \"55e03cc8-84b5-40a1-a72b-b66b112ecdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql"
Sep 10 14:43:07.349671 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.349638    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55e03cc8-84b5-40a1-a72b-b66b112ecdc8-metrics-certs\") pod \"maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql\" (UID: \"55e03cc8-84b5-40a1-a72b-b66b112ecdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql"
Sep 10 14:43:07.349889 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.349683    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5jwk\" (UniqueName: \"kubernetes.io/projected/55e03cc8-84b5-40a1-a72b-b66b112ecdc8-kube-api-access-x5jwk\") pod \"maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql\" (UID: \"55e03cc8-84b5-40a1-a72b-b66b112ecdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql"
Sep 10 14:43:07.349889 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.349833    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55e03cc8-84b5-40a1-a72b-b66b112ecdc8-maas-api-tls\") pod \"maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql\" (UID: \"55e03cc8-84b5-40a1-a72b-b66b112ecdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql"
Sep 10 14:43:07.349889 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.349877    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55e03cc8-84b5-40a1-a72b-b66b112ecdc8-openshift-service-ca\") pod \"maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql\" (UID: \"55e03cc8-84b5-40a1-a72b-b66b112ecdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql"
Sep 10 14:43:07.350462 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.350436    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55e03cc8-84b5-40a1-a72b-b66b112ecdc8-openshift-service-ca\") pod \"maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql\" (UID: \"55e03cc8-84b5-40a1-a72b-b66b112ecdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql"
Sep 10 14:43:07.352195 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.352164    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55e03cc8-84b5-40a1-a72b-b66b112ecdc8-maas-api-tls\") pod \"maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql\" (UID: \"55e03cc8-84b5-40a1-a72b-b66b112ecdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql"
Sep 10 14:43:07.352286 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.352196    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55e03cc8-84b5-40a1-a72b-b66b112ecdc8-metrics-certs\") pod \"maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql\" (UID: \"55e03cc8-84b5-40a1-a72b-b66b112ecdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql"
Sep 10 14:43:07.358065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.358043    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5jwk\" (UniqueName: \"kubernetes.io/projected/55e03cc8-84b5-40a1-a72b-b66b112ecdc8-kube-api-access-x5jwk\") pod \"maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql\" (UID: \"55e03cc8-84b5-40a1-a72b-b66b112ecdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql"
Sep 10 14:43:07.468263 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.468229    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql"
Sep 10 14:43:07.774552 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.774508    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql"]
Sep 10 14:43:07.776537 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:43:07.776496    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e03cc8_84b5_40a1_a72b_b66b112ecdc8.slice/crio-163cbb85b3d911ecb8ea931a43e9cd92bdf3cfd5607a3fec658c1fe1350469d5 WatchSource:0}: Error finding container 163cbb85b3d911ecb8ea931a43e9cd92bdf3cfd5607a3fec658c1fe1350469d5: Status 404 returned error can't find the container with id 163cbb85b3d911ecb8ea931a43e9cd92bdf3cfd5607a3fec658c1fe1350469d5
Sep 10 14:43:07.778360 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.778334    2547 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:43:07.940262 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:07.940224    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql" event={"ID":"55e03cc8-84b5-40a1-a72b-b66b112ecdc8","Type":"ContainerStarted","Data":"163cbb85b3d911ecb8ea931a43e9cd92bdf3cfd5607a3fec658c1fe1350469d5"}
Sep 10 14:43:10.652931 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:10.652893    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:43:10.960789 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:10.960751    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql" event={"ID":"55e03cc8-84b5-40a1-a72b-b66b112ecdc8","Type":"ContainerStarted","Data":"8ff26fcbcea724e124527bf73ef51c5034c5b14f142522389e5dbd37282ba96b"}
Sep 10 14:43:10.960944 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:10.960869    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql"
Sep 10 14:43:10.986954 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:10.986901    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql" podStartSLOduration=1.685583024 podStartE2EDuration="3.986883653s" totalImagesPullingTime="2.301300629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:43:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:10.985084666 +0000 UTC m=+1403.026959149" watchObservedRunningTime="2026-09-10 14:43:10.986883653 +0000 UTC m=+1403.028758134"
Sep 10 14:43:16.973366 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:16.973330    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql"
Sep 10 14:43:18.016735 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:18.016689    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"]
Sep 10 14:43:18.017205 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:18.016916    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw" podUID="1e93f79d-8a6f-467d-b41c-151e9160b8b3" containerName="istio-proxy" containerID="cri-o://30d3f57c1bae4d834bb77039105161a9f3899a3349b3a9518edfd79a337f6aff" gracePeriod=30
Sep 10 14:43:18.831688 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:18.831650    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"]
Sep 10 14:43:18.832171 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:18.832151    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="8159cea0-0584-4d7b-8532-a136531ec0ed" containerName="istio-proxy"
Sep 10 14:43:18.835103 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:18.835083    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:18.837643 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:18.837622    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-85d86b-openshift-default-dockercfg-24mmt\""
Sep 10 14:43:18.850062 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:18.850037    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"]
Sep 10 14:43:18.963083 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:18.963040    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-workload-certs\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:18.963083 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:18.963086    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/641472f0-b228-4074-b0c2-fa5718d9377a-istio-token\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:18.963345 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:18.963108    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/641472f0-b228-4074-b0c2-fa5718d9377a-istiod-ca-cert\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:18.963345 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:18.963124    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-workload-socket\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:18.963345 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:18.963160    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-istio-envoy\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:18.963345 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:18.963285    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-credential-socket\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:18.963345 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:18.963328    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96b4f\" (UniqueName: \"kubernetes.io/projected/641472f0-b228-4074-b0c2-fa5718d9377a-kube-api-access-96b4f\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:18.963575 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:18.963365    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-istio-data\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:18.963575 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:18.963389    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/641472f0-b228-4074-b0c2-fa5718d9377a-istio-podinfo\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:19.064474 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:19.064435    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-credential-socket\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:19.064474 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:19.064476    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96b4f\" (UniqueName: \"kubernetes.io/projected/641472f0-b228-4074-b0c2-fa5718d9377a-kube-api-access-96b4f\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:19.065011 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:19.064506    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-istio-data\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:19.065011 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:19.064540    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/641472f0-b228-4074-b0c2-fa5718d9377a-istio-podinfo\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:19.065011 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:19.064583    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-workload-certs\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:19.065011 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:19.064605    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/641472f0-b228-4074-b0c2-fa5718d9377a-istio-token\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:19.065011 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:19.064640    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/641472f0-b228-4074-b0c2-fa5718d9377a-istiod-ca-cert\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:19.065011 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:19.064665    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-workload-socket\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:19.065011 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:19.064727    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-istio-envoy\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:19.065011 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:19.064935    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-credential-socket\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:19.065011 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:19.064975    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-istio-data\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:19.065356 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:19.065207    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-workload-socket\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:19.065356 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:19.065258    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-workload-certs\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:19.065428 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:19.065344    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/641472f0-b228-4074-b0c2-fa5718d9377a-istiod-ca-cert\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:19.066995 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:19.066966    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-istio-envoy\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:19.067233 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:19.067213    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/641472f0-b228-4074-b0c2-fa5718d9377a-istio-podinfo\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:19.072786 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:19.072766    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/641472f0-b228-4074-b0c2-fa5718d9377a-istio-token\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:19.072976 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:19.072959    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96b4f\" (UniqueName: \"kubernetes.io/projected/641472f0-b228-4074-b0c2-fa5718d9377a-kube-api-access-96b4f\") pod \"e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") " pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:19.146439 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:19.146348    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:19.424052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:19.424018    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"]
Sep 10 14:43:19.425422 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:43:19.425391    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641472f0_b228_4074_b0c2_fa5718d9377a.slice/crio-f3138dec6ecb9e37dc54356b28c94126ff90ec49d6de4d3aeedd9eaf4b222777 WatchSource:0}: Error finding container f3138dec6ecb9e37dc54356b28c94126ff90ec49d6de4d3aeedd9eaf4b222777: Status 404 returned error can't find the container with id f3138dec6ecb9e37dc54356b28c94126ff90ec49d6de4d3aeedd9eaf4b222777
Sep 10 14:43:19.427627 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:19.427593    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:43:19.427733 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:19.427670    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:43:19.427733 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:19.427724    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:43:20.014148 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:20.014102    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2" event={"ID":"641472f0-b228-4074-b0c2-fa5718d9377a","Type":"ContainerStarted","Data":"a2e9b7cb99f671a82905e800f3039588bed383bd84b2b0b868d89e6c35cd1259"}
Sep 10 14:43:20.014326 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:20.014222    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2" event={"ID":"641472f0-b228-4074-b0c2-fa5718d9377a","Type":"ContainerStarted","Data":"f3138dec6ecb9e37dc54356b28c94126ff90ec49d6de4d3aeedd9eaf4b222777"}
Sep 10 14:43:20.044277 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:20.044226    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2" podStartSLOduration=2.044211951 podStartE2EDuration="2.044211951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:20.042291545 +0000 UTC m=+1412.084166027" watchObservedRunningTime="2026-09-10 14:43:20.044211951 +0000 UTC m=+1412.086086432"
Sep 10 14:43:20.147523 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:20.147485    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:20.152074 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:20.152051    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:20.531682 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:20.531644    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:43:21.019192 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:21.019162    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:21.020126 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:21.020102    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:23.272579 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.272555    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:43:23.405121 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.405003    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istiod-ca-cert\") pod \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") "
Sep 10 14:43:23.405121 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.405064    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-workload-certs\") pod \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") "
Sep 10 14:43:23.405121 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.405120    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmjx9\" (UniqueName: \"kubernetes.io/projected/1e93f79d-8a6f-467d-b41c-151e9160b8b3-kube-api-access-dmjx9\") pod \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") "
Sep 10 14:43:23.405437 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.405143    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-workload-socket\") pod \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") "
Sep 10 14:43:23.405437 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.405168    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istio-envoy\") pod \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") "
Sep 10 14:43:23.405437 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.405186    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istio-token\") pod \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") "
Sep 10 14:43:23.405437 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.405210    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istio-podinfo\") pod \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") "
Sep 10 14:43:23.405437 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.405252    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istio-data\") pod \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") "
Sep 10 14:43:23.405437 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.405290    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-credential-socket\") pod \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\" (UID: \"1e93f79d-8a6f-467d-b41c-151e9160b8b3\") "
Sep 10 14:43:23.405437 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.405407    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1e93f79d-8a6f-467d-b41c-151e9160b8b3" (UID: "1e93f79d-8a6f-467d-b41c-151e9160b8b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:43:23.405822 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.405489    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1e93f79d-8a6f-467d-b41c-151e9160b8b3" (UID: "1e93f79d-8a6f-467d-b41c-151e9160b8b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:43:23.405822 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.405634    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1e93f79d-8a6f-467d-b41c-151e9160b8b3" (UID: "1e93f79d-8a6f-467d-b41c-151e9160b8b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:43:23.405822 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.405664    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istio-data" (OuterVolumeSpecName: "istio-data") pod "1e93f79d-8a6f-467d-b41c-151e9160b8b3" (UID: "1e93f79d-8a6f-467d-b41c-151e9160b8b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:43:23.405822 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.405729    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1e93f79d-8a6f-467d-b41c-151e9160b8b3" (UID: "1e93f79d-8a6f-467d-b41c-151e9160b8b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:43:23.406025 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.405824    2547 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-credential-socket\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:43:23.406025 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.405844    2547 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istiod-ca-cert\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:43:23.406025 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.405859    2547 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-workload-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:43:23.407622 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.407599    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1e93f79d-8a6f-467d-b41c-151e9160b8b3" (UID: "1e93f79d-8a6f-467d-b41c-151e9160b8b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:43:23.407750 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.407613    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istio-token" (OuterVolumeSpecName: "istio-token") pod "1e93f79d-8a6f-467d-b41c-151e9160b8b3" (UID: "1e93f79d-8a6f-467d-b41c-151e9160b8b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:43:23.407910 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.407881    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1e93f79d-8a6f-467d-b41c-151e9160b8b3" (UID: "1e93f79d-8a6f-467d-b41c-151e9160b8b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 14:43:23.407910 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.407886    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e93f79d-8a6f-467d-b41c-151e9160b8b3-kube-api-access-dmjx9" (OuterVolumeSpecName: "kube-api-access-dmjx9") pod "1e93f79d-8a6f-467d-b41c-151e9160b8b3" (UID: "1e93f79d-8a6f-467d-b41c-151e9160b8b3"). InnerVolumeSpecName "kube-api-access-dmjx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:43:23.506302 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.506255    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmjx9\" (UniqueName: \"kubernetes.io/projected/1e93f79d-8a6f-467d-b41c-151e9160b8b3-kube-api-access-dmjx9\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:43:23.506302 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.506298    2547 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-workload-socket\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:43:23.506302 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.506309    2547 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istio-envoy\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:43:23.506302 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.506319    2547 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istio-token\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:43:23.506581 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.506328    2547 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istio-podinfo\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:43:23.506581 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:23.506335    2547 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e93f79d-8a6f-467d-b41c-151e9160b8b3-istio-data\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:43:24.038146 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:24.038052    2547 generic.go:356] "Generic (PLEG): container finished" podID="1e93f79d-8a6f-467d-b41c-151e9160b8b3" containerID="30d3f57c1bae4d834bb77039105161a9f3899a3349b3a9518edfd79a337f6aff" exitCode=0
Sep 10 14:43:24.038146 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:24.038126    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"
Sep 10 14:43:24.038442 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:24.038175    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw" event={"ID":"1e93f79d-8a6f-467d-b41c-151e9160b8b3","Type":"ContainerDied","Data":"30d3f57c1bae4d834bb77039105161a9f3899a3349b3a9518edfd79a337f6aff"}
Sep 10 14:43:24.038442 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:24.038227    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw" event={"ID":"1e93f79d-8a6f-467d-b41c-151e9160b8b3","Type":"ContainerDied","Data":"3034b8e1794543431d8d66cd7e7bc4a07cd982bd21912b6405fca036d7ecb97d"}
Sep 10 14:43:24.038442 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:24.038250    2547 scope.go:117] "RemoveContainer" containerID="30d3f57c1bae4d834bb77039105161a9f3899a3349b3a9518edfd79a337f6aff"
Sep 10 14:43:24.048322 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:24.048298    2547 scope.go:117] "RemoveContainer" containerID="c085004e758cf25c7479b81b10a3625f26b68cba07426bd5673f849eeaa62304"
Sep 10 14:43:24.057185 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:24.057169    2547 scope.go:117] "RemoveContainer" containerID="30d3f57c1bae4d834bb77039105161a9f3899a3349b3a9518edfd79a337f6aff"
Sep 10 14:43:24.057525 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:43:24.057484    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30d3f57c1bae4d834bb77039105161a9f3899a3349b3a9518edfd79a337f6aff\": container with ID starting with 30d3f57c1bae4d834bb77039105161a9f3899a3349b3a9518edfd79a337f6aff not found: ID does not exist" containerID="30d3f57c1bae4d834bb77039105161a9f3899a3349b3a9518edfd79a337f6aff"
Sep 10 14:43:24.057645 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:24.057537    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30d3f57c1bae4d834bb77039105161a9f3899a3349b3a9518edfd79a337f6aff"} err="failed to get container status \"30d3f57c1bae4d834bb77039105161a9f3899a3349b3a9518edfd79a337f6aff\": rpc error: code = NotFound desc = could not find container \"30d3f57c1bae4d834bb77039105161a9f3899a3349b3a9518edfd79a337f6aff\": container with ID starting with 30d3f57c1bae4d834bb77039105161a9f3899a3349b3a9518edfd79a337f6aff not found: ID does not exist"
Sep 10 14:43:24.057645 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:24.057577    2547 scope.go:117] "RemoveContainer" containerID="c085004e758cf25c7479b81b10a3625f26b68cba07426bd5673f849eeaa62304"
Sep 10 14:43:24.057952 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:43:24.057929    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c085004e758cf25c7479b81b10a3625f26b68cba07426bd5673f849eeaa62304\": container with ID starting with c085004e758cf25c7479b81b10a3625f26b68cba07426bd5673f849eeaa62304 not found: ID does not exist" containerID="c085004e758cf25c7479b81b10a3625f26b68cba07426bd5673f849eeaa62304"
Sep 10 14:43:24.058075 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:24.057958    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c085004e758cf25c7479b81b10a3625f26b68cba07426bd5673f849eeaa62304"} err="failed to get container status \"c085004e758cf25c7479b81b10a3625f26b68cba07426bd5673f849eeaa62304\": rpc error: code = NotFound desc = could not find container \"c085004e758cf25c7479b81b10a3625f26b68cba07426bd5673f849eeaa62304\": container with ID starting with c085004e758cf25c7479b81b10a3625f26b68cba07426bd5673f849eeaa62304 not found: ID does not exist"
Sep 10 14:43:24.067084 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:24.067053    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"]
Sep 10 14:43:24.067200 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:24.067111    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2365fe-openshift-default-657649bbb-h6dvw"]
Sep 10 14:43:24.422013 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:24.421981    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e93f79d-8a6f-467d-b41c-151e9160b8b3" path="/var/lib/kubelet/pods/1e93f79d-8a6f-467d-b41c-151e9160b8b3/volumes"
Sep 10 14:43:30.730509 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:30.730473    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:43:36.348934 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:36.348887    2547 patch_prober.go:28] interesting pod/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": read tcp 10.132.0.2:58414->10.132.0.63:15021: read: connection reset by peer" start-of-body=
Sep 10 14:43:36.349388 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:36.348986    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2" podUID="641472f0-b228-4074-b0c2-fa5718d9377a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": read tcp 10.132.0.2:58414->10.132.0.63:15021: read: connection reset by peer"
Sep 10 14:43:37.119279 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:37.119243    2547 generic.go:356] "Generic (PLEG): container finished" podID="641472f0-b228-4074-b0c2-fa5718d9377a" containerID="a2e9b7cb99f671a82905e800f3039588bed383bd84b2b0b868d89e6c35cd1259" exitCode=0
Sep 10 14:43:37.119450 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:37.119317    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2" event={"ID":"641472f0-b228-4074-b0c2-fa5718d9377a","Type":"ContainerDied","Data":"a2e9b7cb99f671a82905e800f3039588bed383bd84b2b0b868d89e6c35cd1259"}
Sep 10 14:43:37.119785 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:37.119760    2547 scope.go:117] "RemoveContainer" containerID="a2e9b7cb99f671a82905e800f3039588bed383bd84b2b0b868d89e6c35cd1259"
Sep 10 14:43:37.120778 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:37.120749    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:43:37.120900 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:37.120831    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:43:37.120900 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:37.120879    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:43:38.126487 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:38.126450    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2" event={"ID":"641472f0-b228-4074-b0c2-fa5718d9377a","Type":"ContainerStarted","Data":"719eaa189c7b36fd60fdf29dcd9d4aa8cd6ea958d20ee08db73534d1369b9ecc"}
Sep 10 14:43:38.147319 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:38.147287    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:38.147497 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:38.147477    2547 patch_prober.go:28] interesting pod/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 10 14:43:38.147552 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:38.147529    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2" podUID="641472f0-b228-4074-b0c2-fa5718d9377a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 10 14:43:39.147571 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:39.147527    2547 patch_prober.go:28] interesting pod/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 10 14:43:39.148082 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:39.147587    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2" podUID="641472f0-b228-4074-b0c2-fa5718d9377a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 10 14:43:40.147616 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:40.147581    2547 patch_prober.go:28] interesting pod/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 10 14:43:40.148141 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:40.147634    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2" podUID="641472f0-b228-4074-b0c2-fa5718d9377a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 10 14:43:41.147088 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:41.147005    2547 patch_prober.go:28] interesting pod/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 10 14:43:41.147257 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:41.147082    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2" podUID="641472f0-b228-4074-b0c2-fa5718d9377a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 10 14:43:42.147113 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:42.147075    2547 patch_prober.go:28] interesting pod/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 10 14:43:42.147514 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:42.147139    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2" podUID="641472f0-b228-4074-b0c2-fa5718d9377a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 10 14:43:43.150905 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:43.150871    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:43.151367 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:43.151239    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:43.151871 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:43.151852    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:43:47.522594 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.522561    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"]
Sep 10 14:43:47.522967 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.522930    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e93f79d-8a6f-467d-b41c-151e9160b8b3" containerName="istio-proxy"
Sep 10 14:43:47.522967 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.522942    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e93f79d-8a6f-467d-b41c-151e9160b8b3" containerName="istio-proxy"
Sep 10 14:43:47.523044 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.523012    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e93f79d-8a6f-467d-b41c-151e9160b8b3" containerName="istio-proxy"
Sep 10 14:43:47.523044 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.523023    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e93f79d-8a6f-467d-b41c-151e9160b8b3" containerName="istio-proxy"
Sep 10 14:43:47.530827 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.530806    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.533216 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.533199    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-3477ac-openshift-default-dockercfg-c7nhp\""
Sep 10 14:43:47.542276 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.542253    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"]
Sep 10 14:43:47.626331 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.626292    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istiod-ca-cert\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.626505 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.626364    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-workload-socket\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.626505 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.626445    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-credential-socket\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.626607 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.626532    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-workload-certs\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.626607 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.626577    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istio-data\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.626719 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.626626    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istio-envoy\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.626719 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.626655    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istio-podinfo\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.626819 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.626726    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istio-token\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.626819 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.626777    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99wwm\" (UniqueName: \"kubernetes.io/projected/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-kube-api-access-99wwm\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.727424 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.727380    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-credential-socket\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.727424 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.727430    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-workload-certs\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.727670 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.727448    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istio-data\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.727670 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.727476    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istio-envoy\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.727670 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.727497    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istio-podinfo\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.727670 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.727516    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istio-token\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.727670 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.727549    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99wwm\" (UniqueName: \"kubernetes.io/projected/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-kube-api-access-99wwm\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.727670 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.727580    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istiod-ca-cert\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.727670 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.727635    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-workload-socket\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.728073 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.727894    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-credential-socket\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.728073 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.727916    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istio-data\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.728073 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.727989    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-workload-socket\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.728244 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.728225    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-workload-certs\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.728471 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.728449    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istiod-ca-cert\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.729884 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.729867    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istio-envoy\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.730027 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.730009    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istio-podinfo\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.735304 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.735279    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istio-token\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.735540 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.735522    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99wwm\" (UniqueName: \"kubernetes.io/projected/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-kube-api-access-99wwm\") pod \"e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") " pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:47.841348 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:47.841247    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:48.124740 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:48.124712    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"]
Sep 10 14:43:48.126667 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:43:48.126634    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc1b82f_dc7c_4eac_bd63_c2c73bfc931b.slice/crio-dd5891f3258d21c091a1dc13421d954edd4fee3e09528d48a059f1ac4834e83d WatchSource:0}: Error finding container dd5891f3258d21c091a1dc13421d954edd4fee3e09528d48a059f1ac4834e83d: Status 404 returned error can't find the container with id dd5891f3258d21c091a1dc13421d954edd4fee3e09528d48a059f1ac4834e83d
Sep 10 14:43:48.129235 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:48.129197    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:43:48.129321 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:48.129284    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:43:48.129368 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:48.129339    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:43:48.198065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:48.198018    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq" event={"ID":"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b","Type":"ContainerStarted","Data":"a30b0c83bbd04427170aee7c6c498ec94cd839d897c2d7fc55c1b0e94b9f55aa"}
Sep 10 14:43:48.198178 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:48.198088    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq" event={"ID":"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b","Type":"ContainerStarted","Data":"dd5891f3258d21c091a1dc13421d954edd4fee3e09528d48a059f1ac4834e83d"}
Sep 10 14:43:48.220091 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:48.220002    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq" podStartSLOduration=1.219980171 podStartE2EDuration="1.219980171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:48.219857206 +0000 UTC m=+1440.261731688" watchObservedRunningTime="2026-09-10 14:43:48.219980171 +0000 UTC m=+1440.261854650"
Sep 10 14:43:48.842048 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:48.841968    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:49.302894 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:49.302858    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql"]
Sep 10 14:43:49.303124 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:49.303101    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql" podUID="55e03cc8-84b5-40a1-a72b-b66b112ecdc8" containerName="maas-api" containerID="cri-o://8ff26fcbcea724e124527bf73ef51c5034c5b14f142522389e5dbd37282ba96b" gracePeriod=30
Sep 10 14:43:49.551000 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:49.550973    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql"
Sep 10 14:43:49.644575 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:49.644490    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55e03cc8-84b5-40a1-a72b-b66b112ecdc8-metrics-certs\") pod \"55e03cc8-84b5-40a1-a72b-b66b112ecdc8\" (UID: \"55e03cc8-84b5-40a1-a72b-b66b112ecdc8\") "
Sep 10 14:43:49.644575 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:49.644546    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55e03cc8-84b5-40a1-a72b-b66b112ecdc8-maas-api-tls\") pod \"55e03cc8-84b5-40a1-a72b-b66b112ecdc8\" (UID: \"55e03cc8-84b5-40a1-a72b-b66b112ecdc8\") "
Sep 10 14:43:49.644796 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:49.644576    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5jwk\" (UniqueName: \"kubernetes.io/projected/55e03cc8-84b5-40a1-a72b-b66b112ecdc8-kube-api-access-x5jwk\") pod \"55e03cc8-84b5-40a1-a72b-b66b112ecdc8\" (UID: \"55e03cc8-84b5-40a1-a72b-b66b112ecdc8\") "
Sep 10 14:43:49.644796 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:49.644651    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55e03cc8-84b5-40a1-a72b-b66b112ecdc8-openshift-service-ca\") pod \"55e03cc8-84b5-40a1-a72b-b66b112ecdc8\" (UID: \"55e03cc8-84b5-40a1-a72b-b66b112ecdc8\") "
Sep 10 14:43:49.645077 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:49.645052    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e03cc8-84b5-40a1-a72b-b66b112ecdc8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "55e03cc8-84b5-40a1-a72b-b66b112ecdc8" (UID: "55e03cc8-84b5-40a1-a72b-b66b112ecdc8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:43:49.646852 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:49.646826    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55e03cc8-84b5-40a1-a72b-b66b112ecdc8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "55e03cc8-84b5-40a1-a72b-b66b112ecdc8" (UID: "55e03cc8-84b5-40a1-a72b-b66b112ecdc8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:43:49.646936 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:49.646844    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e03cc8-84b5-40a1-a72b-b66b112ecdc8-kube-api-access-x5jwk" (OuterVolumeSpecName: "kube-api-access-x5jwk") pod "55e03cc8-84b5-40a1-a72b-b66b112ecdc8" (UID: "55e03cc8-84b5-40a1-a72b-b66b112ecdc8"). InnerVolumeSpecName "kube-api-access-x5jwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:43:49.646936 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:49.646929    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55e03cc8-84b5-40a1-a72b-b66b112ecdc8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "55e03cc8-84b5-40a1-a72b-b66b112ecdc8" (UID: "55e03cc8-84b5-40a1-a72b-b66b112ecdc8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:43:49.745828 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:49.745790    2547 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55e03cc8-84b5-40a1-a72b-b66b112ecdc8-metrics-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:43:49.745828 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:49.745819    2547 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55e03cc8-84b5-40a1-a72b-b66b112ecdc8-maas-api-tls\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:43:49.745828 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:49.745831    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5jwk\" (UniqueName: \"kubernetes.io/projected/55e03cc8-84b5-40a1-a72b-b66b112ecdc8-kube-api-access-x5jwk\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:43:49.746050 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:49.745842    2547 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55e03cc8-84b5-40a1-a72b-b66b112ecdc8-openshift-service-ca\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:43:49.846988 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:49.846953    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:50.210714 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:50.210640    2547 generic.go:356] "Generic (PLEG): container finished" podID="55e03cc8-84b5-40a1-a72b-b66b112ecdc8" containerID="8ff26fcbcea724e124527bf73ef51c5034c5b14f142522389e5dbd37282ba96b" exitCode=0
Sep 10 14:43:50.210991 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:50.210718    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql"
Sep 10 14:43:50.210991 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:50.210810    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql" event={"ID":"55e03cc8-84b5-40a1-a72b-b66b112ecdc8","Type":"ContainerDied","Data":"8ff26fcbcea724e124527bf73ef51c5034c5b14f142522389e5dbd37282ba96b"}
Sep 10 14:43:50.210991 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:50.210902    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql" event={"ID":"55e03cc8-84b5-40a1-a72b-b66b112ecdc8","Type":"ContainerDied","Data":"163cbb85b3d911ecb8ea931a43e9cd92bdf3cfd5607a3fec658c1fe1350469d5"}
Sep 10 14:43:50.210991 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:50.210942    2547 scope.go:117] "RemoveContainer" containerID="8ff26fcbcea724e124527bf73ef51c5034c5b14f142522389e5dbd37282ba96b"
Sep 10 14:43:50.211316 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:50.211269    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:50.212582 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:50.212557    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:43:50.221477 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:50.221460    2547 scope.go:117] "RemoveContainer" containerID="8ff26fcbcea724e124527bf73ef51c5034c5b14f142522389e5dbd37282ba96b"
Sep 10 14:43:50.221765 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:43:50.221745    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ff26fcbcea724e124527bf73ef51c5034c5b14f142522389e5dbd37282ba96b\": container with ID starting with 8ff26fcbcea724e124527bf73ef51c5034c5b14f142522389e5dbd37282ba96b not found: ID does not exist" containerID="8ff26fcbcea724e124527bf73ef51c5034c5b14f142522389e5dbd37282ba96b"
Sep 10 14:43:50.221837 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:50.221771    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ff26fcbcea724e124527bf73ef51c5034c5b14f142522389e5dbd37282ba96b"} err="failed to get container status \"8ff26fcbcea724e124527bf73ef51c5034c5b14f142522389e5dbd37282ba96b\": rpc error: code = NotFound desc = could not find container \"8ff26fcbcea724e124527bf73ef51c5034c5b14f142522389e5dbd37282ba96b\": container with ID starting with 8ff26fcbcea724e124527bf73ef51c5034c5b14f142522389e5dbd37282ba96b not found: ID does not exist"
Sep 10 14:43:50.266629 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:50.266594    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql"]
Sep 10 14:43:50.276922 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:50.276892    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-db3978e3-6d67f7d98b-b4zql"]
Sep 10 14:43:50.421597 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:50.421563    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55e03cc8-84b5-40a1-a72b-b66b112ecdc8" path="/var/lib/kubelet/pods/55e03cc8-84b5-40a1-a72b-b66b112ecdc8/volumes"
Sep 10 14:43:51.239146 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.239110    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:43:51.332730 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.332682    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt"]
Sep 10 14:43:51.333245 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.333228    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e93f79d-8a6f-467d-b41c-151e9160b8b3" containerName="istio-proxy"
Sep 10 14:43:51.333287 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.333248    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e93f79d-8a6f-467d-b41c-151e9160b8b3" containerName="istio-proxy"
Sep 10 14:43:51.333323 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.333286    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55e03cc8-84b5-40a1-a72b-b66b112ecdc8" containerName="maas-api"
Sep 10 14:43:51.333323 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.333296    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="55e03cc8-84b5-40a1-a72b-b66b112ecdc8" containerName="maas-api"
Sep 10 14:43:51.333399 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.333387    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="55e03cc8-84b5-40a1-a72b-b66b112ecdc8" containerName="maas-api"
Sep 10 14:43:51.338069 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.338047    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt"
Sep 10 14:43:51.340612 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.340589    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-3477ac-serving-cert\""
Sep 10 14:43:51.340851 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.340835    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 14:43:51.342091 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.342071    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 14:43:51.342091 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.342082    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 14:43:51.342254 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.342135    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9r2v9\""
Sep 10 14:43:51.346796 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.346546    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt"]
Sep 10 14:43:51.459377 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.459345    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt\" (UID: \"a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt"
Sep 10 14:43:51.459562 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.459396    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt\" (UID: \"a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt"
Sep 10 14:43:51.459562 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.459419    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt\" (UID: \"a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt"
Sep 10 14:43:51.459562 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.459433    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtdrn\" (UniqueName: \"kubernetes.io/projected/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd-kube-api-access-mtdrn\") pod \"maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt\" (UID: \"a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt"
Sep 10 14:43:51.560812 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.560691    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt\" (UID: \"a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt"
Sep 10 14:43:51.560812 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.560772    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt\" (UID: \"a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt"
Sep 10 14:43:51.560812 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.560793    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt\" (UID: \"a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt"
Sep 10 14:43:51.560812 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.560808    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtdrn\" (UniqueName: \"kubernetes.io/projected/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd-kube-api-access-mtdrn\") pod \"maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt\" (UID: \"a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt"
Sep 10 14:43:51.561457 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.561432    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt\" (UID: \"a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt"
Sep 10 14:43:51.563421 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.563397    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt\" (UID: \"a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt"
Sep 10 14:43:51.563483 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.563462    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt\" (UID: \"a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt"
Sep 10 14:43:51.568751 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.568728    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtdrn\" (UniqueName: \"kubernetes.io/projected/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd-kube-api-access-mtdrn\") pod \"maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt\" (UID: \"a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt"
Sep 10 14:43:51.651902 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.651867    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt"
Sep 10 14:43:51.931123 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:51.931097    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt"]
Sep 10 14:43:51.934503 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:43:51.934459    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e413b9_fa27_4a86_9c4b_1fa6fd56eabd.slice/crio-5ede1d78541994348da1e9e0e209f98fe4873bbafe2883c1a9eec03176f9983e WatchSource:0}: Error finding container 5ede1d78541994348da1e9e0e209f98fe4873bbafe2883c1a9eec03176f9983e: Status 404 returned error can't find the container with id 5ede1d78541994348da1e9e0e209f98fe4873bbafe2883c1a9eec03176f9983e
Sep 10 14:43:52.224490 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:52.224401    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt" event={"ID":"a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd","Type":"ContainerStarted","Data":"5ede1d78541994348da1e9e0e209f98fe4873bbafe2883c1a9eec03176f9983e"}
Sep 10 14:43:53.232140 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:53.232103    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt" event={"ID":"a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd","Type":"ContainerStarted","Data":"721d25fae898b0aa42548f819f0add1e10ff99b36d00b7ff141ff13d24a6189b"}
Sep 10 14:43:53.232599 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:53.232304    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt"
Sep 10 14:43:53.254745 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:53.252834    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt" podStartSLOduration=1.8726166370000001 podStartE2EDuration="2.252813753s" totalImagesPullingTime="380.197116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:43:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:53.248009334 +0000 UTC m=+1445.289883816" watchObservedRunningTime="2026-09-10 14:43:53.252813753 +0000 UTC m=+1445.294688234"
Sep 10 14:43:59.243153 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:59.243119    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt"
Sep 10 14:43:59.823619 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:43:59.823578    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:44:06.208973 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:06.208927    2547 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": read tcp 10.132.0.2:54500->10.132.0.64:15021: read: connection reset by peer" start-of-body=
Sep 10 14:44:06.209351 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:06.209014    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq" podUID="2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": read tcp 10.132.0.2:54500->10.132.0.64:15021: read: connection reset by peer"
Sep 10 14:44:06.316639 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:06.316606    2547 generic.go:356] "Generic (PLEG): container finished" podID="2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" containerID="a30b0c83bbd04427170aee7c6c498ec94cd839d897c2d7fc55c1b0e94b9f55aa" exitCode=0
Sep 10 14:44:06.316870 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:06.316685    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq" event={"ID":"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b","Type":"ContainerDied","Data":"a30b0c83bbd04427170aee7c6c498ec94cd839d897c2d7fc55c1b0e94b9f55aa"}
Sep 10 14:44:06.317179 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:06.317162    2547 scope.go:117] "RemoveContainer" containerID="a30b0c83bbd04427170aee7c6c498ec94cd839d897c2d7fc55c1b0e94b9f55aa"
Sep 10 14:44:06.318187 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:06.318158    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:44:06.318267 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:06.318220    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:44:06.318267 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:06.318250    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:44:07.323787 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:07.323751    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq" event={"ID":"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b","Type":"ContainerStarted","Data":"c78a6d1f80c91afb867db601dbd631909120123860e5d872726830cd7f814545"}
Sep 10 14:44:07.841907 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:07.841861    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:44:07.842160 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:07.842136    2547 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 10 14:44:07.842208 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:07.842192    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq" podUID="2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 10 14:44:08.841893 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:08.841858    2547 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 10 14:44:08.842280 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:08.841913    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq" podUID="2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 10 14:44:09.842602 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:09.842563    2547 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 10 14:44:09.843011 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:09.842620    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq" podUID="2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 10 14:44:10.841981 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:10.841947    2547 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 10 14:44:10.842169 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:10.842000    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq" podUID="2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 10 14:44:11.842676 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:11.842637    2547 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 10 14:44:11.843100 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:11.842724    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq" podUID="2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 10 14:44:12.845154 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:12.845121    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:44:12.845614 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:12.845458    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:44:12.846214 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:12.846191    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:44:14.672181 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:14.672149    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"]
Sep 10 14:44:14.674680 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:14.674660    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"
Sep 10 14:44:14.676928 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:14.676905    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-bc9af3-serving-cert\""
Sep 10 14:44:14.685665 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:14.685646    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"]
Sep 10 14:44:14.789093 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:14.789056    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9\" (UID: \"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"
Sep 10 14:44:14.789093 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:14.789093    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9\" (UID: \"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"
Sep 10 14:44:14.789356 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:14.789118    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9\" (UID: \"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"
Sep 10 14:44:14.789356 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:14.789231    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snf6w\" (UniqueName: \"kubernetes.io/projected/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-kube-api-access-snf6w\") pod \"maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9\" (UID: \"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"
Sep 10 14:44:14.890609 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:14.890576    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9\" (UID: \"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"
Sep 10 14:44:14.890609 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:14.890610    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9\" (UID: \"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"
Sep 10 14:44:14.890888 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:14.890632    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9\" (UID: \"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"
Sep 10 14:44:14.890888 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:14.890679    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snf6w\" (UniqueName: \"kubernetes.io/projected/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-kube-api-access-snf6w\") pod \"maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9\" (UID: \"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"
Sep 10 14:44:14.890888 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:44:14.890739    2547 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-serving-cert: secret "maas-api-e2e-shared-a-w2-bc9af3-serving-cert" not found
Sep 10 14:44:14.890888 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:44:14.890811    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-maas-api-tls podName:e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6 nodeName:}" failed. No retries permitted until 2026-09-10 14:44:15.390794485 +0000 UTC m=+1467.432668949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-maas-api-tls") pod "maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9" (UID: "e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6") : secret "maas-api-e2e-shared-a-w2-bc9af3-serving-cert" not found
Sep 10 14:44:14.891368 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:14.891345    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9\" (UID: \"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"
Sep 10 14:44:14.893028 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:14.893006    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9\" (UID: \"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"
Sep 10 14:44:14.898891 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:14.898870    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snf6w\" (UniqueName: \"kubernetes.io/projected/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-kube-api-access-snf6w\") pod \"maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9\" (UID: \"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"
Sep 10 14:44:15.396142 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:15.396094    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9\" (UID: \"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"
Sep 10 14:44:15.398462 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:15.398435    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9\" (UID: \"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"
Sep 10 14:44:15.586318 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:15.586278    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"
Sep 10 14:44:15.869618 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:15.869586    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"]
Sep 10 14:44:15.870904 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:44:15.870880    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6188f2c_2c5e_40a5_b9fe_9aaaf4159ec6.slice/crio-f2b3a62ec7227b43191c127378cd5360eaf9fd40bdbaad762e852b2e47ef4e86 WatchSource:0}: Error finding container f2b3a62ec7227b43191c127378cd5360eaf9fd40bdbaad762e852b2e47ef4e86: Status 404 returned error can't find the container with id f2b3a62ec7227b43191c127378cd5360eaf9fd40bdbaad762e852b2e47ef4e86
Sep 10 14:44:16.377130 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:16.377096    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9" event={"ID":"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6","Type":"ContainerStarted","Data":"880b46dfff67af6b052204c3c55c7c218c7e71765f5e98057857b53bdec12ec8"}
Sep 10 14:44:16.377303 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:16.377158    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"
Sep 10 14:44:16.377303 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:16.377182    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9" event={"ID":"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6","Type":"ContainerStarted","Data":"f2b3a62ec7227b43191c127378cd5360eaf9fd40bdbaad762e852b2e47ef4e86"}
Sep 10 14:44:16.395389 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:16.395340    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9" podStartSLOduration=2.0618800840000002 podStartE2EDuration="2.395326116s" totalImagesPullingTime="333.446032ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:44:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:44:16.393410067 +0000 UTC m=+1468.435284548" watchObservedRunningTime="2026-09-10 14:44:16.395326116 +0000 UTC m=+1468.437200598"
Sep 10 14:44:19.430110 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:19.430071    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:44:22.387855 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:22.387824    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"
Sep 10 14:44:34.922146 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:34.922110    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"]
Sep 10 14:44:34.925865 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:34.925846    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:44:34.929805 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:34.929771    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-85d86b\"/\"openshift-service-ca.crt\""
Sep 10 14:44:34.930019 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:34.929906    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-85d86b\"/\"kube-root-ca.crt\""
Sep 10 14:44:34.930102 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:34.930012    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-85d86b\"/\"default-dockercfg-8xb57\""
Sep 10 14:44:34.931011 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:34.930864    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-85d86b\"/\"auth-test-model-85d86b-kserve-self-signed-certs\""
Sep 10 14:44:34.934105 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:34.934080    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"]
Sep 10 14:44:34.974496 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:34.974466    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8283c96b-6243-4e5f-9116-e64bbe65b4c9-home\") pod \"auth-test-model-85d86b-kserve-674d4c7d55-sjdn5\" (UID: \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:44:34.974644 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:34.974516    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8283c96b-6243-4e5f-9116-e64bbe65b4c9-tls-certs\") pod \"auth-test-model-85d86b-kserve-674d4c7d55-sjdn5\" (UID: \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:44:34.974644 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:34.974595    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8283c96b-6243-4e5f-9116-e64bbe65b4c9-dshm\") pod \"auth-test-model-85d86b-kserve-674d4c7d55-sjdn5\" (UID: \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:44:34.974743 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:34.974651    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8283c96b-6243-4e5f-9116-e64bbe65b4c9-tmp-dir\") pod \"auth-test-model-85d86b-kserve-674d4c7d55-sjdn5\" (UID: \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:44:34.974787 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:34.974743    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8283c96b-6243-4e5f-9116-e64bbe65b4c9-model-cache\") pod \"auth-test-model-85d86b-kserve-674d4c7d55-sjdn5\" (UID: \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:44:34.974787 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:34.974778    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2dks\" (UniqueName: \"kubernetes.io/projected/8283c96b-6243-4e5f-9116-e64bbe65b4c9-kube-api-access-n2dks\") pod \"auth-test-model-85d86b-kserve-674d4c7d55-sjdn5\" (UID: \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:44:35.076166 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:35.076129    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8283c96b-6243-4e5f-9116-e64bbe65b4c9-dshm\") pod \"auth-test-model-85d86b-kserve-674d4c7d55-sjdn5\" (UID: \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:44:35.076358 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:35.076189    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8283c96b-6243-4e5f-9116-e64bbe65b4c9-tmp-dir\") pod \"auth-test-model-85d86b-kserve-674d4c7d55-sjdn5\" (UID: \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:44:35.076358 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:35.076228    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8283c96b-6243-4e5f-9116-e64bbe65b4c9-model-cache\") pod \"auth-test-model-85d86b-kserve-674d4c7d55-sjdn5\" (UID: \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:44:35.076358 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:35.076248    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2dks\" (UniqueName: \"kubernetes.io/projected/8283c96b-6243-4e5f-9116-e64bbe65b4c9-kube-api-access-n2dks\") pod \"auth-test-model-85d86b-kserve-674d4c7d55-sjdn5\" (UID: \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:44:35.076358 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:35.076279    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8283c96b-6243-4e5f-9116-e64bbe65b4c9-home\") pod \"auth-test-model-85d86b-kserve-674d4c7d55-sjdn5\" (UID: \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:44:35.076358 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:35.076321    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8283c96b-6243-4e5f-9116-e64bbe65b4c9-tls-certs\") pod \"auth-test-model-85d86b-kserve-674d4c7d55-sjdn5\" (UID: \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:44:35.076655 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:35.076630    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8283c96b-6243-4e5f-9116-e64bbe65b4c9-tmp-dir\") pod \"auth-test-model-85d86b-kserve-674d4c7d55-sjdn5\" (UID: \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:44:35.076733 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:35.076646    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8283c96b-6243-4e5f-9116-e64bbe65b4c9-model-cache\") pod \"auth-test-model-85d86b-kserve-674d4c7d55-sjdn5\" (UID: \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:44:35.076733 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:35.076675    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8283c96b-6243-4e5f-9116-e64bbe65b4c9-home\") pod \"auth-test-model-85d86b-kserve-674d4c7d55-sjdn5\" (UID: \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:44:35.078451 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:35.078429    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8283c96b-6243-4e5f-9116-e64bbe65b4c9-dshm\") pod \"auth-test-model-85d86b-kserve-674d4c7d55-sjdn5\" (UID: \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:44:35.078724 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:35.078688    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8283c96b-6243-4e5f-9116-e64bbe65b4c9-tls-certs\") pod \"auth-test-model-85d86b-kserve-674d4c7d55-sjdn5\" (UID: \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:44:35.084923 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:35.084901    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2dks\" (UniqueName: \"kubernetes.io/projected/8283c96b-6243-4e5f-9116-e64bbe65b4c9-kube-api-access-n2dks\") pod \"auth-test-model-85d86b-kserve-674d4c7d55-sjdn5\" (UID: \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:44:35.236041 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:35.235948    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:44:35.507973 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:35.507944    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"]
Sep 10 14:44:35.510749 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:44:35.510718    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8283c96b_6243_4e5f_9116_e64bbe65b4c9.slice/crio-96269ce8f6510fa9ca13836619d134c22865c20435490ffb05180b453560469b WatchSource:0}: Error finding container 96269ce8f6510fa9ca13836619d134c22865c20435490ffb05180b453560469b: Status 404 returned error can't find the container with id 96269ce8f6510fa9ca13836619d134c22865c20435490ffb05180b453560469b
Sep 10 14:44:36.492334 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:36.492292    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5" event={"ID":"8283c96b-6243-4e5f-9116-e64bbe65b4c9","Type":"ContainerStarted","Data":"ca98e534c8f32589d396a270436678cdfdc6d5551b0b2705e14fa04d69d72cbf"}
Sep 10 14:44:36.492721 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:36.492347    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5" event={"ID":"8283c96b-6243-4e5f-9116-e64bbe65b4c9","Type":"ContainerStarted","Data":"96269ce8f6510fa9ca13836619d134c22865c20435490ffb05180b453560469b"}
Sep 10 14:44:38.427508 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:38.427449    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5" podStartSLOduration=4.427433475 podStartE2EDuration="4.427433475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:44:36.5074861 +0000 UTC m=+1488.549360580" watchObservedRunningTime="2026-09-10 14:44:38.427433475 +0000 UTC m=+1490.469307957"
Sep 10 14:44:38.428373 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:38.428351    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:44:45.236779 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:45.236734    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:44:45.236779 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:45.236791    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:44:45.249449 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:45.249426    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:44:45.552595 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:45.552509    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:44:48.435573 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:48.435544    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mg9d6_359d40e9-f428-4cc3-b4e4-991c081d664a/console-operator/1.log"
Sep 10 14:44:48.439850 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:48.439825    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mg9d6_359d40e9-f428-4cc3-b4e4-991c081d664a/console-operator/1.log"
Sep 10 14:44:49.319973 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:44:49.319931    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:45:00.148790 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:00.148748    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc9af3-29817525-dtfb6"]
Sep 10 14:45:00.152636 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:00.152616    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3477ac-29817525-c6k7w"]
Sep 10 14:45:00.152778 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:00.152760    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc9af3-29817525-dtfb6"
Sep 10 14:45:00.155567 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:00.155546    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xnxsg\""
Sep 10 14:45:00.155951 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:00.155932    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3477ac-29817525-c6k7w"
Sep 10 14:45:00.164002 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:00.163975    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc9af3-29817525-dtfb6"]
Sep 10 14:45:00.166389 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:00.166365    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc9af3-29817525-dtfb6"
Sep 10 14:45:00.167925 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:00.167900    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3477ac-29817525-c6k7w"]
Sep 10 14:45:00.176667 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:00.176645    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3477ac-29817525-c6k7w"
Sep 10 14:45:00.446673 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:00.446474    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc9af3-29817525-dtfb6"]
Sep 10 14:45:00.449558 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:45:00.449530    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c1263a1_00d5_4f5d_94eb_b7cdfbc471d9.slice/crio-5abf0de9dfdbc6dd94416f052b895245c02c2755b2f225748eca1433cbb623e5 WatchSource:0}: Error finding container 5abf0de9dfdbc6dd94416f052b895245c02c2755b2f225748eca1433cbb623e5: Status 404 returned error can't find the container with id 5abf0de9dfdbc6dd94416f052b895245c02c2755b2f225748eca1433cbb623e5
Sep 10 14:45:00.458596 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:00.458557    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3477ac-29817525-c6k7w"]
Sep 10 14:45:00.463219 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:45:00.463187    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc199420_d1ca_419e_93af_bfcc8c0956ac.slice/crio-27c8b33b87992a98a51ef766b494f980b7e3a8179bafa43b8dddfd71f28ccbe3 WatchSource:0}: Error finding container 27c8b33b87992a98a51ef766b494f980b7e3a8179bafa43b8dddfd71f28ccbe3: Status 404 returned error can't find the container with id 27c8b33b87992a98a51ef766b494f980b7e3a8179bafa43b8dddfd71f28ccbe3
Sep 10 14:45:00.634636 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:00.634592    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3477ac-29817525-c6k7w" event={"ID":"dc199420-d1ca-419e-93af-bfcc8c0956ac","Type":"ContainerStarted","Data":"27c8b33b87992a98a51ef766b494f980b7e3a8179bafa43b8dddfd71f28ccbe3"}
Sep 10 14:45:00.635916 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:00.635889    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc9af3-29817525-dtfb6" event={"ID":"1c1263a1-00d5-4f5d-94eb-b7cdfbc471d9","Type":"ContainerStarted","Data":"5abf0de9dfdbc6dd94416f052b895245c02c2755b2f225748eca1433cbb623e5"}
Sep 10 14:45:01.643530 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:01.643440    2547 generic.go:356] "Generic (PLEG): container finished" podID="1c1263a1-00d5-4f5d-94eb-b7cdfbc471d9" containerID="dfe99d48f2c944d626222f4bb947fee02d8e41bd75c0f915071b55b9cdcc6ef1" exitCode=0
Sep 10 14:45:01.643951 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:01.643539    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc9af3-29817525-dtfb6" event={"ID":"1c1263a1-00d5-4f5d-94eb-b7cdfbc471d9","Type":"ContainerDied","Data":"dfe99d48f2c944d626222f4bb947fee02d8e41bd75c0f915071b55b9cdcc6ef1"}
Sep 10 14:45:01.645167 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:01.645145    2547 generic.go:356] "Generic (PLEG): container finished" podID="dc199420-d1ca-419e-93af-bfcc8c0956ac" containerID="27724a40f64675ddf4e3c6437f20816f68c793af0db005ad11551fb4a02edfbc" exitCode=0
Sep 10 14:45:01.645283 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:01.645204    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3477ac-29817525-c6k7w" event={"ID":"dc199420-d1ca-419e-93af-bfcc8c0956ac","Type":"ContainerDied","Data":"27724a40f64675ddf4e3c6437f20816f68c793af0db005ad11551fb4a02edfbc"}
Sep 10 14:45:02.791038 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:02.791010    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc9af3-29817525-dtfb6"
Sep 10 14:45:02.818490 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:02.818456    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3477ac-29817525-c6k7w"
Sep 10 14:45:03.657682 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:03.657650    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3477ac-29817525-c6k7w"
Sep 10 14:45:03.657882 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:03.657683    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3477ac-29817525-c6k7w" event={"ID":"dc199420-d1ca-419e-93af-bfcc8c0956ac","Type":"ContainerDied","Data":"27c8b33b87992a98a51ef766b494f980b7e3a8179bafa43b8dddfd71f28ccbe3"}
Sep 10 14:45:03.657882 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:03.657741    2547 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27c8b33b87992a98a51ef766b494f980b7e3a8179bafa43b8dddfd71f28ccbe3"
Sep 10 14:45:03.659084 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:03.659064    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc9af3-29817525-dtfb6"
Sep 10 14:45:03.659185 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:03.659066    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc9af3-29817525-dtfb6" event={"ID":"1c1263a1-00d5-4f5d-94eb-b7cdfbc471d9","Type":"ContainerDied","Data":"5abf0de9dfdbc6dd94416f052b895245c02c2755b2f225748eca1433cbb623e5"}
Sep 10 14:45:03.659230 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:03.659206    2547 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5abf0de9dfdbc6dd94416f052b895245c02c2755b2f225748eca1433cbb623e5"
Sep 10 14:45:07.212715 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:07.212659    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7697-e89777"]
Sep 10 14:45:07.213107 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:07.213061    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c1263a1-00d5-4f5d-94eb-b7cdfbc471d9" containerName="cleanup"
Sep 10 14:45:07.213107 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:07.213078    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c1263a1-00d5-4f5d-94eb-b7cdfbc471d9" containerName="cleanup"
Sep 10 14:45:07.213183 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:07.213108    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc199420-d1ca-419e-93af-bfcc8c0956ac" containerName="cleanup"
Sep 10 14:45:07.213183 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:07.213114    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc199420-d1ca-419e-93af-bfcc8c0956ac" containerName="cleanup"
Sep 10 14:45:07.213183 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:07.213181    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc199420-d1ca-419e-93af-bfcc8c0956ac" containerName="cleanup"
Sep 10 14:45:07.213277 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:07.213192    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c1263a1-00d5-4f5d-94eb-b7cdfbc471d9" containerName="cleanup"
Sep 10 14:45:07.216427 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:07.216410    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-e89777"
Sep 10 14:45:07.219231 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:07.219206    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4cjjz\""
Sep 10 14:45:07.221365 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:07.221340    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-e89777"]
Sep 10 14:45:07.273718 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:07.273660    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jkv8\" (UniqueName: \"kubernetes.io/projected/348f6074-bcea-45f0-a12f-bbca0f93e5cb-kube-api-access-8jkv8\") pod \"test-curl-7697-e89777\" (UID: \"348f6074-bcea-45f0-a12f-bbca0f93e5cb\") " pod="opendatahub/test-curl-7697-e89777"
Sep 10 14:45:07.374622 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:07.374577    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jkv8\" (UniqueName: \"kubernetes.io/projected/348f6074-bcea-45f0-a12f-bbca0f93e5cb-kube-api-access-8jkv8\") pod \"test-curl-7697-e89777\" (UID: \"348f6074-bcea-45f0-a12f-bbca0f93e5cb\") " pod="opendatahub/test-curl-7697-e89777"
Sep 10 14:45:07.383334 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:07.383300    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jkv8\" (UniqueName: \"kubernetes.io/projected/348f6074-bcea-45f0-a12f-bbca0f93e5cb-kube-api-access-8jkv8\") pod \"test-curl-7697-e89777\" (UID: \"348f6074-bcea-45f0-a12f-bbca0f93e5cb\") " pod="opendatahub/test-curl-7697-e89777"
Sep 10 14:45:07.526442 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:07.526350    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-e89777"
Sep 10 14:45:07.796397 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:07.796373    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-e89777"]
Sep 10 14:45:07.798047 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:45:07.798021    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod348f6074_bcea_45f0_a12f_bbca0f93e5cb.slice/crio-728dfef35216c7a320adc642f81271c52f8353a372bcd4c7e10603e0fa25a501 WatchSource:0}: Error finding container 728dfef35216c7a320adc642f81271c52f8353a372bcd4c7e10603e0fa25a501: Status 404 returned error can't find the container with id 728dfef35216c7a320adc642f81271c52f8353a372bcd4c7e10603e0fa25a501
Sep 10 14:45:08.689028 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:08.688990    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-e89777" event={"ID":"348f6074-bcea-45f0-a12f-bbca0f93e5cb","Type":"ContainerStarted","Data":"e53de1014f3ecd64e2432d8398d60f9191adc1b159584bdee93b889a03b1d2dd"}
Sep 10 14:45:08.689028 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:08.689032    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-e89777" event={"ID":"348f6074-bcea-45f0-a12f-bbca0f93e5cb","Type":"ContainerStarted","Data":"728dfef35216c7a320adc642f81271c52f8353a372bcd4c7e10603e0fa25a501"}
Sep 10 14:45:08.705365 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:08.705317    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7697-e89777" podStartSLOduration=1.7053022260000001 podStartE2EDuration="1.705302226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:45:08.702622818 +0000 UTC m=+1520.744497301" watchObservedRunningTime="2026-09-10 14:45:08.705302226 +0000 UTC m=+1520.747176707"
Sep 10 14:45:09.277382 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:09.277343    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7697-e89777"]
Sep 10 14:45:09.287298 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:09.287260    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7697-e89777"]
Sep 10 14:45:09.394774 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:09.394670    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7697-fa8c93"]
Sep 10 14:45:09.395086 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:09.395072    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="348f6074-bcea-45f0-a12f-bbca0f93e5cb" containerName="test-curl-7697-e89777"
Sep 10 14:45:09.395129 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:09.395088    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="348f6074-bcea-45f0-a12f-bbca0f93e5cb" containerName="test-curl-7697-e89777"
Sep 10 14:45:09.395164 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:09.395149    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="348f6074-bcea-45f0-a12f-bbca0f93e5cb" containerName="test-curl-7697-e89777"
Sep 10 14:45:09.398315 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:09.398297    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-fa8c93"
Sep 10 14:45:09.402114 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:09.402088    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-fa8c93"]
Sep 10 14:45:09.494819 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:09.494779    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkhk8\" (UniqueName: \"kubernetes.io/projected/06bc19ec-f2ce-466f-bac4-a7ad29cc92cd-kube-api-access-lkhk8\") pod \"test-curl-7697-fa8c93\" (UID: \"06bc19ec-f2ce-466f-bac4-a7ad29cc92cd\") " pod="opendatahub/test-curl-7697-fa8c93"
Sep 10 14:45:09.595861 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:09.595825    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkhk8\" (UniqueName: \"kubernetes.io/projected/06bc19ec-f2ce-466f-bac4-a7ad29cc92cd-kube-api-access-lkhk8\") pod \"test-curl-7697-fa8c93\" (UID: \"06bc19ec-f2ce-466f-bac4-a7ad29cc92cd\") " pod="opendatahub/test-curl-7697-fa8c93"
Sep 10 14:45:09.604443 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:09.604419    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkhk8\" (UniqueName: \"kubernetes.io/projected/06bc19ec-f2ce-466f-bac4-a7ad29cc92cd-kube-api-access-lkhk8\") pod \"test-curl-7697-fa8c93\" (UID: \"06bc19ec-f2ce-466f-bac4-a7ad29cc92cd\") " pod="opendatahub/test-curl-7697-fa8c93"
Sep 10 14:45:09.694473 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:45:09.694443    2547 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7697-e89777" container="test-curl-7697-e89777"
Sep 10 14:45:09.708792 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:09.708770    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-fa8c93"
Sep 10 14:45:09.978043 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:09.978016    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-fa8c93"]
Sep 10 14:45:09.979155 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:45:09.979131    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06bc19ec_f2ce_466f_bac4_a7ad29cc92cd.slice/crio-2b1aa4ea36ba0019d89a3c7944b0c85e4dbd96c1d60d22d1dcf4b6b9a3478788 WatchSource:0}: Error finding container 2b1aa4ea36ba0019d89a3c7944b0c85e4dbd96c1d60d22d1dcf4b6b9a3478788: Status 404 returned error can't find the container with id 2b1aa4ea36ba0019d89a3c7944b0c85e4dbd96c1d60d22d1dcf4b6b9a3478788
Sep 10 14:45:10.210223 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.210184    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"]
Sep 10 14:45:10.213775 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.213754    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:45:10.217733 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.217706    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3477ac\"/\"kube-root-ca.crt\""
Sep 10 14:45:10.217733 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.217723    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3477ac\"/\"auth-test-model-3477ac-kserve-self-signed-certs\""
Sep 10 14:45:10.217904 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.217771    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3477ac\"/\"default-dockercfg-f4brn\""
Sep 10 14:45:10.217904 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.217772    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3477ac\"/\"openshift-service-ca.crt\""
Sep 10 14:45:10.226573 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.226515    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"]
Sep 10 14:45:10.301354 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.301320    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wncsl\" (UniqueName: \"kubernetes.io/projected/0381c71b-7613-461f-bcfd-9d19e14fe4d5-kube-api-access-wncsl\") pod \"auth-test-model-3477ac-kserve-b946b8b47-286xt\" (UID: \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:45:10.301525 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.301384    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0381c71b-7613-461f-bcfd-9d19e14fe4d5-tls-certs\") pod \"auth-test-model-3477ac-kserve-b946b8b47-286xt\" (UID: \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:45:10.301525 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.301442    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0381c71b-7613-461f-bcfd-9d19e14fe4d5-home\") pod \"auth-test-model-3477ac-kserve-b946b8b47-286xt\" (UID: \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:45:10.301525 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.301516    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0381c71b-7613-461f-bcfd-9d19e14fe4d5-dshm\") pod \"auth-test-model-3477ac-kserve-b946b8b47-286xt\" (UID: \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:45:10.301739 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.301566    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0381c71b-7613-461f-bcfd-9d19e14fe4d5-model-cache\") pod \"auth-test-model-3477ac-kserve-b946b8b47-286xt\" (UID: \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:45:10.301739 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.301602    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0381c71b-7613-461f-bcfd-9d19e14fe4d5-tmp-dir\") pod \"auth-test-model-3477ac-kserve-b946b8b47-286xt\" (UID: \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:45:10.402943 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.402907    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0381c71b-7613-461f-bcfd-9d19e14fe4d5-model-cache\") pod \"auth-test-model-3477ac-kserve-b946b8b47-286xt\" (UID: \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:45:10.402943 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.402952    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0381c71b-7613-461f-bcfd-9d19e14fe4d5-tmp-dir\") pod \"auth-test-model-3477ac-kserve-b946b8b47-286xt\" (UID: \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:45:10.403201 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.402976    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wncsl\" (UniqueName: \"kubernetes.io/projected/0381c71b-7613-461f-bcfd-9d19e14fe4d5-kube-api-access-wncsl\") pod \"auth-test-model-3477ac-kserve-b946b8b47-286xt\" (UID: \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:45:10.403201 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.403031    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0381c71b-7613-461f-bcfd-9d19e14fe4d5-tls-certs\") pod \"auth-test-model-3477ac-kserve-b946b8b47-286xt\" (UID: \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:45:10.403201 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.403060    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0381c71b-7613-461f-bcfd-9d19e14fe4d5-home\") pod \"auth-test-model-3477ac-kserve-b946b8b47-286xt\" (UID: \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:45:10.403201 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.403088    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0381c71b-7613-461f-bcfd-9d19e14fe4d5-dshm\") pod \"auth-test-model-3477ac-kserve-b946b8b47-286xt\" (UID: \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:45:10.403404 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.403358    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0381c71b-7613-461f-bcfd-9d19e14fe4d5-model-cache\") pod \"auth-test-model-3477ac-kserve-b946b8b47-286xt\" (UID: \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:45:10.403460 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.403408    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0381c71b-7613-461f-bcfd-9d19e14fe4d5-tmp-dir\") pod \"auth-test-model-3477ac-kserve-b946b8b47-286xt\" (UID: \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:45:10.403460 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.403445    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0381c71b-7613-461f-bcfd-9d19e14fe4d5-home\") pod \"auth-test-model-3477ac-kserve-b946b8b47-286xt\" (UID: \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:45:10.405187 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.405168    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0381c71b-7613-461f-bcfd-9d19e14fe4d5-dshm\") pod \"auth-test-model-3477ac-kserve-b946b8b47-286xt\" (UID: \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:45:10.405449 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.405432    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0381c71b-7613-461f-bcfd-9d19e14fe4d5-tls-certs\") pod \"auth-test-model-3477ac-kserve-b946b8b47-286xt\" (UID: \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:45:10.412552 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.412532    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wncsl\" (UniqueName: \"kubernetes.io/projected/0381c71b-7613-461f-bcfd-9d19e14fe4d5-kube-api-access-wncsl\") pod \"auth-test-model-3477ac-kserve-b946b8b47-286xt\" (UID: \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:45:10.525860 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.525773    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:45:10.702003 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.701964    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-fa8c93" event={"ID":"06bc19ec-f2ce-466f-bac4-a7ad29cc92cd","Type":"ContainerStarted","Data":"d6e68147abf0f2af419e08b9bc03f12a26a98d4a835916fc6a5e3b25c7c9c5d4"}
Sep 10 14:45:10.702003 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.702016    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-fa8c93" event={"ID":"06bc19ec-f2ce-466f-bac4-a7ad29cc92cd","Type":"ContainerStarted","Data":"2b1aa4ea36ba0019d89a3c7944b0c85e4dbd96c1d60d22d1dcf4b6b9a3478788"}
Sep 10 14:45:10.702467 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.702041    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7697-e89777" podUID="348f6074-bcea-45f0-a12f-bbca0f93e5cb" containerName="test-curl-7697-e89777" containerID="cri-o://e53de1014f3ecd64e2432d8398d60f9191adc1b159584bdee93b889a03b1d2dd" gracePeriod=2
Sep 10 14:45:10.718299 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.718254    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7697-fa8c93" podStartSLOduration=1.718240183 podStartE2EDuration="1.718240183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:45:10.716725993 +0000 UTC m=+1522.758600473" watchObservedRunningTime="2026-09-10 14:45:10.718240183 +0000 UTC m=+1522.760114664"
Sep 10 14:45:10.798425 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:10.798394    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"]
Sep 10 14:45:10.799819 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:45:10.799790    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0381c71b_7613_461f_bcfd_9d19e14fe4d5.slice/crio-4f1e992c30522742db77338e6e917c5f56d11cd764deedc687f3693c3d4ab33b WatchSource:0}: Error finding container 4f1e992c30522742db77338e6e917c5f56d11cd764deedc687f3693c3d4ab33b: Status 404 returned error can't find the container with id 4f1e992c30522742db77338e6e917c5f56d11cd764deedc687f3693c3d4ab33b
Sep 10 14:45:11.343991 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:11.343949    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7697-fa8c93"]
Sep 10 14:45:11.349026 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:11.348997    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7697-fa8c93"]
Sep 10 14:45:11.465006 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:11.464978    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7697-85374e"]
Sep 10 14:45:11.465371 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:11.465358    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06bc19ec-f2ce-466f-bac4-a7ad29cc92cd" containerName="test-curl-7697-fa8c93"
Sep 10 14:45:11.465419 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:11.465372    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="06bc19ec-f2ce-466f-bac4-a7ad29cc92cd" containerName="test-curl-7697-fa8c93"
Sep 10 14:45:11.465454 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:11.465431    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="06bc19ec-f2ce-466f-bac4-a7ad29cc92cd" containerName="test-curl-7697-fa8c93"
Sep 10 14:45:11.468868 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:11.468849    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-85374e"
Sep 10 14:45:11.474999 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:11.474975    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-85374e"]
Sep 10 14:45:11.512986 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:11.512955    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwqtd\" (UniqueName: \"kubernetes.io/projected/37b6c568-3605-41aa-88c9-c498dd044ae1-kube-api-access-lwqtd\") pod \"test-curl-7697-85374e\" (UID: \"37b6c568-3605-41aa-88c9-c498dd044ae1\") " pod="opendatahub/test-curl-7697-85374e"
Sep 10 14:45:11.614383 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:11.614346    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwqtd\" (UniqueName: \"kubernetes.io/projected/37b6c568-3605-41aa-88c9-c498dd044ae1-kube-api-access-lwqtd\") pod \"test-curl-7697-85374e\" (UID: \"37b6c568-3605-41aa-88c9-c498dd044ae1\") " pod="opendatahub/test-curl-7697-85374e"
Sep 10 14:45:11.622488 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:11.622449    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwqtd\" (UniqueName: \"kubernetes.io/projected/37b6c568-3605-41aa-88c9-c498dd044ae1-kube-api-access-lwqtd\") pod \"test-curl-7697-85374e\" (UID: \"37b6c568-3605-41aa-88c9-c498dd044ae1\") " pod="opendatahub/test-curl-7697-85374e"
Sep 10 14:45:11.709522 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:11.709476    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt" event={"ID":"0381c71b-7613-461f-bcfd-9d19e14fe4d5","Type":"ContainerStarted","Data":"31c5792b62f23f223717748ef78a58f54025c2eb39d2998ab01ef812dbeedd73"}
Sep 10 14:45:11.709899 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:11.709534    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt" event={"ID":"0381c71b-7613-461f-bcfd-9d19e14fe4d5","Type":"ContainerStarted","Data":"4f1e992c30522742db77338e6e917c5f56d11cd764deedc687f3693c3d4ab33b"}
Sep 10 14:45:11.709899 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:45:11.709673    2547 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7697-fa8c93" container="test-curl-7697-fa8c93"
Sep 10 14:45:11.727881 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:11.727794    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt" podStartSLOduration=1.727779754 podStartE2EDuration="1.727779754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:45:11.725276509 +0000 UTC m=+1523.767151001" watchObservedRunningTime="2026-09-10 14:45:11.727779754 +0000 UTC m=+1523.769654235"
Sep 10 14:45:11.779810 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:11.779782    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-85374e"
Sep 10 14:45:12.051970 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:12.051947    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-85374e"]
Sep 10 14:45:12.054215 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:45:12.054182    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37b6c568_3605_41aa_88c9_c498dd044ae1.slice/crio-1a7c8756c7ca67ac2ef72a408510e898c1ffe7bcac53970d4197580b8cf170ef WatchSource:0}: Error finding container 1a7c8756c7ca67ac2ef72a408510e898c1ffe7bcac53970d4197580b8cf170ef: Status 404 returned error can't find the container with id 1a7c8756c7ca67ac2ef72a408510e898c1ffe7bcac53970d4197580b8cf170ef
Sep 10 14:45:12.717074 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:12.717036    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-85374e" event={"ID":"37b6c568-3605-41aa-88c9-c498dd044ae1","Type":"ContainerStarted","Data":"8a708ac2cb386bb56dda095d9967b64e84d6aaa96a6c6bdb22e9c8ea48799354"}
Sep 10 14:45:12.717443 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:12.717092    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-85374e" event={"ID":"37b6c568-3605-41aa-88c9-c498dd044ae1","Type":"ContainerStarted","Data":"1a7c8756c7ca67ac2ef72a408510e898c1ffe7bcac53970d4197580b8cf170ef"}
Sep 10 14:45:12.717510 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:12.717421    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7697-fa8c93" podUID="06bc19ec-f2ce-466f-bac4-a7ad29cc92cd" containerName="test-curl-7697-fa8c93" containerID="cri-o://d6e68147abf0f2af419e08b9bc03f12a26a98d4a835916fc6a5e3b25c7c9c5d4" gracePeriod=2
Sep 10 14:45:12.733689 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:12.733641    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7697-85374e" podStartSLOduration=1.7336273210000002 podStartE2EDuration="1.733627321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:45:12.730799684 +0000 UTC m=+1524.772674159" watchObservedRunningTime="2026-09-10 14:45:12.733627321 +0000 UTC m=+1524.775501869"
Sep 10 14:45:12.881566 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:12.881541    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-e89777"
Sep 10 14:45:12.925854 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:12.925824    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jkv8\" (UniqueName: \"kubernetes.io/projected/348f6074-bcea-45f0-a12f-bbca0f93e5cb-kube-api-access-8jkv8\") pod \"348f6074-bcea-45f0-a12f-bbca0f93e5cb\" (UID: \"348f6074-bcea-45f0-a12f-bbca0f93e5cb\") "
Sep 10 14:45:12.927850 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:12.927823    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348f6074-bcea-45f0-a12f-bbca0f93e5cb-kube-api-access-8jkv8" (OuterVolumeSpecName: "kube-api-access-8jkv8") pod "348f6074-bcea-45f0-a12f-bbca0f93e5cb" (UID: "348f6074-bcea-45f0-a12f-bbca0f93e5cb"). InnerVolumeSpecName "kube-api-access-8jkv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:45:13.027134 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:13.027056    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jkv8\" (UniqueName: \"kubernetes.io/projected/348f6074-bcea-45f0-a12f-bbca0f93e5cb-kube-api-access-8jkv8\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:45:13.232166 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:13.232133    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:45:13.300425 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:13.300335    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7697-85374e"]
Sep 10 14:45:13.308404 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:13.308371    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7697-85374e"]
Sep 10 14:45:13.420608 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:13.420573    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7697-806a30"]
Sep 10 14:45:13.421049 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:13.421031    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b6c568-3605-41aa-88c9-c498dd044ae1" containerName="test-curl-7697-85374e"
Sep 10 14:45:13.421098 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:13.421051    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b6c568-3605-41aa-88c9-c498dd044ae1" containerName="test-curl-7697-85374e"
Sep 10 14:45:13.421131 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:13.421109    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="37b6c568-3605-41aa-88c9-c498dd044ae1" containerName="test-curl-7697-85374e"
Sep 10 14:45:13.424124 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:13.424110    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-806a30"
Sep 10 14:45:13.427868 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:13.427845    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-806a30"]
Sep 10 14:45:13.531295 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:13.531259    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6ggm\" (UniqueName: \"kubernetes.io/projected/620d2282-75cd-4948-a5a4-afa716c5f87b-kube-api-access-b6ggm\") pod \"test-curl-7697-806a30\" (UID: \"620d2282-75cd-4948-a5a4-afa716c5f87b\") " pod="opendatahub/test-curl-7697-806a30"
Sep 10 14:45:13.632469 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:13.632385    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6ggm\" (UniqueName: \"kubernetes.io/projected/620d2282-75cd-4948-a5a4-afa716c5f87b-kube-api-access-b6ggm\") pod \"test-curl-7697-806a30\" (UID: \"620d2282-75cd-4948-a5a4-afa716c5f87b\") " pod="opendatahub/test-curl-7697-806a30"
Sep 10 14:45:13.641147 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:13.641110    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6ggm\" (UniqueName: \"kubernetes.io/projected/620d2282-75cd-4948-a5a4-afa716c5f87b-kube-api-access-b6ggm\") pod \"test-curl-7697-806a30\" (UID: \"620d2282-75cd-4948-a5a4-afa716c5f87b\") " pod="opendatahub/test-curl-7697-806a30"
Sep 10 14:45:13.723669 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:13.723629    2547 generic.go:356] "Generic (PLEG): container finished" podID="348f6074-bcea-45f0-a12f-bbca0f93e5cb" containerID="e53de1014f3ecd64e2432d8398d60f9191adc1b159584bdee93b889a03b1d2dd" exitCode=137
Sep 10 14:45:13.724200 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:13.723673    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-e89777"
Sep 10 14:45:13.724200 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:13.723764    2547 scope.go:117] "RemoveContainer" containerID="e53de1014f3ecd64e2432d8398d60f9191adc1b159584bdee93b889a03b1d2dd"
Sep 10 14:45:13.724200 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:45:13.724056    2547 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7697-85374e" container="test-curl-7697-85374e"
Sep 10 14:45:13.733234 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:13.733214    2547 scope.go:117] "RemoveContainer" containerID="e53de1014f3ecd64e2432d8398d60f9191adc1b159584bdee93b889a03b1d2dd"
Sep 10 14:45:13.733517 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:45:13.733499    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e53de1014f3ecd64e2432d8398d60f9191adc1b159584bdee93b889a03b1d2dd\": container with ID starting with e53de1014f3ecd64e2432d8398d60f9191adc1b159584bdee93b889a03b1d2dd not found: ID does not exist" containerID="e53de1014f3ecd64e2432d8398d60f9191adc1b159584bdee93b889a03b1d2dd"
Sep 10 14:45:13.733563 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:13.733526    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e53de1014f3ecd64e2432d8398d60f9191adc1b159584bdee93b889a03b1d2dd"} err="failed to get container status \"e53de1014f3ecd64e2432d8398d60f9191adc1b159584bdee93b889a03b1d2dd\": rpc error: code = NotFound desc = could not find container \"e53de1014f3ecd64e2432d8398d60f9191adc1b159584bdee93b889a03b1d2dd\": container with ID starting with e53de1014f3ecd64e2432d8398d60f9191adc1b159584bdee93b889a03b1d2dd not found: ID does not exist"
Sep 10 14:45:13.734737 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:45:13.734672    2547 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7697-e89777\" is forbidden: User \"system:node:ip-10-0-130-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-249.ec2.internal' and this object" podUID="348f6074-bcea-45f0-a12f-bbca0f93e5cb" pod="opendatahub/test-curl-7697-e89777"
Sep 10 14:45:13.735752 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:13.735735    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-806a30"
Sep 10 14:45:14.012608 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:14.012580    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-806a30"]
Sep 10 14:45:14.015359 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:45:14.015327    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod620d2282_75cd_4948_a5a4_afa716c5f87b.slice/crio-a864bffba18bd63fe6056819b977ed4574fac2474e8520bb83970a8063ca2f70 WatchSource:0}: Error finding container a864bffba18bd63fe6056819b977ed4574fac2474e8520bb83970a8063ca2f70: Status 404 returned error can't find the container with id a864bffba18bd63fe6056819b977ed4574fac2474e8520bb83970a8063ca2f70
Sep 10 14:45:14.421683 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:14.421645    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="348f6074-bcea-45f0-a12f-bbca0f93e5cb" path="/var/lib/kubelet/pods/348f6074-bcea-45f0-a12f-bbca0f93e5cb/volumes"
Sep 10 14:45:14.730529 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:14.730497    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-806a30" event={"ID":"620d2282-75cd-4948-a5a4-afa716c5f87b","Type":"ContainerStarted","Data":"8a8636019a67b3d21972a33defc47e6cea41fc681ea5371e7428f098845a5f08"}
Sep 10 14:45:14.730887 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:14.730547    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-806a30" event={"ID":"620d2282-75cd-4948-a5a4-afa716c5f87b","Type":"ContainerStarted","Data":"a864bffba18bd63fe6056819b977ed4574fac2474e8520bb83970a8063ca2f70"}
Sep 10 14:45:14.731675 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:14.731650    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7697-85374e" podUID="37b6c568-3605-41aa-88c9-c498dd044ae1" containerName="test-curl-7697-85374e" containerID="cri-o://8a708ac2cb386bb56dda095d9967b64e84d6aaa96a6c6bdb22e9c8ea48799354" gracePeriod=2
Sep 10 14:45:14.745592 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:14.745552    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7697-806a30" podStartSLOduration=1.745536708 podStartE2EDuration="1.745536708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:45:14.743978451 +0000 UTC m=+1526.785852933" watchObservedRunningTime="2026-09-10 14:45:14.745536708 +0000 UTC m=+1526.787411190"
Sep 10 14:45:14.904905 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:14.904875    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-fa8c93"
Sep 10 14:45:14.944560 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:14.944532    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkhk8\" (UniqueName: \"kubernetes.io/projected/06bc19ec-f2ce-466f-bac4-a7ad29cc92cd-kube-api-access-lkhk8\") pod \"06bc19ec-f2ce-466f-bac4-a7ad29cc92cd\" (UID: \"06bc19ec-f2ce-466f-bac4-a7ad29cc92cd\") "
Sep 10 14:45:14.946460 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:14.946432    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06bc19ec-f2ce-466f-bac4-a7ad29cc92cd-kube-api-access-lkhk8" (OuterVolumeSpecName: "kube-api-access-lkhk8") pod "06bc19ec-f2ce-466f-bac4-a7ad29cc92cd" (UID: "06bc19ec-f2ce-466f-bac4-a7ad29cc92cd"). InnerVolumeSpecName "kube-api-access-lkhk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:45:15.045571 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:15.045487    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkhk8\" (UniqueName: \"kubernetes.io/projected/06bc19ec-f2ce-466f-bac4-a7ad29cc92cd-kube-api-access-lkhk8\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:45:15.286756 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:15.286718    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7697-806a30"]
Sep 10 14:45:15.290873 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:15.290844    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7697-806a30"]
Sep 10 14:45:15.737857 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:15.737823    2547 generic.go:356] "Generic (PLEG): container finished" podID="06bc19ec-f2ce-466f-bac4-a7ad29cc92cd" containerID="d6e68147abf0f2af419e08b9bc03f12a26a98d4a835916fc6a5e3b25c7c9c5d4" exitCode=137
Sep 10 14:45:15.738241 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:15.737867    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-fa8c93"
Sep 10 14:45:15.738241 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:15.737914    2547 scope.go:117] "RemoveContainer" containerID="d6e68147abf0f2af419e08b9bc03f12a26a98d4a835916fc6a5e3b25c7c9c5d4"
Sep 10 14:45:15.738324 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:45:15.738270    2547 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7697-806a30" container="test-curl-7697-806a30"
Sep 10 14:45:15.747467 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:15.747450    2547 scope.go:117] "RemoveContainer" containerID="d6e68147abf0f2af419e08b9bc03f12a26a98d4a835916fc6a5e3b25c7c9c5d4"
Sep 10 14:45:15.747740 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:45:15.747722    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6e68147abf0f2af419e08b9bc03f12a26a98d4a835916fc6a5e3b25c7c9c5d4\": container with ID starting with d6e68147abf0f2af419e08b9bc03f12a26a98d4a835916fc6a5e3b25c7c9c5d4 not found: ID does not exist" containerID="d6e68147abf0f2af419e08b9bc03f12a26a98d4a835916fc6a5e3b25c7c9c5d4"
Sep 10 14:45:15.747791 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:15.747748    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6e68147abf0f2af419e08b9bc03f12a26a98d4a835916fc6a5e3b25c7c9c5d4"} err="failed to get container status \"d6e68147abf0f2af419e08b9bc03f12a26a98d4a835916fc6a5e3b25c7c9c5d4\": rpc error: code = NotFound desc = could not find container \"d6e68147abf0f2af419e08b9bc03f12a26a98d4a835916fc6a5e3b25c7c9c5d4\": container with ID starting with d6e68147abf0f2af419e08b9bc03f12a26a98d4a835916fc6a5e3b25c7c9c5d4 not found: ID does not exist"
Sep 10 14:45:15.748922 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:45:15.748898    2547 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7697-fa8c93\" is forbidden: User \"system:node:ip-10-0-130-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-249.ec2.internal' and this object" podUID="06bc19ec-f2ce-466f-bac4-a7ad29cc92cd" pod="opendatahub/test-curl-7697-fa8c93"
Sep 10 14:45:16.421888 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:16.421855    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06bc19ec-f2ce-466f-bac4-a7ad29cc92cd" path="/var/lib/kubelet/pods/06bc19ec-f2ce-466f-bac4-a7ad29cc92cd/volumes"
Sep 10 14:45:16.747486 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:16.747451    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7697-806a30" podUID="620d2282-75cd-4948-a5a4-afa716c5f87b" containerName="test-curl-7697-806a30" containerID="cri-o://8a8636019a67b3d21972a33defc47e6cea41fc681ea5371e7428f098845a5f08" gracePeriod=2
Sep 10 14:45:16.866155 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:16.866127    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-85374e"
Sep 10 14:45:16.965396 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:16.965356    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwqtd\" (UniqueName: \"kubernetes.io/projected/37b6c568-3605-41aa-88c9-c498dd044ae1-kube-api-access-lwqtd\") pod \"37b6c568-3605-41aa-88c9-c498dd044ae1\" (UID: \"37b6c568-3605-41aa-88c9-c498dd044ae1\") "
Sep 10 14:45:16.967577 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:16.967551    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b6c568-3605-41aa-88c9-c498dd044ae1-kube-api-access-lwqtd" (OuterVolumeSpecName: "kube-api-access-lwqtd") pod "37b6c568-3605-41aa-88c9-c498dd044ae1" (UID: "37b6c568-3605-41aa-88c9-c498dd044ae1"). InnerVolumeSpecName "kube-api-access-lwqtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:45:17.066635 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:17.066551    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwqtd\" (UniqueName: \"kubernetes.io/projected/37b6c568-3605-41aa-88c9-c498dd044ae1-kube-api-access-lwqtd\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:45:17.754233 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:17.754199    2547 generic.go:356] "Generic (PLEG): container finished" podID="37b6c568-3605-41aa-88c9-c498dd044ae1" containerID="8a708ac2cb386bb56dda095d9967b64e84d6aaa96a6c6bdb22e9c8ea48799354" exitCode=137
Sep 10 14:45:17.754671 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:17.754241    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-85374e"
Sep 10 14:45:17.754671 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:17.754264    2547 scope.go:117] "RemoveContainer" containerID="8a708ac2cb386bb56dda095d9967b64e84d6aaa96a6c6bdb22e9c8ea48799354"
Sep 10 14:45:17.764172 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:17.764155    2547 scope.go:117] "RemoveContainer" containerID="8a708ac2cb386bb56dda095d9967b64e84d6aaa96a6c6bdb22e9c8ea48799354"
Sep 10 14:45:17.764465 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:45:17.764444    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a708ac2cb386bb56dda095d9967b64e84d6aaa96a6c6bdb22e9c8ea48799354\": container with ID starting with 8a708ac2cb386bb56dda095d9967b64e84d6aaa96a6c6bdb22e9c8ea48799354 not found: ID does not exist" containerID="8a708ac2cb386bb56dda095d9967b64e84d6aaa96a6c6bdb22e9c8ea48799354"
Sep 10 14:45:17.764538 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:17.764472    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a708ac2cb386bb56dda095d9967b64e84d6aaa96a6c6bdb22e9c8ea48799354"} err="failed to get container status \"8a708ac2cb386bb56dda095d9967b64e84d6aaa96a6c6bdb22e9c8ea48799354\": rpc error: code = NotFound desc = could not find container \"8a708ac2cb386bb56dda095d9967b64e84d6aaa96a6c6bdb22e9c8ea48799354\": container with ID starting with 8a708ac2cb386bb56dda095d9967b64e84d6aaa96a6c6bdb22e9c8ea48799354 not found: ID does not exist"
Sep 10 14:45:17.765744 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:45:17.765718    2547 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7697-85374e\" is forbidden: User \"system:node:ip-10-0-130-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-249.ec2.internal' and this object" podUID="37b6c568-3605-41aa-88c9-c498dd044ae1" pod="opendatahub/test-curl-7697-85374e"
Sep 10 14:45:18.421014 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:45:18.420974    2547 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7697-85374e\" is forbidden: User \"system:node:ip-10-0-130-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-249.ec2.internal' and this object" podUID="37b6c568-3605-41aa-88c9-c498dd044ae1" pod="opendatahub/test-curl-7697-85374e"
Sep 10 14:45:18.422029 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:18.422010    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37b6c568-3605-41aa-88c9-c498dd044ae1" path="/var/lib/kubelet/pods/37b6c568-3605-41aa-88c9-c498dd044ae1/volumes"
Sep 10 14:45:18.892257 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:18.892231    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-806a30"
Sep 10 14:45:18.984004 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:18.983971    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6ggm\" (UniqueName: \"kubernetes.io/projected/620d2282-75cd-4948-a5a4-afa716c5f87b-kube-api-access-b6ggm\") pod \"620d2282-75cd-4948-a5a4-afa716c5f87b\" (UID: \"620d2282-75cd-4948-a5a4-afa716c5f87b\") "
Sep 10 14:45:18.986049 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:18.986026    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620d2282-75cd-4948-a5a4-afa716c5f87b-kube-api-access-b6ggm" (OuterVolumeSpecName: "kube-api-access-b6ggm") pod "620d2282-75cd-4948-a5a4-afa716c5f87b" (UID: "620d2282-75cd-4948-a5a4-afa716c5f87b"). InnerVolumeSpecName "kube-api-access-b6ggm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:45:19.085032 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:19.084948    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6ggm\" (UniqueName: \"kubernetes.io/projected/620d2282-75cd-4948-a5a4-afa716c5f87b-kube-api-access-b6ggm\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:45:19.768809 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:19.768772    2547 generic.go:356] "Generic (PLEG): container finished" podID="620d2282-75cd-4948-a5a4-afa716c5f87b" containerID="8a8636019a67b3d21972a33defc47e6cea41fc681ea5371e7428f098845a5f08" exitCode=137
Sep 10 14:45:19.768981 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:19.768817    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-806a30"
Sep 10 14:45:19.768981 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:19.768878    2547 scope.go:117] "RemoveContainer" containerID="8a8636019a67b3d21972a33defc47e6cea41fc681ea5371e7428f098845a5f08"
Sep 10 14:45:19.778501 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:19.778484    2547 scope.go:117] "RemoveContainer" containerID="8a8636019a67b3d21972a33defc47e6cea41fc681ea5371e7428f098845a5f08"
Sep 10 14:45:19.778774 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:45:19.778757    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a8636019a67b3d21972a33defc47e6cea41fc681ea5371e7428f098845a5f08\": container with ID starting with 8a8636019a67b3d21972a33defc47e6cea41fc681ea5371e7428f098845a5f08 not found: ID does not exist" containerID="8a8636019a67b3d21972a33defc47e6cea41fc681ea5371e7428f098845a5f08"
Sep 10 14:45:19.778841 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:19.778780    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a8636019a67b3d21972a33defc47e6cea41fc681ea5371e7428f098845a5f08"} err="failed to get container status \"8a8636019a67b3d21972a33defc47e6cea41fc681ea5371e7428f098845a5f08\": rpc error: code = NotFound desc = could not find container \"8a8636019a67b3d21972a33defc47e6cea41fc681ea5371e7428f098845a5f08\": container with ID starting with 8a8636019a67b3d21972a33defc47e6cea41fc681ea5371e7428f098845a5f08 not found: ID does not exist"
Sep 10 14:45:19.779805 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:45:19.779785    2547 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7697-806a30\" is forbidden: User \"system:node:ip-10-0-130-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-249.ec2.internal' and this object" podUID="620d2282-75cd-4948-a5a4-afa716c5f87b" pod="opendatahub/test-curl-7697-806a30"
Sep 10 14:45:20.420888 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:20.420853    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="620d2282-75cd-4948-a5a4-afa716c5f87b" path="/var/lib/kubelet/pods/620d2282-75cd-4948-a5a4-afa716c5f87b/volumes"
Sep 10 14:45:20.526251 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:20.526215    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:45:20.526405 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:20.526268    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:45:20.539136 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:20.539111    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:45:20.787057 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:20.786968    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:45:29.523861 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:29.523817    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:45:43.227497 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:43.227454    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:45:48.549666 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:48.549632    2547 scope.go:117] "RemoveContainer" containerID="50c27e45597ad24a0e5544de0714d0743304897e55431f6259cd7d5ee5ddab30"
Sep 10 14:45:56.122306 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.122273    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"]
Sep 10 14:45:56.122788 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.122686    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="620d2282-75cd-4948-a5a4-afa716c5f87b" containerName="test-curl-7697-806a30"
Sep 10 14:45:56.122788 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.122729    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="620d2282-75cd-4948-a5a4-afa716c5f87b" containerName="test-curl-7697-806a30"
Sep 10 14:45:56.122884 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.122807    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="620d2282-75cd-4948-a5a4-afa716c5f87b" containerName="test-curl-7697-806a30"
Sep 10 14:45:56.125801 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.125785    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:45:56.129996 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.129973    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-dfd05fbc\"/\"openshift-service-ca.crt\""
Sep 10 14:45:56.130111 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.130023    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-dfd05fbc\"/\"kube-root-ca.crt\""
Sep 10 14:45:56.130111 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.130076    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-dfd05fbc\"/\"default-dockercfg-ffhwp\""
Sep 10 14:45:56.130259 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.130237    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-dfd05fbc\"/\"e2e-lifecycle-model-dfd05fbc-kserve-self-signed-certs\""
Sep 10 14:45:56.136026 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.136004    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"]
Sep 10 14:45:56.232224 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.232188    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40de8b4b-aca8-4f15-ba77-0f922b075063-dshm\") pod \"e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt\" (UID: \"40de8b4b-aca8-4f15-ba77-0f922b075063\") " pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:45:56.232397 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.232241    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40de8b4b-aca8-4f15-ba77-0f922b075063-home\") pod \"e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt\" (UID: \"40de8b4b-aca8-4f15-ba77-0f922b075063\") " pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:45:56.232397 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.232270    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40de8b4b-aca8-4f15-ba77-0f922b075063-tls-certs\") pod \"e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt\" (UID: \"40de8b4b-aca8-4f15-ba77-0f922b075063\") " pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:45:56.232397 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.232344    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40de8b4b-aca8-4f15-ba77-0f922b075063-model-cache\") pod \"e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt\" (UID: \"40de8b4b-aca8-4f15-ba77-0f922b075063\") " pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:45:56.232510 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.232396    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvg7g\" (UniqueName: \"kubernetes.io/projected/40de8b4b-aca8-4f15-ba77-0f922b075063-kube-api-access-nvg7g\") pod \"e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt\" (UID: \"40de8b4b-aca8-4f15-ba77-0f922b075063\") " pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:45:56.232510 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.232441    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40de8b4b-aca8-4f15-ba77-0f922b075063-tmp-dir\") pod \"e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt\" (UID: \"40de8b4b-aca8-4f15-ba77-0f922b075063\") " pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:45:56.333840 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.333801    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40de8b4b-aca8-4f15-ba77-0f922b075063-dshm\") pod \"e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt\" (UID: \"40de8b4b-aca8-4f15-ba77-0f922b075063\") " pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:45:56.334033 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.333851    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40de8b4b-aca8-4f15-ba77-0f922b075063-home\") pod \"e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt\" (UID: \"40de8b4b-aca8-4f15-ba77-0f922b075063\") " pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:45:56.334033 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.333870    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40de8b4b-aca8-4f15-ba77-0f922b075063-tls-certs\") pod \"e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt\" (UID: \"40de8b4b-aca8-4f15-ba77-0f922b075063\") " pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:45:56.334033 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.333890    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40de8b4b-aca8-4f15-ba77-0f922b075063-model-cache\") pod \"e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt\" (UID: \"40de8b4b-aca8-4f15-ba77-0f922b075063\") " pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:45:56.334033 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.333918    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvg7g\" (UniqueName: \"kubernetes.io/projected/40de8b4b-aca8-4f15-ba77-0f922b075063-kube-api-access-nvg7g\") pod \"e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt\" (UID: \"40de8b4b-aca8-4f15-ba77-0f922b075063\") " pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:45:56.334033 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.333935    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40de8b4b-aca8-4f15-ba77-0f922b075063-tmp-dir\") pod \"e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt\" (UID: \"40de8b4b-aca8-4f15-ba77-0f922b075063\") " pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:45:56.334317 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.334296    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40de8b4b-aca8-4f15-ba77-0f922b075063-home\") pod \"e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt\" (UID: \"40de8b4b-aca8-4f15-ba77-0f922b075063\") " pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:45:56.334381 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.334349    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40de8b4b-aca8-4f15-ba77-0f922b075063-tmp-dir\") pod \"e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt\" (UID: \"40de8b4b-aca8-4f15-ba77-0f922b075063\") " pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:45:56.334443 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.334399    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40de8b4b-aca8-4f15-ba77-0f922b075063-model-cache\") pod \"e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt\" (UID: \"40de8b4b-aca8-4f15-ba77-0f922b075063\") " pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:45:56.336057 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.336033    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40de8b4b-aca8-4f15-ba77-0f922b075063-dshm\") pod \"e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt\" (UID: \"40de8b4b-aca8-4f15-ba77-0f922b075063\") " pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:45:56.336270 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.336252    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40de8b4b-aca8-4f15-ba77-0f922b075063-tls-certs\") pod \"e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt\" (UID: \"40de8b4b-aca8-4f15-ba77-0f922b075063\") " pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:45:56.341150 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.341127    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvg7g\" (UniqueName: \"kubernetes.io/projected/40de8b4b-aca8-4f15-ba77-0f922b075063-kube-api-access-nvg7g\") pod \"e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt\" (UID: \"40de8b4b-aca8-4f15-ba77-0f922b075063\") " pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:45:56.436691 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.436661    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:45:56.707131 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.707098    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"]
Sep 10 14:45:56.708115 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:45:56.708089    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40de8b4b_aca8_4f15_ba77_0f922b075063.slice/crio-b18592fa0c76dd43fcf1cf673c0d31be293ac14043245e492418ac6d01efb461 WatchSource:0}: Error finding container b18592fa0c76dd43fcf1cf673c0d31be293ac14043245e492418ac6d01efb461: Status 404 returned error can't find the container with id b18592fa0c76dd43fcf1cf673c0d31be293ac14043245e492418ac6d01efb461
Sep 10 14:45:56.979689 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.979602    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt" event={"ID":"40de8b4b-aca8-4f15-ba77-0f922b075063","Type":"ContainerStarted","Data":"c4f9bb2e15a1a278164db5fe3ff5ad25c334149aa0549d24196e9175c4411378"}
Sep 10 14:45:56.979689 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.979644    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt" event={"ID":"40de8b4b-aca8-4f15-ba77-0f922b075063","Type":"ContainerStarted","Data":"b18592fa0c76dd43fcf1cf673c0d31be293ac14043245e492418ac6d01efb461"}
Sep 10 14:45:56.998829 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:45:56.998783    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt" podStartSLOduration=0.998770392 podStartE2EDuration="998.770392ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:45:56.997036658 +0000 UTC m=+1569.038911139" watchObservedRunningTime="2026-09-10 14:45:56.998770392 +0000 UTC m=+1569.040644873"
Sep 10 14:46:00.031624 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:46:00.031575    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:46:06.437228 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:46:06.437190    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:46:06.437665 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:46:06.437243    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:46:06.449961 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:46:06.449937    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:46:07.048593 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:46:07.048564    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:46:17.728903 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:46:17.728866    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:46:34.622649 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:46:34.622613    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:46:47.553816 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:46:47.553721    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"]
Sep 10 14:46:48.602685 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:46:48.597414    2547 scope.go:117] "RemoveContainer" containerID="d2c1393b630b45e96d89fc2c271974987f7ed4317f68e5369af55ac6a0d270f1"
Sep 10 14:46:48.610616 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:46:48.610591    2547 scope.go:117] "RemoveContainer" containerID="b4a7fe77397bda021f219cfd669d0e41dff2bf92f773b3adde268e33daae0b4f"
Sep 10 14:46:55.118715 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:46:55.118651    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:47:02.573287 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:02.573221    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt" podUID="40de8b4b-aca8-4f15-ba77-0f922b075063" containerName="main" containerID="cri-o://c4f9bb2e15a1a278164db5fe3ff5ad25c334149aa0549d24196e9175c4411378" gracePeriod=45
Sep 10 14:47:02.830978 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:02.830914    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:47:02.966812 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:02.966776    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40de8b4b-aca8-4f15-ba77-0f922b075063-dshm\") pod \"40de8b4b-aca8-4f15-ba77-0f922b075063\" (UID: \"40de8b4b-aca8-4f15-ba77-0f922b075063\") "
Sep 10 14:47:02.967041 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:02.966824    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvg7g\" (UniqueName: \"kubernetes.io/projected/40de8b4b-aca8-4f15-ba77-0f922b075063-kube-api-access-nvg7g\") pod \"40de8b4b-aca8-4f15-ba77-0f922b075063\" (UID: \"40de8b4b-aca8-4f15-ba77-0f922b075063\") "
Sep 10 14:47:02.967041 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:02.966841    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40de8b4b-aca8-4f15-ba77-0f922b075063-home\") pod \"40de8b4b-aca8-4f15-ba77-0f922b075063\" (UID: \"40de8b4b-aca8-4f15-ba77-0f922b075063\") "
Sep 10 14:47:02.967041 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:02.966869    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40de8b4b-aca8-4f15-ba77-0f922b075063-tmp-dir\") pod \"40de8b4b-aca8-4f15-ba77-0f922b075063\" (UID: \"40de8b4b-aca8-4f15-ba77-0f922b075063\") "
Sep 10 14:47:02.967041 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:02.966917    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40de8b4b-aca8-4f15-ba77-0f922b075063-model-cache\") pod \"40de8b4b-aca8-4f15-ba77-0f922b075063\" (UID: \"40de8b4b-aca8-4f15-ba77-0f922b075063\") "
Sep 10 14:47:02.967041 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:02.966949    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40de8b4b-aca8-4f15-ba77-0f922b075063-tls-certs\") pod \"40de8b4b-aca8-4f15-ba77-0f922b075063\" (UID: \"40de8b4b-aca8-4f15-ba77-0f922b075063\") "
Sep 10 14:47:02.967380 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:02.967153    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40de8b4b-aca8-4f15-ba77-0f922b075063-home" (OuterVolumeSpecName: "home") pod "40de8b4b-aca8-4f15-ba77-0f922b075063" (UID: "40de8b4b-aca8-4f15-ba77-0f922b075063"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:47:02.967380 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:02.967278    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40de8b4b-aca8-4f15-ba77-0f922b075063-home\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:47:02.967488 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:02.967376    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40de8b4b-aca8-4f15-ba77-0f922b075063-model-cache" (OuterVolumeSpecName: "model-cache") pod "40de8b4b-aca8-4f15-ba77-0f922b075063" (UID: "40de8b4b-aca8-4f15-ba77-0f922b075063"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:47:02.967488 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:02.967446    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40de8b4b-aca8-4f15-ba77-0f922b075063-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "40de8b4b-aca8-4f15-ba77-0f922b075063" (UID: "40de8b4b-aca8-4f15-ba77-0f922b075063"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:47:02.969506 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:02.969480    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40de8b4b-aca8-4f15-ba77-0f922b075063-kube-api-access-nvg7g" (OuterVolumeSpecName: "kube-api-access-nvg7g") pod "40de8b4b-aca8-4f15-ba77-0f922b075063" (UID: "40de8b4b-aca8-4f15-ba77-0f922b075063"). InnerVolumeSpecName "kube-api-access-nvg7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:47:02.969636 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:02.969621    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40de8b4b-aca8-4f15-ba77-0f922b075063-dshm" (OuterVolumeSpecName: "dshm") pod "40de8b4b-aca8-4f15-ba77-0f922b075063" (UID: "40de8b4b-aca8-4f15-ba77-0f922b075063"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:47:02.969690 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:02.969641    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40de8b4b-aca8-4f15-ba77-0f922b075063-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40de8b4b-aca8-4f15-ba77-0f922b075063" (UID: "40de8b4b-aca8-4f15-ba77-0f922b075063"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:47:03.068101 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:03.068067    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40de8b4b-aca8-4f15-ba77-0f922b075063-dshm\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:47:03.068101 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:03.068095    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvg7g\" (UniqueName: \"kubernetes.io/projected/40de8b4b-aca8-4f15-ba77-0f922b075063-kube-api-access-nvg7g\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:47:03.068101 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:03.068105    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40de8b4b-aca8-4f15-ba77-0f922b075063-tmp-dir\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:47:03.068344 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:03.068114    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40de8b4b-aca8-4f15-ba77-0f922b075063-model-cache\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:47:03.068344 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:03.068122    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40de8b4b-aca8-4f15-ba77-0f922b075063-tls-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:47:03.353798 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:03.353767    2547 generic.go:356] "Generic (PLEG): container finished" podID="40de8b4b-aca8-4f15-ba77-0f922b075063" containerID="c4f9bb2e15a1a278164db5fe3ff5ad25c334149aa0549d24196e9175c4411378" exitCode=0
Sep 10 14:47:03.353971 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:03.353812    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"
Sep 10 14:47:03.353971 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:03.353854    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt" event={"ID":"40de8b4b-aca8-4f15-ba77-0f922b075063","Type":"ContainerDied","Data":"c4f9bb2e15a1a278164db5fe3ff5ad25c334149aa0549d24196e9175c4411378"}
Sep 10 14:47:03.353971 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:03.353900    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt" event={"ID":"40de8b4b-aca8-4f15-ba77-0f922b075063","Type":"ContainerDied","Data":"b18592fa0c76dd43fcf1cf673c0d31be293ac14043245e492418ac6d01efb461"}
Sep 10 14:47:03.353971 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:03.353919    2547 scope.go:117] "RemoveContainer" containerID="c4f9bb2e15a1a278164db5fe3ff5ad25c334149aa0549d24196e9175c4411378"
Sep 10 14:47:03.363143 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:03.363126    2547 scope.go:117] "RemoveContainer" containerID="c4f9bb2e15a1a278164db5fe3ff5ad25c334149aa0549d24196e9175c4411378"
Sep 10 14:47:03.363422 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:47:03.363402    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f9bb2e15a1a278164db5fe3ff5ad25c334149aa0549d24196e9175c4411378\": container with ID starting with c4f9bb2e15a1a278164db5fe3ff5ad25c334149aa0549d24196e9175c4411378 not found: ID does not exist" containerID="c4f9bb2e15a1a278164db5fe3ff5ad25c334149aa0549d24196e9175c4411378"
Sep 10 14:47:03.363475 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:03.363431    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f9bb2e15a1a278164db5fe3ff5ad25c334149aa0549d24196e9175c4411378"} err="failed to get container status \"c4f9bb2e15a1a278164db5fe3ff5ad25c334149aa0549d24196e9175c4411378\": rpc error: code = NotFound desc = could not find container \"c4f9bb2e15a1a278164db5fe3ff5ad25c334149aa0549d24196e9175c4411378\": container with ID starting with c4f9bb2e15a1a278164db5fe3ff5ad25c334149aa0549d24196e9175c4411378 not found: ID does not exist"
Sep 10 14:47:03.375479 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:03.375456    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"]
Sep 10 14:47:03.378338 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:03.378317    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-dfd05fbc/e2e-lifecycle-model-dfd05fbc-kserve-7f6bb69855-qmjjt"]
Sep 10 14:47:04.421845 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:04.421808    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40de8b4b-aca8-4f15-ba77-0f922b075063" path="/var/lib/kubelet/pods/40de8b4b-aca8-4f15-ba77-0f922b075063/volumes"
Sep 10 14:47:11.230356 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:11.230319    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:47:14.270142 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:14.270105    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7700-5c5a04"]
Sep 10 14:47:14.270523 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:14.270508    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40de8b4b-aca8-4f15-ba77-0f922b075063" containerName="main"
Sep 10 14:47:14.270571 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:14.270524    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="40de8b4b-aca8-4f15-ba77-0f922b075063" containerName="main"
Sep 10 14:47:14.270604 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:14.270586    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="40de8b4b-aca8-4f15-ba77-0f922b075063" containerName="main"
Sep 10 14:47:14.274847 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:14.274830    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-5c5a04"
Sep 10 14:47:14.277478 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:14.277454    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4cjjz\""
Sep 10 14:47:14.280080 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:14.280055    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7700-5c5a04"]
Sep 10 14:47:14.373828 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:14.373780    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6rqv\" (UniqueName: \"kubernetes.io/projected/96e92ea8-71a9-41e8-9cb2-fa1b53e71834-kube-api-access-q6rqv\") pod \"test-curl-7700-5c5a04\" (UID: \"96e92ea8-71a9-41e8-9cb2-fa1b53e71834\") " pod="opendatahub/test-curl-7700-5c5a04"
Sep 10 14:47:14.475241 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:14.475203    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6rqv\" (UniqueName: \"kubernetes.io/projected/96e92ea8-71a9-41e8-9cb2-fa1b53e71834-kube-api-access-q6rqv\") pod \"test-curl-7700-5c5a04\" (UID: \"96e92ea8-71a9-41e8-9cb2-fa1b53e71834\") " pod="opendatahub/test-curl-7700-5c5a04"
Sep 10 14:47:14.483929 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:14.483903    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6rqv\" (UniqueName: \"kubernetes.io/projected/96e92ea8-71a9-41e8-9cb2-fa1b53e71834-kube-api-access-q6rqv\") pod \"test-curl-7700-5c5a04\" (UID: \"96e92ea8-71a9-41e8-9cb2-fa1b53e71834\") " pod="opendatahub/test-curl-7700-5c5a04"
Sep 10 14:47:14.585956 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:14.585873    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-5c5a04"
Sep 10 14:47:14.854746 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:14.854715    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7700-5c5a04"]
Sep 10 14:47:14.856299 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:47:14.856273    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96e92ea8_71a9_41e8_9cb2_fa1b53e71834.slice/crio-ace3239a941319c4dd868917efa54d2256cfd0e3d88e82cf5c1b71af53826b8e WatchSource:0}: Error finding container ace3239a941319c4dd868917efa54d2256cfd0e3d88e82cf5c1b71af53826b8e: Status 404 returned error can't find the container with id ace3239a941319c4dd868917efa54d2256cfd0e3d88e82cf5c1b71af53826b8e
Sep 10 14:47:15.433816 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:15.433780    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7700-5c5a04" event={"ID":"96e92ea8-71a9-41e8-9cb2-fa1b53e71834","Type":"ContainerStarted","Data":"6a5ccb3539e97b46fa08ac793f24e36753f2f910338c16230aae7545321d4028"}
Sep 10 14:47:15.434213 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:15.433826    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7700-5c5a04" event={"ID":"96e92ea8-71a9-41e8-9cb2-fa1b53e71834","Type":"ContainerStarted","Data":"ace3239a941319c4dd868917efa54d2256cfd0e3d88e82cf5c1b71af53826b8e"}
Sep 10 14:47:15.448402 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:15.448353    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7700-5c5a04" podStartSLOduration=1.448337747 podStartE2EDuration="1.448337747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:15.448108918 +0000 UTC m=+1647.489983399" watchObservedRunningTime="2026-09-10 14:47:15.448337747 +0000 UTC m=+1647.490212230"
Sep 10 14:47:16.036692 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:16.036656    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7700-5c5a04"]
Sep 10 14:47:16.043592 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:16.043564    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7700-5c5a04"]
Sep 10 14:47:16.438754 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:47:16.438722    2547 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7700-5c5a04" container="test-curl-7700-5c5a04"
Sep 10 14:47:17.400432 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:17.400392    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"]
Sep 10 14:47:17.444209 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:17.444173    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7700-5c5a04" podUID="96e92ea8-71a9-41e8-9cb2-fa1b53e71834" containerName="test-curl-7700-5c5a04" containerID="cri-o://6a5ccb3539e97b46fa08ac793f24e36753f2f910338c16230aae7545321d4028" gracePeriod=2
Sep 10 14:47:18.514548 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:18.514515    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"]
Sep 10 14:47:19.603994 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:19.603967    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-5c5a04"
Sep 10 14:47:19.631180 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:19.631152    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6rqv\" (UniqueName: \"kubernetes.io/projected/96e92ea8-71a9-41e8-9cb2-fa1b53e71834-kube-api-access-q6rqv\") pod \"96e92ea8-71a9-41e8-9cb2-fa1b53e71834\" (UID: \"96e92ea8-71a9-41e8-9cb2-fa1b53e71834\") "
Sep 10 14:47:19.633148 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:19.633113    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e92ea8-71a9-41e8-9cb2-fa1b53e71834-kube-api-access-q6rqv" (OuterVolumeSpecName: "kube-api-access-q6rqv") pod "96e92ea8-71a9-41e8-9cb2-fa1b53e71834" (UID: "96e92ea8-71a9-41e8-9cb2-fa1b53e71834"). InnerVolumeSpecName "kube-api-access-q6rqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:47:19.732046 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:19.731949    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6rqv\" (UniqueName: \"kubernetes.io/projected/96e92ea8-71a9-41e8-9cb2-fa1b53e71834-kube-api-access-q6rqv\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:47:20.421447 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:20.421413    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96e92ea8-71a9-41e8-9cb2-fa1b53e71834" path="/var/lib/kubelet/pods/96e92ea8-71a9-41e8-9cb2-fa1b53e71834/volumes"
Sep 10 14:47:20.462567 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:20.462536    2547 generic.go:356] "Generic (PLEG): container finished" podID="96e92ea8-71a9-41e8-9cb2-fa1b53e71834" containerID="6a5ccb3539e97b46fa08ac793f24e36753f2f910338c16230aae7545321d4028" exitCode=137
Sep 10 14:47:20.462739 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:20.462574    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-5c5a04"
Sep 10 14:47:20.462739 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:20.462628    2547 scope.go:117] "RemoveContainer" containerID="6a5ccb3539e97b46fa08ac793f24e36753f2f910338c16230aae7545321d4028"
Sep 10 14:47:20.471456 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:20.471428    2547 scope.go:117] "RemoveContainer" containerID="6a5ccb3539e97b46fa08ac793f24e36753f2f910338c16230aae7545321d4028"
Sep 10 14:47:20.471738 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:47:20.471720    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a5ccb3539e97b46fa08ac793f24e36753f2f910338c16230aae7545321d4028\": container with ID starting with 6a5ccb3539e97b46fa08ac793f24e36753f2f910338c16230aae7545321d4028 not found: ID does not exist" containerID="6a5ccb3539e97b46fa08ac793f24e36753f2f910338c16230aae7545321d4028"
Sep 10 14:47:20.471800 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:20.471746    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a5ccb3539e97b46fa08ac793f24e36753f2f910338c16230aae7545321d4028"} err="failed to get container status \"6a5ccb3539e97b46fa08ac793f24e36753f2f910338c16230aae7545321d4028\": rpc error: code = NotFound desc = could not find container \"6a5ccb3539e97b46fa08ac793f24e36753f2f910338c16230aae7545321d4028\": container with ID starting with 6a5ccb3539e97b46fa08ac793f24e36753f2f910338c16230aae7545321d4028 not found: ID does not exist"
Sep 10 14:47:22.923019 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:22.922982    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:47:31.925887 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:31.925841    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"]
Sep 10 14:47:31.926289 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:31.926235    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96e92ea8-71a9-41e8-9cb2-fa1b53e71834" containerName="test-curl-7700-5c5a04"
Sep 10 14:47:31.926289 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:31.926247    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="96e92ea8-71a9-41e8-9cb2-fa1b53e71834" containerName="test-curl-7700-5c5a04"
Sep 10 14:47:31.926367 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:31.926303    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="96e92ea8-71a9-41e8-9cb2-fa1b53e71834" containerName="test-curl-7700-5c5a04"
Sep 10 14:47:31.930670 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:31.930647    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:47:31.933408 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:31.933384    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-85d86b\"/\"sub-test-model-85d86b-kserve-self-signed-certs\""
Sep 10 14:47:31.941593 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:31.941540    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"]
Sep 10 14:47:32.046537 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.046497    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d904258-3797-4239-81e8-2a3488fa9436-tls-certs\") pod \"sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb\" (UID: \"6d904258-3797-4239-81e8-2a3488fa9436\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:47:32.046537 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.046536    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d904258-3797-4239-81e8-2a3488fa9436-dshm\") pod \"sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb\" (UID: \"6d904258-3797-4239-81e8-2a3488fa9436\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:47:32.046780 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.046636    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d904258-3797-4239-81e8-2a3488fa9436-model-cache\") pod \"sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb\" (UID: \"6d904258-3797-4239-81e8-2a3488fa9436\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:47:32.046780 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.046680    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6j6k\" (UniqueName: \"kubernetes.io/projected/6d904258-3797-4239-81e8-2a3488fa9436-kube-api-access-v6j6k\") pod \"sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb\" (UID: \"6d904258-3797-4239-81e8-2a3488fa9436\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:47:32.046780 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.046728    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d904258-3797-4239-81e8-2a3488fa9436-home\") pod \"sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb\" (UID: \"6d904258-3797-4239-81e8-2a3488fa9436\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:47:32.046780 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.046775    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d904258-3797-4239-81e8-2a3488fa9436-tmp-dir\") pod \"sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb\" (UID: \"6d904258-3797-4239-81e8-2a3488fa9436\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:47:32.147347 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.147306    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d904258-3797-4239-81e8-2a3488fa9436-tmp-dir\") pod \"sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb\" (UID: \"6d904258-3797-4239-81e8-2a3488fa9436\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:47:32.147539 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.147361    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d904258-3797-4239-81e8-2a3488fa9436-tls-certs\") pod \"sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb\" (UID: \"6d904258-3797-4239-81e8-2a3488fa9436\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:47:32.147539 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.147378    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d904258-3797-4239-81e8-2a3488fa9436-dshm\") pod \"sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb\" (UID: \"6d904258-3797-4239-81e8-2a3488fa9436\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:47:32.147539 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.147423    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d904258-3797-4239-81e8-2a3488fa9436-model-cache\") pod \"sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb\" (UID: \"6d904258-3797-4239-81e8-2a3488fa9436\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:47:32.147539 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.147449    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6j6k\" (UniqueName: \"kubernetes.io/projected/6d904258-3797-4239-81e8-2a3488fa9436-kube-api-access-v6j6k\") pod \"sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb\" (UID: \"6d904258-3797-4239-81e8-2a3488fa9436\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:47:32.147539 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.147478    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d904258-3797-4239-81e8-2a3488fa9436-home\") pod \"sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb\" (UID: \"6d904258-3797-4239-81e8-2a3488fa9436\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:47:32.147848 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.147823    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d904258-3797-4239-81e8-2a3488fa9436-tmp-dir\") pod \"sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb\" (UID: \"6d904258-3797-4239-81e8-2a3488fa9436\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:47:32.147890 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.147875    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d904258-3797-4239-81e8-2a3488fa9436-model-cache\") pod \"sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb\" (UID: \"6d904258-3797-4239-81e8-2a3488fa9436\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:47:32.147967 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.147943    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d904258-3797-4239-81e8-2a3488fa9436-home\") pod \"sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb\" (UID: \"6d904258-3797-4239-81e8-2a3488fa9436\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:47:32.149833 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.149812    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d904258-3797-4239-81e8-2a3488fa9436-dshm\") pod \"sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb\" (UID: \"6d904258-3797-4239-81e8-2a3488fa9436\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:47:32.149974 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.149955    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d904258-3797-4239-81e8-2a3488fa9436-tls-certs\") pod \"sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb\" (UID: \"6d904258-3797-4239-81e8-2a3488fa9436\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:47:32.156162 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.156141    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6j6k\" (UniqueName: \"kubernetes.io/projected/6d904258-3797-4239-81e8-2a3488fa9436-kube-api-access-v6j6k\") pod \"sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb\" (UID: \"6d904258-3797-4239-81e8-2a3488fa9436\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:47:32.241767 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.241655    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:47:32.420516 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.420476    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5" podUID="8283c96b-6243-4e5f-9116-e64bbe65b4c9" containerName="main" containerID="cri-o://ca98e534c8f32589d396a270436678cdfdc6d5551b0b2705e14fa04d69d72cbf" gracePeriod=45
Sep 10 14:47:32.519277 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.517244    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"]
Sep 10 14:47:32.519356 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:47:32.519330    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d904258_3797_4239_81e8_2a3488fa9436.slice/crio-8048314fb1b5710af58c4742cc4d8f408eed7ea56c5ca167061ccfe227a3c4d8 WatchSource:0}: Error finding container 8048314fb1b5710af58c4742cc4d8f408eed7ea56c5ca167061ccfe227a3c4d8: Status 404 returned error can't find the container with id 8048314fb1b5710af58c4742cc4d8f408eed7ea56c5ca167061ccfe227a3c4d8
Sep 10 14:47:32.538439 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.538410    2547 generic.go:356] "Generic (PLEG): container finished" podID="8283c96b-6243-4e5f-9116-e64bbe65b4c9" containerID="ca98e534c8f32589d396a270436678cdfdc6d5551b0b2705e14fa04d69d72cbf" exitCode=0
Sep 10 14:47:32.538586 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.538489    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5" event={"ID":"8283c96b-6243-4e5f-9116-e64bbe65b4c9","Type":"ContainerDied","Data":"ca98e534c8f32589d396a270436678cdfdc6d5551b0b2705e14fa04d69d72cbf"}
Sep 10 14:47:32.539946 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.539922    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb" event={"ID":"6d904258-3797-4239-81e8-2a3488fa9436","Type":"ContainerStarted","Data":"8048314fb1b5710af58c4742cc4d8f408eed7ea56c5ca167061ccfe227a3c4d8"}
Sep 10 14:47:32.649398 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.649371    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:47:32.752164 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.752133    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8283c96b-6243-4e5f-9116-e64bbe65b4c9-home\") pod \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\" (UID: \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\") "
Sep 10 14:47:32.752361 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.752226    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8283c96b-6243-4e5f-9116-e64bbe65b4c9-model-cache\") pod \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\" (UID: \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\") "
Sep 10 14:47:32.752361 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.752259    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8283c96b-6243-4e5f-9116-e64bbe65b4c9-tmp-dir\") pod \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\" (UID: \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\") "
Sep 10 14:47:32.752361 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.752298    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2dks\" (UniqueName: \"kubernetes.io/projected/8283c96b-6243-4e5f-9116-e64bbe65b4c9-kube-api-access-n2dks\") pod \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\" (UID: \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\") "
Sep 10 14:47:32.752361 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.752329    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8283c96b-6243-4e5f-9116-e64bbe65b4c9-dshm\") pod \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\" (UID: \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\") "
Sep 10 14:47:32.752581 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.752417    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8283c96b-6243-4e5f-9116-e64bbe65b4c9-home" (OuterVolumeSpecName: "home") pod "8283c96b-6243-4e5f-9116-e64bbe65b4c9" (UID: "8283c96b-6243-4e5f-9116-e64bbe65b4c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:47:32.752581 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.752444    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8283c96b-6243-4e5f-9116-e64bbe65b4c9-tls-certs\") pod \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\" (UID: \"8283c96b-6243-4e5f-9116-e64bbe65b4c9\") "
Sep 10 14:47:32.752581 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.752498    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8283c96b-6243-4e5f-9116-e64bbe65b4c9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8283c96b-6243-4e5f-9116-e64bbe65b4c9" (UID: "8283c96b-6243-4e5f-9116-e64bbe65b4c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:47:32.752581 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.752531    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8283c96b-6243-4e5f-9116-e64bbe65b4c9-model-cache" (OuterVolumeSpecName: "model-cache") pod "8283c96b-6243-4e5f-9116-e64bbe65b4c9" (UID: "8283c96b-6243-4e5f-9116-e64bbe65b4c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:47:32.752806 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.752779    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8283c96b-6243-4e5f-9116-e64bbe65b4c9-home\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:47:32.752806 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.752794    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8283c96b-6243-4e5f-9116-e64bbe65b4c9-model-cache\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:47:32.752912 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.752808    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8283c96b-6243-4e5f-9116-e64bbe65b4c9-tmp-dir\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:47:32.754474 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.754446    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8283c96b-6243-4e5f-9116-e64bbe65b4c9-dshm" (OuterVolumeSpecName: "dshm") pod "8283c96b-6243-4e5f-9116-e64bbe65b4c9" (UID: "8283c96b-6243-4e5f-9116-e64bbe65b4c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:47:32.754587 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.754501    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8283c96b-6243-4e5f-9116-e64bbe65b4c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8283c96b-6243-4e5f-9116-e64bbe65b4c9" (UID: "8283c96b-6243-4e5f-9116-e64bbe65b4c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:47:32.754587 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.754548    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8283c96b-6243-4e5f-9116-e64bbe65b4c9-kube-api-access-n2dks" (OuterVolumeSpecName: "kube-api-access-n2dks") pod "8283c96b-6243-4e5f-9116-e64bbe65b4c9" (UID: "8283c96b-6243-4e5f-9116-e64bbe65b4c9"). InnerVolumeSpecName "kube-api-access-n2dks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:47:32.853347 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.853263    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8283c96b-6243-4e5f-9116-e64bbe65b4c9-dshm\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:47:32.853347 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.853292    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8283c96b-6243-4e5f-9116-e64bbe65b4c9-tls-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:47:32.853347 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:32.853303    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2dks\" (UniqueName: \"kubernetes.io/projected/8283c96b-6243-4e5f-9116-e64bbe65b4c9-kube-api-access-n2dks\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:47:33.533998 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.533959    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt" podUID="0381c71b-7613-461f-bcfd-9d19e14fe4d5" containerName="main" containerID="cri-o://31c5792b62f23f223717748ef78a58f54025c2eb39d2998ab01ef812dbeedd73" gracePeriod=45
Sep 10 14:47:33.546622 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.546603    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"
Sep 10 14:47:33.546719 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.546619    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5" event={"ID":"8283c96b-6243-4e5f-9116-e64bbe65b4c9","Type":"ContainerDied","Data":"96269ce8f6510fa9ca13836619d134c22865c20435490ffb05180b453560469b"}
Sep 10 14:47:33.546719 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.546671    2547 scope.go:117] "RemoveContainer" containerID="ca98e534c8f32589d396a270436678cdfdc6d5551b0b2705e14fa04d69d72cbf"
Sep 10 14:47:33.548467 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.548446    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb" event={"ID":"6d904258-3797-4239-81e8-2a3488fa9436","Type":"ContainerStarted","Data":"443e30b9df9e89af556bf0d8907bc4b9a3103bc6eb5a49bfa8177b4e6099cb25"}
Sep 10 14:47:33.567060 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.567011    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb" podStartSLOduration=2.566998222 podStartE2EDuration="2.566998222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:33.565683464 +0000 UTC m=+1665.607557946" watchObservedRunningTime="2026-09-10 14:47:33.566998222 +0000 UTC m=+1665.608872703"
Sep 10 14:47:33.603707 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.603671    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"]
Sep 10 14:47:33.607583 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.607557    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-85d86b/auth-test-model-85d86b-kserve-674d4c7d55-sjdn5"]
Sep 10 14:47:33.795076 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.795019    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:47:33.861562 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.861531    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0381c71b-7613-461f-bcfd-9d19e14fe4d5-home\") pod \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\" (UID: \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\") "
Sep 10 14:47:33.861781 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.861591    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0381c71b-7613-461f-bcfd-9d19e14fe4d5-tmp-dir\") pod \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\" (UID: \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\") "
Sep 10 14:47:33.861781 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.861643    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wncsl\" (UniqueName: \"kubernetes.io/projected/0381c71b-7613-461f-bcfd-9d19e14fe4d5-kube-api-access-wncsl\") pod \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\" (UID: \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\") "
Sep 10 14:47:33.861781 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.861682    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0381c71b-7613-461f-bcfd-9d19e14fe4d5-model-cache\") pod \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\" (UID: \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\") "
Sep 10 14:47:33.861781 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.861730    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0381c71b-7613-461f-bcfd-9d19e14fe4d5-dshm\") pod \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\" (UID: \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\") "
Sep 10 14:47:33.861781 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.861766    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0381c71b-7613-461f-bcfd-9d19e14fe4d5-tls-certs\") pod \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\" (UID: \"0381c71b-7613-461f-bcfd-9d19e14fe4d5\") "
Sep 10 14:47:33.862053 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.861879    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0381c71b-7613-461f-bcfd-9d19e14fe4d5-home" (OuterVolumeSpecName: "home") pod "0381c71b-7613-461f-bcfd-9d19e14fe4d5" (UID: "0381c71b-7613-461f-bcfd-9d19e14fe4d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:47:33.862053 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.861927    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0381c71b-7613-461f-bcfd-9d19e14fe4d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0381c71b-7613-461f-bcfd-9d19e14fe4d5" (UID: "0381c71b-7613-461f-bcfd-9d19e14fe4d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:47:33.862053 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.862011    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0381c71b-7613-461f-bcfd-9d19e14fe4d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "0381c71b-7613-461f-bcfd-9d19e14fe4d5" (UID: "0381c71b-7613-461f-bcfd-9d19e14fe4d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:47:33.862171 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.862133    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0381c71b-7613-461f-bcfd-9d19e14fe4d5-model-cache\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:47:33.862171 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.862150    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0381c71b-7613-461f-bcfd-9d19e14fe4d5-home\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:47:33.862171 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.862162    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0381c71b-7613-461f-bcfd-9d19e14fe4d5-tmp-dir\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:47:33.864004 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.863981    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0381c71b-7613-461f-bcfd-9d19e14fe4d5-dshm" (OuterVolumeSpecName: "dshm") pod "0381c71b-7613-461f-bcfd-9d19e14fe4d5" (UID: "0381c71b-7613-461f-bcfd-9d19e14fe4d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:47:33.864082 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.864060    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0381c71b-7613-461f-bcfd-9d19e14fe4d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0381c71b-7613-461f-bcfd-9d19e14fe4d5" (UID: "0381c71b-7613-461f-bcfd-9d19e14fe4d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:47:33.864125 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.864077    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0381c71b-7613-461f-bcfd-9d19e14fe4d5-kube-api-access-wncsl" (OuterVolumeSpecName: "kube-api-access-wncsl") pod "0381c71b-7613-461f-bcfd-9d19e14fe4d5" (UID: "0381c71b-7613-461f-bcfd-9d19e14fe4d5"). InnerVolumeSpecName "kube-api-access-wncsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:47:33.963483 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.963450    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wncsl\" (UniqueName: \"kubernetes.io/projected/0381c71b-7613-461f-bcfd-9d19e14fe4d5-kube-api-access-wncsl\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:47:33.963483 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.963477    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0381c71b-7613-461f-bcfd-9d19e14fe4d5-dshm\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:47:33.963483 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:33.963488    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0381c71b-7613-461f-bcfd-9d19e14fe4d5-tls-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:47:34.421746 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:34.421683    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8283c96b-6243-4e5f-9116-e64bbe65b4c9" path="/var/lib/kubelet/pods/8283c96b-6243-4e5f-9116-e64bbe65b4c9/volumes"
Sep 10 14:47:34.555894 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:34.555864    2547 generic.go:356] "Generic (PLEG): container finished" podID="0381c71b-7613-461f-bcfd-9d19e14fe4d5" containerID="31c5792b62f23f223717748ef78a58f54025c2eb39d2998ab01ef812dbeedd73" exitCode=0
Sep 10 14:47:34.556306 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:34.555910    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"
Sep 10 14:47:34.556306 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:34.555957    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt" event={"ID":"0381c71b-7613-461f-bcfd-9d19e14fe4d5","Type":"ContainerDied","Data":"31c5792b62f23f223717748ef78a58f54025c2eb39d2998ab01ef812dbeedd73"}
Sep 10 14:47:34.556306 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:34.555993    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt" event={"ID":"0381c71b-7613-461f-bcfd-9d19e14fe4d5","Type":"ContainerDied","Data":"4f1e992c30522742db77338e6e917c5f56d11cd764deedc687f3693c3d4ab33b"}
Sep 10 14:47:34.556306 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:34.556015    2547 scope.go:117] "RemoveContainer" containerID="31c5792b62f23f223717748ef78a58f54025c2eb39d2998ab01ef812dbeedd73"
Sep 10 14:47:34.565321 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:34.565292    2547 scope.go:117] "RemoveContainer" containerID="31c5792b62f23f223717748ef78a58f54025c2eb39d2998ab01ef812dbeedd73"
Sep 10 14:47:34.565565 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:47:34.565548    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31c5792b62f23f223717748ef78a58f54025c2eb39d2998ab01ef812dbeedd73\": container with ID starting with 31c5792b62f23f223717748ef78a58f54025c2eb39d2998ab01ef812dbeedd73 not found: ID does not exist" containerID="31c5792b62f23f223717748ef78a58f54025c2eb39d2998ab01ef812dbeedd73"
Sep 10 14:47:34.565622 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:34.565572    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31c5792b62f23f223717748ef78a58f54025c2eb39d2998ab01ef812dbeedd73"} err="failed to get container status \"31c5792b62f23f223717748ef78a58f54025c2eb39d2998ab01ef812dbeedd73\": rpc error: code = NotFound desc = could not find container \"31c5792b62f23f223717748ef78a58f54025c2eb39d2998ab01ef812dbeedd73\": container with ID starting with 31c5792b62f23f223717748ef78a58f54025c2eb39d2998ab01ef812dbeedd73 not found: ID does not exist"
Sep 10 14:47:34.576011 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:34.575985    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"]
Sep 10 14:47:34.579196 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:34.579176    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3477ac/auth-test-model-3477ac-kserve-b946b8b47-286xt"]
Sep 10 14:47:35.022230 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:35.022194    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:47:36.422105 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:36.422074    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0381c71b-7613-461f-bcfd-9d19e14fe4d5" path="/var/lib/kubelet/pods/0381c71b-7613-461f-bcfd-9d19e14fe4d5/volumes"
Sep 10 14:47:42.241969 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:42.241930    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:47:42.242416 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:42.242196    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:47:42.254450 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:42.254427    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:47:42.613566 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:42.613482    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:47:48.640453 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:48.640419    2547 scope.go:117] "RemoveContainer" containerID="7913c7568eed7dbd8e1098b40534135c3e07d310ba254930999c2a80e917ddfb"
Sep 10 14:47:54.848523 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:54.848486    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"]
Sep 10 14:47:54.849003 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:54.848896    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8283c96b-6243-4e5f-9116-e64bbe65b4c9" containerName="main"
Sep 10 14:47:54.849003 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:54.848909    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="8283c96b-6243-4e5f-9116-e64bbe65b4c9" containerName="main"
Sep 10 14:47:54.849003 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:54.848925    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0381c71b-7613-461f-bcfd-9d19e14fe4d5" containerName="main"
Sep 10 14:47:54.849003 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:54.848930    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="0381c71b-7613-461f-bcfd-9d19e14fe4d5" containerName="main"
Sep 10 14:47:54.849176 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:54.849051    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="8283c96b-6243-4e5f-9116-e64bbe65b4c9" containerName="main"
Sep 10 14:47:54.849176 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:54.849064    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="0381c71b-7613-461f-bcfd-9d19e14fe4d5" containerName="main"
Sep 10 14:47:54.852145 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:54.852128    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:47:54.855912 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:54.855887    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3477ac\"/\"kube-root-ca.crt\""
Sep 10 14:47:54.855912 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:54.855905    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3477ac\"/\"openshift-service-ca.crt\""
Sep 10 14:47:54.856090 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:54.855888    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3477ac\"/\"sub-test-model-3477ac-kserve-self-signed-certs\""
Sep 10 14:47:54.856090 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:54.855926    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3477ac\"/\"default-dockercfg-f4brn\""
Sep 10 14:47:54.867450 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:54.867429    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"]
Sep 10 14:47:54.961656 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:54.961618    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-dshm\") pod \"sub-test-model-3477ac-kserve-745fb87dc7-qp8kt\" (UID: \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:47:54.961656 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:54.961656    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-tls-certs\") pod \"sub-test-model-3477ac-kserve-745fb87dc7-qp8kt\" (UID: \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:47:54.961904 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:54.961678    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-home\") pod \"sub-test-model-3477ac-kserve-745fb87dc7-qp8kt\" (UID: \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:47:54.961904 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:54.961785    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-tmp-dir\") pod \"sub-test-model-3477ac-kserve-745fb87dc7-qp8kt\" (UID: \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:47:54.961904 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:54.961856    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l97kh\" (UniqueName: \"kubernetes.io/projected/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-kube-api-access-l97kh\") pod \"sub-test-model-3477ac-kserve-745fb87dc7-qp8kt\" (UID: \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:47:54.961904 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:54.961881    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-model-cache\") pod \"sub-test-model-3477ac-kserve-745fb87dc7-qp8kt\" (UID: \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:47:55.062469 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:55.062434    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l97kh\" (UniqueName: \"kubernetes.io/projected/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-kube-api-access-l97kh\") pod \"sub-test-model-3477ac-kserve-745fb87dc7-qp8kt\" (UID: \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:47:55.062655 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:55.062477    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-model-cache\") pod \"sub-test-model-3477ac-kserve-745fb87dc7-qp8kt\" (UID: \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:47:55.062655 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:55.062514    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-dshm\") pod \"sub-test-model-3477ac-kserve-745fb87dc7-qp8kt\" (UID: \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:47:55.062655 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:55.062531    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-tls-certs\") pod \"sub-test-model-3477ac-kserve-745fb87dc7-qp8kt\" (UID: \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:47:55.062655 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:55.062549    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-home\") pod \"sub-test-model-3477ac-kserve-745fb87dc7-qp8kt\" (UID: \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:47:55.062655 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:55.062595    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-tmp-dir\") pod \"sub-test-model-3477ac-kserve-745fb87dc7-qp8kt\" (UID: \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:47:55.063064 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:55.063038    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-home\") pod \"sub-test-model-3477ac-kserve-745fb87dc7-qp8kt\" (UID: \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:47:55.063129 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:55.063073    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-tmp-dir\") pod \"sub-test-model-3477ac-kserve-745fb87dc7-qp8kt\" (UID: \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:47:55.063129 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:55.063097    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-model-cache\") pod \"sub-test-model-3477ac-kserve-745fb87dc7-qp8kt\" (UID: \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:47:55.064843 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:55.064826    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-dshm\") pod \"sub-test-model-3477ac-kserve-745fb87dc7-qp8kt\" (UID: \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:47:55.065144 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:55.065125    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-tls-certs\") pod \"sub-test-model-3477ac-kserve-745fb87dc7-qp8kt\" (UID: \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:47:55.074631 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:55.074610    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l97kh\" (UniqueName: \"kubernetes.io/projected/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-kube-api-access-l97kh\") pod \"sub-test-model-3477ac-kserve-745fb87dc7-qp8kt\" (UID: \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:47:55.160924 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:55.160892    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:47:55.442878 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:55.442853    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"]
Sep 10 14:47:55.443354 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:47:55.443332    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f7b6bc0_f728_47f5_8fc6_3d3dbd2df4f5.slice/crio-425c31e678e0500526459c55f17417a594e48f6e88993ce492b8b9261a0bf9c4 WatchSource:0}: Error finding container 425c31e678e0500526459c55f17417a594e48f6e88993ce492b8b9261a0bf9c4: Status 404 returned error can't find the container with id 425c31e678e0500526459c55f17417a594e48f6e88993ce492b8b9261a0bf9c4
Sep 10 14:47:55.673141 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:55.673099    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt" event={"ID":"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5","Type":"ContainerStarted","Data":"1efab1a9f1c382ebd6935b2cb66c9904dcbf50333cff78cdd23919dc47f564e9"}
Sep 10 14:47:55.673322 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:55.673149    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt" event={"ID":"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5","Type":"ContainerStarted","Data":"425c31e678e0500526459c55f17417a594e48f6e88993ce492b8b9261a0bf9c4"}
Sep 10 14:47:55.690915 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:55.690809    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt" podStartSLOduration=1.69079288 podStartE2EDuration="1.69079288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:55.690480574 +0000 UTC m=+1687.732355058" watchObservedRunningTime="2026-09-10 14:47:55.69079288 +0000 UTC m=+1687.732667361"
Sep 10 14:47:58.724109 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:47:58.724071    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:48:05.161845 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:05.161807    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:48:05.161845 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:05.161857    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:48:05.174375 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:05.174352    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:48:05.746746 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:05.746716    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:48:21.822709 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:21.822655    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:48:34.050575 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:34.050541    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"]
Sep 10 14:48:34.054641 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:34.054615    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"
Sep 10 14:48:34.057015 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:34.056994    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9c318b-serving-cert\""
Sep 10 14:48:34.067231 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:34.067206    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"]
Sep 10 14:48:34.215210 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:34.215170    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/340a1e3b-febd-467d-a31f-c151e896192c-openshift-service-ca\") pod \"maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr\" (UID: \"340a1e3b-febd-467d-a31f-c151e896192c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"
Sep 10 14:48:34.215210 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:34.215213    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/340a1e3b-febd-467d-a31f-c151e896192c-maas-api-tls\") pod \"maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr\" (UID: \"340a1e3b-febd-467d-a31f-c151e896192c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"
Sep 10 14:48:34.215438 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:34.215238    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/340a1e3b-febd-467d-a31f-c151e896192c-metrics-certs\") pod \"maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr\" (UID: \"340a1e3b-febd-467d-a31f-c151e896192c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"
Sep 10 14:48:34.215438 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:34.215341    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7b6m\" (UniqueName: \"kubernetes.io/projected/340a1e3b-febd-467d-a31f-c151e896192c-kube-api-access-d7b6m\") pod \"maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr\" (UID: \"340a1e3b-febd-467d-a31f-c151e896192c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"
Sep 10 14:48:34.316449 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:34.316364    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/340a1e3b-febd-467d-a31f-c151e896192c-metrics-certs\") pod \"maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr\" (UID: \"340a1e3b-febd-467d-a31f-c151e896192c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"
Sep 10 14:48:34.316449 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:34.316426    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7b6m\" (UniqueName: \"kubernetes.io/projected/340a1e3b-febd-467d-a31f-c151e896192c-kube-api-access-d7b6m\") pod \"maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr\" (UID: \"340a1e3b-febd-467d-a31f-c151e896192c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"
Sep 10 14:48:34.316636 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:34.316492    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/340a1e3b-febd-467d-a31f-c151e896192c-openshift-service-ca\") pod \"maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr\" (UID: \"340a1e3b-febd-467d-a31f-c151e896192c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"
Sep 10 14:48:34.316636 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:34.316513    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/340a1e3b-febd-467d-a31f-c151e896192c-maas-api-tls\") pod \"maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr\" (UID: \"340a1e3b-febd-467d-a31f-c151e896192c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"
Sep 10 14:48:34.316636 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:48:34.316605    2547 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-serving-cert: secret "maas-api-e2e-api-a-9c318b-serving-cert" not found
Sep 10 14:48:34.316786 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:48:34.316666    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/340a1e3b-febd-467d-a31f-c151e896192c-maas-api-tls podName:340a1e3b-febd-467d-a31f-c151e896192c nodeName:}" failed. No retries permitted until 2026-09-10 14:48:34.816647553 +0000 UTC m=+1726.858522013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/340a1e3b-febd-467d-a31f-c151e896192c-maas-api-tls") pod "maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr" (UID: "340a1e3b-febd-467d-a31f-c151e896192c") : secret "maas-api-e2e-api-a-9c318b-serving-cert" not found
Sep 10 14:48:34.317173 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:34.317151    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/340a1e3b-febd-467d-a31f-c151e896192c-openshift-service-ca\") pod \"maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr\" (UID: \"340a1e3b-febd-467d-a31f-c151e896192c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"
Sep 10 14:48:34.318878 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:34.318860    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/340a1e3b-febd-467d-a31f-c151e896192c-metrics-certs\") pod \"maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr\" (UID: \"340a1e3b-febd-467d-a31f-c151e896192c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"
Sep 10 14:48:34.325798 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:34.325766    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7b6m\" (UniqueName: \"kubernetes.io/projected/340a1e3b-febd-467d-a31f-c151e896192c-kube-api-access-d7b6m\") pod \"maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr\" (UID: \"340a1e3b-febd-467d-a31f-c151e896192c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"
Sep 10 14:48:34.821436 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:34.821400    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/340a1e3b-febd-467d-a31f-c151e896192c-maas-api-tls\") pod \"maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr\" (UID: \"340a1e3b-febd-467d-a31f-c151e896192c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"
Sep 10 14:48:34.823962 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:34.823934    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/340a1e3b-febd-467d-a31f-c151e896192c-maas-api-tls\") pod \"maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr\" (UID: \"340a1e3b-febd-467d-a31f-c151e896192c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"
Sep 10 14:48:34.969099 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:34.969058    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"
Sep 10 14:48:35.240106 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:35.240069    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"]
Sep 10 14:48:35.241327 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:48:35.241297    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod340a1e3b_febd_467d_a31f_c151e896192c.slice/crio-43e5da9a60267f6cf4a12990b33ed1531e4c5e3174a49e056e7adb098c6c8bb4 WatchSource:0}: Error finding container 43e5da9a60267f6cf4a12990b33ed1531e4c5e3174a49e056e7adb098c6c8bb4: Status 404 returned error can't find the container with id 43e5da9a60267f6cf4a12990b33ed1531e4c5e3174a49e056e7adb098c6c8bb4
Sep 10 14:48:35.242648 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:35.242632    2547 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:48:35.901647 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:35.901617    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr" event={"ID":"340a1e3b-febd-467d-a31f-c151e896192c","Type":"ContainerStarted","Data":"43e5da9a60267f6cf4a12990b33ed1531e4c5e3174a49e056e7adb098c6c8bb4"}
Sep 10 14:48:36.909062 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:36.909020    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr" event={"ID":"340a1e3b-febd-467d-a31f-c151e896192c","Type":"ContainerStarted","Data":"4de4548d16cfb22cf1ae4a53abe25f650f124f5d1c283b7d2750114156daa687"}
Sep 10 14:48:36.909449 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:36.909126    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"
Sep 10 14:48:36.931163 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:36.931109    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr" podStartSLOduration=2.313383142 podStartE2EDuration="2.931095324s" totalImagesPullingTime="617.712182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:36.929301244 +0000 UTC m=+1728.971175725" watchObservedRunningTime="2026-09-10 14:48:36.931095324 +0000 UTC m=+1728.972970200"
Sep 10 14:48:39.527577 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:39.527542    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:48:42.921005 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:42.920974    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"
Sep 10 14:48:43.780599 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:43.780565    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7700-1001b5"]
Sep 10 14:48:43.784419 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:43.784399    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-1001b5"
Sep 10 14:48:43.786874 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:43.786844    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4cjjz\""
Sep 10 14:48:43.790863 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:43.790840    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7700-1001b5"]
Sep 10 14:48:43.807350 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:43.807319    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shqq6\" (UniqueName: \"kubernetes.io/projected/63837edd-54e2-4bea-9dc6-aa19f903aa32-kube-api-access-shqq6\") pod \"test-curl-7700-1001b5\" (UID: \"63837edd-54e2-4bea-9dc6-aa19f903aa32\") " pod="opendatahub/test-curl-7700-1001b5"
Sep 10 14:48:43.908551 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:43.908515    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shqq6\" (UniqueName: \"kubernetes.io/projected/63837edd-54e2-4bea-9dc6-aa19f903aa32-kube-api-access-shqq6\") pod \"test-curl-7700-1001b5\" (UID: \"63837edd-54e2-4bea-9dc6-aa19f903aa32\") " pod="opendatahub/test-curl-7700-1001b5"
Sep 10 14:48:43.917395 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:43.917361    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shqq6\" (UniqueName: \"kubernetes.io/projected/63837edd-54e2-4bea-9dc6-aa19f903aa32-kube-api-access-shqq6\") pod \"test-curl-7700-1001b5\" (UID: \"63837edd-54e2-4bea-9dc6-aa19f903aa32\") " pod="opendatahub/test-curl-7700-1001b5"
Sep 10 14:48:44.095336 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:44.095247    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-1001b5"
Sep 10 14:48:44.372989 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:44.372965    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7700-1001b5"]
Sep 10 14:48:44.374886 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:48:44.374856    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63837edd_54e2_4bea_9dc6_aa19f903aa32.slice/crio-2041561ba62ae03c8facd47684f560c2248ede3b4bbbda525d2f9ab9cb0f73c1 WatchSource:0}: Error finding container 2041561ba62ae03c8facd47684f560c2248ede3b4bbbda525d2f9ab9cb0f73c1: Status 404 returned error can't find the container with id 2041561ba62ae03c8facd47684f560c2248ede3b4bbbda525d2f9ab9cb0f73c1
Sep 10 14:48:44.958444 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:44.958408    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7700-1001b5" event={"ID":"63837edd-54e2-4bea-9dc6-aa19f903aa32","Type":"ContainerStarted","Data":"6133e031f40d4102d7dba4676973656d59b9e8e62ee227a1051b2acc190ad8b5"}
Sep 10 14:48:44.958444 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:44.958457    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7700-1001b5" event={"ID":"63837edd-54e2-4bea-9dc6-aa19f903aa32","Type":"ContainerStarted","Data":"2041561ba62ae03c8facd47684f560c2248ede3b4bbbda525d2f9ab9cb0f73c1"}
Sep 10 14:48:44.973405 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:44.973356    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7700-1001b5" podStartSLOduration=1.9733395919999999 podStartE2EDuration="1.973339592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:44.972122784 +0000 UTC m=+1737.013997267" watchObservedRunningTime="2026-09-10 14:48:44.973339592 +0000 UTC m=+1737.015214073"
Sep 10 14:48:45.643273 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:45.643239    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7700-1001b5"]
Sep 10 14:48:45.646829 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:45.646803    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7700-1001b5"]
Sep 10 14:48:45.760475 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:45.760394    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7700-a0981d"]
Sep 10 14:48:45.760806 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:45.760793    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63837edd-54e2-4bea-9dc6-aa19f903aa32" containerName="test-curl-7700-1001b5"
Sep 10 14:48:45.760849 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:45.760807    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="63837edd-54e2-4bea-9dc6-aa19f903aa32" containerName="test-curl-7700-1001b5"
Sep 10 14:48:45.760888 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:45.760873    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="63837edd-54e2-4bea-9dc6-aa19f903aa32" containerName="test-curl-7700-1001b5"
Sep 10 14:48:45.763882 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:45.763865    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-a0981d"
Sep 10 14:48:45.768094 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:45.768069    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7700-a0981d"]
Sep 10 14:48:45.825751 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:45.825713    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw9vp\" (UniqueName: \"kubernetes.io/projected/8c2777c7-5db2-4b6e-8ec9-eb21976374c0-kube-api-access-fw9vp\") pod \"test-curl-7700-a0981d\" (UID: \"8c2777c7-5db2-4b6e-8ec9-eb21976374c0\") " pod="opendatahub/test-curl-7700-a0981d"
Sep 10 14:48:45.926931 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:45.926889    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw9vp\" (UniqueName: \"kubernetes.io/projected/8c2777c7-5db2-4b6e-8ec9-eb21976374c0-kube-api-access-fw9vp\") pod \"test-curl-7700-a0981d\" (UID: \"8c2777c7-5db2-4b6e-8ec9-eb21976374c0\") " pod="opendatahub/test-curl-7700-a0981d"
Sep 10 14:48:45.934999 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:45.934968    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw9vp\" (UniqueName: \"kubernetes.io/projected/8c2777c7-5db2-4b6e-8ec9-eb21976374c0-kube-api-access-fw9vp\") pod \"test-curl-7700-a0981d\" (UID: \"8c2777c7-5db2-4b6e-8ec9-eb21976374c0\") " pod="opendatahub/test-curl-7700-a0981d"
Sep 10 14:48:45.963845 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:48:45.963820    2547 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7700-1001b5" container="test-curl-7700-1001b5"
Sep 10 14:48:46.074542 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:46.074462    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-a0981d"
Sep 10 14:48:46.344462 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:46.344433    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7700-a0981d"]
Sep 10 14:48:46.346000 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:48:46.345978    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c2777c7_5db2_4b6e_8ec9_eb21976374c0.slice/crio-9615e436a466b57f2f11800f8182f55c01419e03402d960386830a9a574f033e WatchSource:0}: Error finding container 9615e436a466b57f2f11800f8182f55c01419e03402d960386830a9a574f033e: Status 404 returned error can't find the container with id 9615e436a466b57f2f11800f8182f55c01419e03402d960386830a9a574f033e
Sep 10 14:48:46.971108 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:46.971073    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7700-a0981d" event={"ID":"8c2777c7-5db2-4b6e-8ec9-eb21976374c0","Type":"ContainerStarted","Data":"ef1846bc2fc01fda339daf0c25de4c42731df1a7f2f66d864fcbe366eb0a9f25"}
Sep 10 14:48:46.971108 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:46.971116    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7700-a0981d" event={"ID":"8c2777c7-5db2-4b6e-8ec9-eb21976374c0","Type":"ContainerStarted","Data":"9615e436a466b57f2f11800f8182f55c01419e03402d960386830a9a574f033e"}
Sep 10 14:48:46.971547 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:46.971234    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7700-1001b5" podUID="63837edd-54e2-4bea-9dc6-aa19f903aa32" containerName="test-curl-7700-1001b5" containerID="cri-o://6133e031f40d4102d7dba4676973656d59b9e8e62ee227a1051b2acc190ad8b5" gracePeriod=2
Sep 10 14:48:46.986121 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:46.986075    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7700-a0981d" podStartSLOduration=1.986060191 podStartE2EDuration="1.986060191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:46.984176488 +0000 UTC m=+1739.026050972" watchObservedRunningTime="2026-09-10 14:48:46.986060191 +0000 UTC m=+1739.027934674"
Sep 10 14:48:47.562557 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:47.562524    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7700-a0981d"]
Sep 10 14:48:47.567351 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:47.567323    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7700-a0981d"]
Sep 10 14:48:47.977827 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:48:47.977795    2547 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7700-a0981d" container="test-curl-7700-a0981d"
Sep 10 14:48:48.679787 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:48.679755    2547 scope.go:117] "RemoveContainer" containerID="450161f09c1d8d5ef4f35be1c4e2315251ed83aeb266eec913358efaeb8f2450"
Sep 10 14:48:48.984312 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:48.984275    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7700-a0981d" podUID="8c2777c7-5db2-4b6e-8ec9-eb21976374c0" containerName="test-curl-7700-a0981d" containerID="cri-o://ef1846bc2fc01fda339daf0c25de4c42731df1a7f2f66d864fcbe366eb0a9f25" gracePeriod=2
Sep 10 14:48:49.145782 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:49.145753    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-1001b5"
Sep 10 14:48:49.258030 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:49.257960    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shqq6\" (UniqueName: \"kubernetes.io/projected/63837edd-54e2-4bea-9dc6-aa19f903aa32-kube-api-access-shqq6\") pod \"63837edd-54e2-4bea-9dc6-aa19f903aa32\" (UID: \"63837edd-54e2-4bea-9dc6-aa19f903aa32\") "
Sep 10 14:48:49.259927 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:49.259900    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63837edd-54e2-4bea-9dc6-aa19f903aa32-kube-api-access-shqq6" (OuterVolumeSpecName: "kube-api-access-shqq6") pod "63837edd-54e2-4bea-9dc6-aa19f903aa32" (UID: "63837edd-54e2-4bea-9dc6-aa19f903aa32"). InnerVolumeSpecName "kube-api-access-shqq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:48:49.359488 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:49.359454    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shqq6\" (UniqueName: \"kubernetes.io/projected/63837edd-54e2-4bea-9dc6-aa19f903aa32-kube-api-access-shqq6\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:48:49.509739 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:49.509645    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"]
Sep 10 14:48:49.991207 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:49.991168    2547 generic.go:356] "Generic (PLEG): container finished" podID="63837edd-54e2-4bea-9dc6-aa19f903aa32" containerID="6133e031f40d4102d7dba4676973656d59b9e8e62ee227a1051b2acc190ad8b5" exitCode=137
Sep 10 14:48:49.991672 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:49.991225    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-1001b5"
Sep 10 14:48:49.991672 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:49.991283    2547 scope.go:117] "RemoveContainer" containerID="6133e031f40d4102d7dba4676973656d59b9e8e62ee227a1051b2acc190ad8b5"
Sep 10 14:48:50.001062 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:50.001037    2547 scope.go:117] "RemoveContainer" containerID="6133e031f40d4102d7dba4676973656d59b9e8e62ee227a1051b2acc190ad8b5"
Sep 10 14:48:50.001329 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:48:50.001310    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6133e031f40d4102d7dba4676973656d59b9e8e62ee227a1051b2acc190ad8b5\": container with ID starting with 6133e031f40d4102d7dba4676973656d59b9e8e62ee227a1051b2acc190ad8b5 not found: ID does not exist" containerID="6133e031f40d4102d7dba4676973656d59b9e8e62ee227a1051b2acc190ad8b5"
Sep 10 14:48:50.001382 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:50.001338    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6133e031f40d4102d7dba4676973656d59b9e8e62ee227a1051b2acc190ad8b5"} err="failed to get container status \"6133e031f40d4102d7dba4676973656d59b9e8e62ee227a1051b2acc190ad8b5\": rpc error: code = NotFound desc = could not find container \"6133e031f40d4102d7dba4676973656d59b9e8e62ee227a1051b2acc190ad8b5\": container with ID starting with 6133e031f40d4102d7dba4676973656d59b9e8e62ee227a1051b2acc190ad8b5 not found: ID does not exist"
Sep 10 14:48:50.002319 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:48:50.002299    2547 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7700-1001b5\" is forbidden: User \"system:node:ip-10-0-130-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-249.ec2.internal' and this object" podUID="63837edd-54e2-4bea-9dc6-aa19f903aa32" pod="opendatahub/test-curl-7700-1001b5"
Sep 10 14:48:50.421998 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:50.421966    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63837edd-54e2-4bea-9dc6-aa19f903aa32" path="/var/lib/kubelet/pods/63837edd-54e2-4bea-9dc6-aa19f903aa32/volumes"
Sep 10 14:48:50.422260 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:50.422243    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"]
Sep 10 14:48:51.177611 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:51.177577    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-a0981d"
Sep 10 14:48:51.283035 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:51.282954    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw9vp\" (UniqueName: \"kubernetes.io/projected/8c2777c7-5db2-4b6e-8ec9-eb21976374c0-kube-api-access-fw9vp\") pod \"8c2777c7-5db2-4b6e-8ec9-eb21976374c0\" (UID: \"8c2777c7-5db2-4b6e-8ec9-eb21976374c0\") "
Sep 10 14:48:51.284950 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:51.284924    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2777c7-5db2-4b6e-8ec9-eb21976374c0-kube-api-access-fw9vp" (OuterVolumeSpecName: "kube-api-access-fw9vp") pod "8c2777c7-5db2-4b6e-8ec9-eb21976374c0" (UID: "8c2777c7-5db2-4b6e-8ec9-eb21976374c0"). InnerVolumeSpecName "kube-api-access-fw9vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:48:51.384331 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:51.384294    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw9vp\" (UniqueName: \"kubernetes.io/projected/8c2777c7-5db2-4b6e-8ec9-eb21976374c0-kube-api-access-fw9vp\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:48:52.005447 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:52.005408    2547 generic.go:356] "Generic (PLEG): container finished" podID="8c2777c7-5db2-4b6e-8ec9-eb21976374c0" containerID="ef1846bc2fc01fda339daf0c25de4c42731df1a7f2f66d864fcbe366eb0a9f25" exitCode=137
Sep 10 14:48:52.005447 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:52.005449    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-a0981d"
Sep 10 14:48:52.005648 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:52.005508    2547 scope.go:117] "RemoveContainer" containerID="ef1846bc2fc01fda339daf0c25de4c42731df1a7f2f66d864fcbe366eb0a9f25"
Sep 10 14:48:52.014685 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:52.014666    2547 scope.go:117] "RemoveContainer" containerID="ef1846bc2fc01fda339daf0c25de4c42731df1a7f2f66d864fcbe366eb0a9f25"
Sep 10 14:48:52.015003 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:48:52.014984    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef1846bc2fc01fda339daf0c25de4c42731df1a7f2f66d864fcbe366eb0a9f25\": container with ID starting with ef1846bc2fc01fda339daf0c25de4c42731df1a7f2f66d864fcbe366eb0a9f25 not found: ID does not exist" containerID="ef1846bc2fc01fda339daf0c25de4c42731df1a7f2f66d864fcbe366eb0a9f25"
Sep 10 14:48:52.015049 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:52.015013    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef1846bc2fc01fda339daf0c25de4c42731df1a7f2f66d864fcbe366eb0a9f25"} err="failed to get container status \"ef1846bc2fc01fda339daf0c25de4c42731df1a7f2f66d864fcbe366eb0a9f25\": rpc error: code = NotFound desc = could not find container \"ef1846bc2fc01fda339daf0c25de4c42731df1a7f2f66d864fcbe366eb0a9f25\": container with ID starting with ef1846bc2fc01fda339daf0c25de4c42731df1a7f2f66d864fcbe366eb0a9f25 not found: ID does not exist"
Sep 10 14:48:52.015635 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:48:52.015613    2547 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7700-a0981d\" is forbidden: User \"system:node:ip-10-0-130-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-249.ec2.internal' and this object" podUID="8c2777c7-5db2-4b6e-8ec9-eb21976374c0" pod="opendatahub/test-curl-7700-a0981d"
Sep 10 14:48:52.422268 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:52.422236    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c2777c7-5db2-4b6e-8ec9-eb21976374c0" path="/var/lib/kubelet/pods/8c2777c7-5db2-4b6e-8ec9-eb21976374c0/volumes"
Sep 10 14:48:55.427278 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:48:55.427239    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:49:00.723674 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.723637    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"]
Sep 10 14:49:00.724098 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.724032    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c2777c7-5db2-4b6e-8ec9-eb21976374c0" containerName="test-curl-7700-a0981d"
Sep 10 14:49:00.724098 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.724044    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c2777c7-5db2-4b6e-8ec9-eb21976374c0" containerName="test-curl-7700-a0981d"
Sep 10 14:49:00.724182 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.724113    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c2777c7-5db2-4b6e-8ec9-eb21976374c0" containerName="test-curl-7700-a0981d"
Sep 10 14:49:00.727673 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.727656    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:49:00.730225 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.730203    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-85d86b\"/\"rate-test-model-85d86b-kserve-self-signed-certs\""
Sep 10 14:49:00.736852 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.736827    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"]
Sep 10 14:49:00.770100 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.770053    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4281eed-bdd5-4b02-878f-3e9670bb1665-dshm\") pod \"rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk\" (UID: \"f4281eed-bdd5-4b02-878f-3e9670bb1665\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:49:00.770100 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.770099    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4281eed-bdd5-4b02-878f-3e9670bb1665-tmp-dir\") pod \"rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk\" (UID: \"f4281eed-bdd5-4b02-878f-3e9670bb1665\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:49:00.770307 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.770130    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kwws\" (UniqueName: \"kubernetes.io/projected/f4281eed-bdd5-4b02-878f-3e9670bb1665-kube-api-access-6kwws\") pod \"rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk\" (UID: \"f4281eed-bdd5-4b02-878f-3e9670bb1665\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:49:00.770307 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.770215    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4281eed-bdd5-4b02-878f-3e9670bb1665-tls-certs\") pod \"rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk\" (UID: \"f4281eed-bdd5-4b02-878f-3e9670bb1665\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:49:00.770307 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.770252    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4281eed-bdd5-4b02-878f-3e9670bb1665-model-cache\") pod \"rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk\" (UID: \"f4281eed-bdd5-4b02-878f-3e9670bb1665\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:49:00.770307 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.770287    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4281eed-bdd5-4b02-878f-3e9670bb1665-home\") pod \"rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk\" (UID: \"f4281eed-bdd5-4b02-878f-3e9670bb1665\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:49:00.871467 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.871424    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4281eed-bdd5-4b02-878f-3e9670bb1665-dshm\") pod \"rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk\" (UID: \"f4281eed-bdd5-4b02-878f-3e9670bb1665\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:49:00.871467 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.871466    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4281eed-bdd5-4b02-878f-3e9670bb1665-tmp-dir\") pod \"rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk\" (UID: \"f4281eed-bdd5-4b02-878f-3e9670bb1665\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:49:00.871749 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.871490    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kwws\" (UniqueName: \"kubernetes.io/projected/f4281eed-bdd5-4b02-878f-3e9670bb1665-kube-api-access-6kwws\") pod \"rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk\" (UID: \"f4281eed-bdd5-4b02-878f-3e9670bb1665\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:49:00.871749 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.871524    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4281eed-bdd5-4b02-878f-3e9670bb1665-tls-certs\") pod \"rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk\" (UID: \"f4281eed-bdd5-4b02-878f-3e9670bb1665\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:49:00.871749 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.871542    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4281eed-bdd5-4b02-878f-3e9670bb1665-model-cache\") pod \"rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk\" (UID: \"f4281eed-bdd5-4b02-878f-3e9670bb1665\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:49:00.871749 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.871568    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4281eed-bdd5-4b02-878f-3e9670bb1665-home\") pod \"rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk\" (UID: \"f4281eed-bdd5-4b02-878f-3e9670bb1665\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:49:00.871976 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.871951    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4281eed-bdd5-4b02-878f-3e9670bb1665-tmp-dir\") pod \"rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk\" (UID: \"f4281eed-bdd5-4b02-878f-3e9670bb1665\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:49:00.872033 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.872010    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4281eed-bdd5-4b02-878f-3e9670bb1665-model-cache\") pod \"rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk\" (UID: \"f4281eed-bdd5-4b02-878f-3e9670bb1665\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:49:00.872113 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.872037    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4281eed-bdd5-4b02-878f-3e9670bb1665-home\") pod \"rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk\" (UID: \"f4281eed-bdd5-4b02-878f-3e9670bb1665\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:49:00.873861 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.873838    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4281eed-bdd5-4b02-878f-3e9670bb1665-dshm\") pod \"rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk\" (UID: \"f4281eed-bdd5-4b02-878f-3e9670bb1665\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:49:00.874040 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.874022    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4281eed-bdd5-4b02-878f-3e9670bb1665-tls-certs\") pod \"rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk\" (UID: \"f4281eed-bdd5-4b02-878f-3e9670bb1665\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:49:00.879563 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:00.879539    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kwws\" (UniqueName: \"kubernetes.io/projected/f4281eed-bdd5-4b02-878f-3e9670bb1665-kube-api-access-6kwws\") pod \"rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk\" (UID: \"f4281eed-bdd5-4b02-878f-3e9670bb1665\") " pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:49:01.038115 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:01.038027    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:49:01.312128 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:01.312101    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"]
Sep 10 14:49:01.313416 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:49:01.313389    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4281eed_bdd5_4b02_878f_3e9670bb1665.slice/crio-97a6c6dc6ec6c287c6d7f845573e42982e7018a2e6d4dbda9efacfcdbca45461 WatchSource:0}: Error finding container 97a6c6dc6ec6c287c6d7f845573e42982e7018a2e6d4dbda9efacfcdbca45461: Status 404 returned error can't find the container with id 97a6c6dc6ec6c287c6d7f845573e42982e7018a2e6d4dbda9efacfcdbca45461
Sep 10 14:49:02.064050 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:02.064009    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk" event={"ID":"f4281eed-bdd5-4b02-878f-3e9670bb1665","Type":"ContainerStarted","Data":"026693be601af5ae0dd77f60c8855e273bbe1fb50873c778a72a2d3e41982fa3"}
Sep 10 14:49:02.064050 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:02.064054    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk" event={"ID":"f4281eed-bdd5-4b02-878f-3e9670bb1665","Type":"ContainerStarted","Data":"97a6c6dc6ec6c287c6d7f845573e42982e7018a2e6d4dbda9efacfcdbca45461"}
Sep 10 14:49:02.101612 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:02.101550    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk" podStartSLOduration=2.101534344 podStartE2EDuration="2.101534344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:02.099938342 +0000 UTC m=+1754.141812824" watchObservedRunningTime="2026-09-10 14:49:02.101534344 +0000 UTC m=+1754.143408826"
Sep 10 14:49:03.213749 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:03.213716    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"]
Sep 10 14:49:03.217317 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:03.217297    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"
Sep 10 14:49:03.219784 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:03.219758    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-84eb72-serving-cert\""
Sep 10 14:49:03.229443 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:03.229394    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"]
Sep 10 14:49:03.296440 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:03.296396    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/501c3d61-510f-4779-9043-d85bad10b23d-maas-api-tls\") pod \"maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5\" (UID: \"501c3d61-510f-4779-9043-d85bad10b23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"
Sep 10 14:49:03.296612 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:03.296508    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfwlr\" (UniqueName: \"kubernetes.io/projected/501c3d61-510f-4779-9043-d85bad10b23d-kube-api-access-bfwlr\") pod \"maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5\" (UID: \"501c3d61-510f-4779-9043-d85bad10b23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"
Sep 10 14:49:03.296612 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:03.296537    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/501c3d61-510f-4779-9043-d85bad10b23d-metrics-certs\") pod \"maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5\" (UID: \"501c3d61-510f-4779-9043-d85bad10b23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"
Sep 10 14:49:03.296688 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:03.296635    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/501c3d61-510f-4779-9043-d85bad10b23d-openshift-service-ca\") pod \"maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5\" (UID: \"501c3d61-510f-4779-9043-d85bad10b23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"
Sep 10 14:49:03.397422 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:03.397379    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfwlr\" (UniqueName: \"kubernetes.io/projected/501c3d61-510f-4779-9043-d85bad10b23d-kube-api-access-bfwlr\") pod \"maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5\" (UID: \"501c3d61-510f-4779-9043-d85bad10b23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"
Sep 10 14:49:03.397422 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:03.397422    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/501c3d61-510f-4779-9043-d85bad10b23d-metrics-certs\") pod \"maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5\" (UID: \"501c3d61-510f-4779-9043-d85bad10b23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"
Sep 10 14:49:03.397650 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:03.397459    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/501c3d61-510f-4779-9043-d85bad10b23d-openshift-service-ca\") pod \"maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5\" (UID: \"501c3d61-510f-4779-9043-d85bad10b23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"
Sep 10 14:49:03.397650 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:03.397488    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/501c3d61-510f-4779-9043-d85bad10b23d-maas-api-tls\") pod \"maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5\" (UID: \"501c3d61-510f-4779-9043-d85bad10b23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"
Sep 10 14:49:03.397748 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:49:03.397666    2547 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-serving-cert: secret "maas-api-e2e-api-b-84eb72-serving-cert" not found
Sep 10 14:49:03.397787 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:49:03.397748    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/501c3d61-510f-4779-9043-d85bad10b23d-maas-api-tls podName:501c3d61-510f-4779-9043-d85bad10b23d nodeName:}" failed. No retries permitted until 2026-09-10 14:49:03.897728182 +0000 UTC m=+1755.939602653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/501c3d61-510f-4779-9043-d85bad10b23d-maas-api-tls") pod "maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5" (UID: "501c3d61-510f-4779-9043-d85bad10b23d") : secret "maas-api-e2e-api-b-84eb72-serving-cert" not found
Sep 10 14:49:03.398215 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:03.398196    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/501c3d61-510f-4779-9043-d85bad10b23d-openshift-service-ca\") pod \"maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5\" (UID: \"501c3d61-510f-4779-9043-d85bad10b23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"
Sep 10 14:49:03.400019 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:03.399992    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/501c3d61-510f-4779-9043-d85bad10b23d-metrics-certs\") pod \"maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5\" (UID: \"501c3d61-510f-4779-9043-d85bad10b23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"
Sep 10 14:49:03.407669 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:03.407639    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfwlr\" (UniqueName: \"kubernetes.io/projected/501c3d61-510f-4779-9043-d85bad10b23d-kube-api-access-bfwlr\") pod \"maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5\" (UID: \"501c3d61-510f-4779-9043-d85bad10b23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"
Sep 10 14:49:03.900922 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:03.900879    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/501c3d61-510f-4779-9043-d85bad10b23d-maas-api-tls\") pod \"maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5\" (UID: \"501c3d61-510f-4779-9043-d85bad10b23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"
Sep 10 14:49:03.903340 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:03.903308    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/501c3d61-510f-4779-9043-d85bad10b23d-maas-api-tls\") pod \"maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5\" (UID: \"501c3d61-510f-4779-9043-d85bad10b23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"
Sep 10 14:49:04.130138 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:04.130092    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"
Sep 10 14:49:04.404963 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:04.404936    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"]
Sep 10 14:49:04.407022 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:49:04.406990    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod501c3d61_510f_4779_9043_d85bad10b23d.slice/crio-9256bd9c70b53e6487cb8f28739516216658945d136b2df012dcf0bcb9b830cc WatchSource:0}: Error finding container 9256bd9c70b53e6487cb8f28739516216658945d136b2df012dcf0bcb9b830cc: Status 404 returned error can't find the container with id 9256bd9c70b53e6487cb8f28739516216658945d136b2df012dcf0bcb9b830cc
Sep 10 14:49:04.529251 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:04.529157    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb" podUID="6d904258-3797-4239-81e8-2a3488fa9436" containerName="main" containerID="cri-o://443e30b9df9e89af556bf0d8907bc4b9a3103bc6eb5a49bfa8177b4e6099cb25" gracePeriod=45
Sep 10 14:49:04.791577 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:04.791551    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:49:04.908953 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:04.908919    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d904258-3797-4239-81e8-2a3488fa9436-model-cache\") pod \"6d904258-3797-4239-81e8-2a3488fa9436\" (UID: \"6d904258-3797-4239-81e8-2a3488fa9436\") "
Sep 10 14:49:04.909150 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:04.909021    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d904258-3797-4239-81e8-2a3488fa9436-tls-certs\") pod \"6d904258-3797-4239-81e8-2a3488fa9436\" (UID: \"6d904258-3797-4239-81e8-2a3488fa9436\") "
Sep 10 14:49:04.909150 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:04.909062    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6j6k\" (UniqueName: \"kubernetes.io/projected/6d904258-3797-4239-81e8-2a3488fa9436-kube-api-access-v6j6k\") pod \"6d904258-3797-4239-81e8-2a3488fa9436\" (UID: \"6d904258-3797-4239-81e8-2a3488fa9436\") "
Sep 10 14:49:04.909150 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:04.909091    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d904258-3797-4239-81e8-2a3488fa9436-tmp-dir\") pod \"6d904258-3797-4239-81e8-2a3488fa9436\" (UID: \"6d904258-3797-4239-81e8-2a3488fa9436\") "
Sep 10 14:49:04.909150 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:04.909132    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d904258-3797-4239-81e8-2a3488fa9436-dshm\") pod \"6d904258-3797-4239-81e8-2a3488fa9436\" (UID: \"6d904258-3797-4239-81e8-2a3488fa9436\") "
Sep 10 14:49:04.909363 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:04.909172    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d904258-3797-4239-81e8-2a3488fa9436-home\") pod \"6d904258-3797-4239-81e8-2a3488fa9436\" (UID: \"6d904258-3797-4239-81e8-2a3488fa9436\") "
Sep 10 14:49:04.909363 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:04.909198    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d904258-3797-4239-81e8-2a3488fa9436-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d904258-3797-4239-81e8-2a3488fa9436" (UID: "6d904258-3797-4239-81e8-2a3488fa9436"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:49:04.909442 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:04.909369    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d904258-3797-4239-81e8-2a3488fa9436-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6d904258-3797-4239-81e8-2a3488fa9436" (UID: "6d904258-3797-4239-81e8-2a3488fa9436"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:49:04.909508 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:04.909468    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d904258-3797-4239-81e8-2a3488fa9436-home" (OuterVolumeSpecName: "home") pod "6d904258-3797-4239-81e8-2a3488fa9436" (UID: "6d904258-3797-4239-81e8-2a3488fa9436"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:49:04.909599 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:04.909574    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d904258-3797-4239-81e8-2a3488fa9436-tmp-dir\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:49:04.909599 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:04.909597    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d904258-3797-4239-81e8-2a3488fa9436-model-cache\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:49:04.911380 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:04.911339    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d904258-3797-4239-81e8-2a3488fa9436-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6d904258-3797-4239-81e8-2a3488fa9436" (UID: "6d904258-3797-4239-81e8-2a3488fa9436"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:49:04.911380 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:04.911363    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d904258-3797-4239-81e8-2a3488fa9436-dshm" (OuterVolumeSpecName: "dshm") pod "6d904258-3797-4239-81e8-2a3488fa9436" (UID: "6d904258-3797-4239-81e8-2a3488fa9436"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:49:04.911524 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:04.911449    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d904258-3797-4239-81e8-2a3488fa9436-kube-api-access-v6j6k" (OuterVolumeSpecName: "kube-api-access-v6j6k") pod "6d904258-3797-4239-81e8-2a3488fa9436" (UID: "6d904258-3797-4239-81e8-2a3488fa9436"). InnerVolumeSpecName "kube-api-access-v6j6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:49:05.011102 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.011061    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6j6k\" (UniqueName: \"kubernetes.io/projected/6d904258-3797-4239-81e8-2a3488fa9436-kube-api-access-v6j6k\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:49:05.011102 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.011091    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d904258-3797-4239-81e8-2a3488fa9436-dshm\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:49:05.011102 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.011101    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d904258-3797-4239-81e8-2a3488fa9436-home\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:49:05.011102 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.011109    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d904258-3797-4239-81e8-2a3488fa9436-tls-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:49:05.086409 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.086320    2547 generic.go:356] "Generic (PLEG): container finished" podID="6d904258-3797-4239-81e8-2a3488fa9436" containerID="443e30b9df9e89af556bf0d8907bc4b9a3103bc6eb5a49bfa8177b4e6099cb25" exitCode=0
Sep 10 14:49:05.086409 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.086365    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"
Sep 10 14:49:05.086632 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.086406    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb" event={"ID":"6d904258-3797-4239-81e8-2a3488fa9436","Type":"ContainerDied","Data":"443e30b9df9e89af556bf0d8907bc4b9a3103bc6eb5a49bfa8177b4e6099cb25"}
Sep 10 14:49:05.086632 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.086447    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb" event={"ID":"6d904258-3797-4239-81e8-2a3488fa9436","Type":"ContainerDied","Data":"8048314fb1b5710af58c4742cc4d8f408eed7ea56c5ca167061ccfe227a3c4d8"}
Sep 10 14:49:05.086632 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.086466    2547 scope.go:117] "RemoveContainer" containerID="443e30b9df9e89af556bf0d8907bc4b9a3103bc6eb5a49bfa8177b4e6099cb25"
Sep 10 14:49:05.088293 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.088268    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5" event={"ID":"501c3d61-510f-4779-9043-d85bad10b23d","Type":"ContainerStarted","Data":"89f82d4cc1b7cd09a53456f631b14205e9235a3402f1cf0d7d1b9de88f8d2a6c"}
Sep 10 14:49:05.088394 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.088316    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5" event={"ID":"501c3d61-510f-4779-9043-d85bad10b23d","Type":"ContainerStarted","Data":"9256bd9c70b53e6487cb8f28739516216658945d136b2df012dcf0bcb9b830cc"}
Sep 10 14:49:05.088448 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.088422    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"
Sep 10 14:49:05.096732 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.096691    2547 scope.go:117] "RemoveContainer" containerID="443e30b9df9e89af556bf0d8907bc4b9a3103bc6eb5a49bfa8177b4e6099cb25"
Sep 10 14:49:05.097000 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:49:05.096982    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"443e30b9df9e89af556bf0d8907bc4b9a3103bc6eb5a49bfa8177b4e6099cb25\": container with ID starting with 443e30b9df9e89af556bf0d8907bc4b9a3103bc6eb5a49bfa8177b4e6099cb25 not found: ID does not exist" containerID="443e30b9df9e89af556bf0d8907bc4b9a3103bc6eb5a49bfa8177b4e6099cb25"
Sep 10 14:49:05.097065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.097006    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443e30b9df9e89af556bf0d8907bc4b9a3103bc6eb5a49bfa8177b4e6099cb25"} err="failed to get container status \"443e30b9df9e89af556bf0d8907bc4b9a3103bc6eb5a49bfa8177b4e6099cb25\": rpc error: code = NotFound desc = could not find container \"443e30b9df9e89af556bf0d8907bc4b9a3103bc6eb5a49bfa8177b4e6099cb25\": container with ID starting with 443e30b9df9e89af556bf0d8907bc4b9a3103bc6eb5a49bfa8177b4e6099cb25 not found: ID does not exist"
Sep 10 14:49:05.111275 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.111226    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5" podStartSLOduration=1.8098978479999999 podStartE2EDuration="2.111211433s" totalImagesPullingTime="301.313585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:05.108186544 +0000 UTC m=+1757.150061026" watchObservedRunningTime="2026-09-10 14:49:05.111211433 +0000 UTC m=+1757.153085913"
Sep 10 14:49:05.124754 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.124729    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"]
Sep 10 14:49:05.127615 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.127591    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-85d86b/sub-test-model-85d86b-kserve-5dc9c76f69-tp4kb"]
Sep 10 14:49:05.443311 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.443272    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt" podUID="8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5" containerName="main" containerID="cri-o://1efab1a9f1c382ebd6935b2cb66c9904dcbf50333cff78cdd23919dc47f564e9" gracePeriod=45
Sep 10 14:49:05.695843 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.695783    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:49:05.817508 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.817476    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-tls-certs\") pod \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\" (UID: \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\") "
Sep 10 14:49:05.817715 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.817515    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-model-cache\") pod \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\" (UID: \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\") "
Sep 10 14:49:05.817715 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.817539    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-tmp-dir\") pod \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\" (UID: \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\") "
Sep 10 14:49:05.817715 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.817565    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-home\") pod \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\" (UID: \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\") "
Sep 10 14:49:05.817715 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.817594    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l97kh\" (UniqueName: \"kubernetes.io/projected/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-kube-api-access-l97kh\") pod \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\" (UID: \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\") "
Sep 10 14:49:05.817715 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.817680    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-dshm\") pod \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\" (UID: \"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5\") "
Sep 10 14:49:05.818004 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.817775    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5" (UID: "8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:49:05.818004 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.817854    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-home" (OuterVolumeSpecName: "home") pod "8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5" (UID: "8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:49:05.818004 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.817871    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5" (UID: "8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:49:05.818130 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.818082    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-model-cache\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:49:05.818130 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.818108    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-tmp-dir\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:49:05.818130 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.818121    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-home\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:49:05.819633 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.819605    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-kube-api-access-l97kh" (OuterVolumeSpecName: "kube-api-access-l97kh") pod "8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5" (UID: "8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5"). InnerVolumeSpecName "kube-api-access-l97kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:49:05.819779 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.819629    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5" (UID: "8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:49:05.819779 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.819673    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-dshm" (OuterVolumeSpecName: "dshm") pod "8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5" (UID: "8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:49:05.919448 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.919403    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l97kh\" (UniqueName: \"kubernetes.io/projected/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-kube-api-access-l97kh\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:49:05.919448 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.919440    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-dshm\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:49:05.919657 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:05.919459    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5-tls-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:49:06.095881 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:06.095786    2547 generic.go:356] "Generic (PLEG): container finished" podID="8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5" containerID="1efab1a9f1c382ebd6935b2cb66c9904dcbf50333cff78cdd23919dc47f564e9" exitCode=0
Sep 10 14:49:06.095881 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:06.095836    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"
Sep 10 14:49:06.095881 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:06.095870    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt" event={"ID":"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5","Type":"ContainerDied","Data":"1efab1a9f1c382ebd6935b2cb66c9904dcbf50333cff78cdd23919dc47f564e9"}
Sep 10 14:49:06.096155 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:06.095912    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt" event={"ID":"8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5","Type":"ContainerDied","Data":"425c31e678e0500526459c55f17417a594e48f6e88993ce492b8b9261a0bf9c4"}
Sep 10 14:49:06.096155 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:06.095930    2547 scope.go:117] "RemoveContainer" containerID="1efab1a9f1c382ebd6935b2cb66c9904dcbf50333cff78cdd23919dc47f564e9"
Sep 10 14:49:06.113158 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:06.113139    2547 scope.go:117] "RemoveContainer" containerID="1efab1a9f1c382ebd6935b2cb66c9904dcbf50333cff78cdd23919dc47f564e9"
Sep 10 14:49:06.113439 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:49:06.113419    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1efab1a9f1c382ebd6935b2cb66c9904dcbf50333cff78cdd23919dc47f564e9\": container with ID starting with 1efab1a9f1c382ebd6935b2cb66c9904dcbf50333cff78cdd23919dc47f564e9 not found: ID does not exist" containerID="1efab1a9f1c382ebd6935b2cb66c9904dcbf50333cff78cdd23919dc47f564e9"
Sep 10 14:49:06.113506 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:06.113446    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1efab1a9f1c382ebd6935b2cb66c9904dcbf50333cff78cdd23919dc47f564e9"} err="failed to get container status \"1efab1a9f1c382ebd6935b2cb66c9904dcbf50333cff78cdd23919dc47f564e9\": rpc error: code = NotFound desc = could not find container \"1efab1a9f1c382ebd6935b2cb66c9904dcbf50333cff78cdd23919dc47f564e9\": container with ID starting with 1efab1a9f1c382ebd6935b2cb66c9904dcbf50333cff78cdd23919dc47f564e9 not found: ID does not exist"
Sep 10 14:49:06.122864 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:06.122840    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"]
Sep 10 14:49:06.126634 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:06.126614    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3477ac/sub-test-model-3477ac-kserve-745fb87dc7-qp8kt"]
Sep 10 14:49:06.422277 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:06.422246    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d904258-3797-4239-81e8-2a3488fa9436" path="/var/lib/kubelet/pods/6d904258-3797-4239-81e8-2a3488fa9436/volumes"
Sep 10 14:49:06.424469 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:06.422646    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5" path="/var/lib/kubelet/pods/8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5/volumes"
Sep 10 14:49:10.330635 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:10.330595    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:49:11.038826 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:11.038782    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:49:11.039012 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:11.038867    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:49:11.051791 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:11.051757    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:49:11.103065 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:11.103033    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"
Sep 10 14:49:11.137823 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:11.137789    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:49:31.523773 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.523731    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"]
Sep 10 14:49:31.524334 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.524305    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d904258-3797-4239-81e8-2a3488fa9436" containerName="main"
Sep 10 14:49:31.524334 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.524335    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d904258-3797-4239-81e8-2a3488fa9436" containerName="main"
Sep 10 14:49:31.524479 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.524350    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5" containerName="main"
Sep 10 14:49:31.524479 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.524358    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5" containerName="main"
Sep 10 14:49:31.524479 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.524451    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f7b6bc0-f728-47f5-8fc6-3d3dbd2df4f5" containerName="main"
Sep 10 14:49:31.524479 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.524465    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d904258-3797-4239-81e8-2a3488fa9436" containerName="main"
Sep 10 14:49:31.528845 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.528823    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:49:31.532885 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.532865    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3477ac\"/\"kube-root-ca.crt\""
Sep 10 14:49:31.533027 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.532868    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3477ac\"/\"rate-test-model-3477ac-kserve-self-signed-certs\""
Sep 10 14:49:31.533027 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.532867    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3477ac\"/\"openshift-service-ca.crt\""
Sep 10 14:49:31.533138 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.532944    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3477ac\"/\"default-dockercfg-f4brn\""
Sep 10 14:49:31.538067 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.538042    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"]
Sep 10 14:49:31.562684 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.562654    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a44c1a0-e2be-4486-9440-07117e660cad-dshm\") pod \"rate-test-model-3477ac-kserve-6d74bdfccf-sh77b\" (UID: \"7a44c1a0-e2be-4486-9440-07117e660cad\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:49:31.562861 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.562711    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a44c1a0-e2be-4486-9440-07117e660cad-tls-certs\") pod \"rate-test-model-3477ac-kserve-6d74bdfccf-sh77b\" (UID: \"7a44c1a0-e2be-4486-9440-07117e660cad\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:49:31.562861 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.562737    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a44c1a0-e2be-4486-9440-07117e660cad-model-cache\") pod \"rate-test-model-3477ac-kserve-6d74bdfccf-sh77b\" (UID: \"7a44c1a0-e2be-4486-9440-07117e660cad\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:49:31.562861 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.562841    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a44c1a0-e2be-4486-9440-07117e660cad-home\") pod \"rate-test-model-3477ac-kserve-6d74bdfccf-sh77b\" (UID: \"7a44c1a0-e2be-4486-9440-07117e660cad\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:49:31.562992 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.562878    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a44c1a0-e2be-4486-9440-07117e660cad-tmp-dir\") pod \"rate-test-model-3477ac-kserve-6d74bdfccf-sh77b\" (UID: \"7a44c1a0-e2be-4486-9440-07117e660cad\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:49:31.562992 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.562940    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zvch\" (UniqueName: \"kubernetes.io/projected/7a44c1a0-e2be-4486-9440-07117e660cad-kube-api-access-4zvch\") pod \"rate-test-model-3477ac-kserve-6d74bdfccf-sh77b\" (UID: \"7a44c1a0-e2be-4486-9440-07117e660cad\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:49:31.663786 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.663738    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a44c1a0-e2be-4486-9440-07117e660cad-home\") pod \"rate-test-model-3477ac-kserve-6d74bdfccf-sh77b\" (UID: \"7a44c1a0-e2be-4486-9440-07117e660cad\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:49:31.663957 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.663806    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a44c1a0-e2be-4486-9440-07117e660cad-tmp-dir\") pod \"rate-test-model-3477ac-kserve-6d74bdfccf-sh77b\" (UID: \"7a44c1a0-e2be-4486-9440-07117e660cad\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:49:31.663957 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.663832    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zvch\" (UniqueName: \"kubernetes.io/projected/7a44c1a0-e2be-4486-9440-07117e660cad-kube-api-access-4zvch\") pod \"rate-test-model-3477ac-kserve-6d74bdfccf-sh77b\" (UID: \"7a44c1a0-e2be-4486-9440-07117e660cad\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:49:31.663957 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.663878    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a44c1a0-e2be-4486-9440-07117e660cad-dshm\") pod \"rate-test-model-3477ac-kserve-6d74bdfccf-sh77b\" (UID: \"7a44c1a0-e2be-4486-9440-07117e660cad\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:49:31.663957 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.663908    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a44c1a0-e2be-4486-9440-07117e660cad-tls-certs\") pod \"rate-test-model-3477ac-kserve-6d74bdfccf-sh77b\" (UID: \"7a44c1a0-e2be-4486-9440-07117e660cad\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:49:31.663957 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.663932    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a44c1a0-e2be-4486-9440-07117e660cad-model-cache\") pod \"rate-test-model-3477ac-kserve-6d74bdfccf-sh77b\" (UID: \"7a44c1a0-e2be-4486-9440-07117e660cad\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:49:31.664179 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.664155    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a44c1a0-e2be-4486-9440-07117e660cad-home\") pod \"rate-test-model-3477ac-kserve-6d74bdfccf-sh77b\" (UID: \"7a44c1a0-e2be-4486-9440-07117e660cad\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:49:31.664237 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.664214    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a44c1a0-e2be-4486-9440-07117e660cad-tmp-dir\") pod \"rate-test-model-3477ac-kserve-6d74bdfccf-sh77b\" (UID: \"7a44c1a0-e2be-4486-9440-07117e660cad\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:49:31.664284 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.664274    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a44c1a0-e2be-4486-9440-07117e660cad-model-cache\") pod \"rate-test-model-3477ac-kserve-6d74bdfccf-sh77b\" (UID: \"7a44c1a0-e2be-4486-9440-07117e660cad\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:49:31.666148 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.666124    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a44c1a0-e2be-4486-9440-07117e660cad-dshm\") pod \"rate-test-model-3477ac-kserve-6d74bdfccf-sh77b\" (UID: \"7a44c1a0-e2be-4486-9440-07117e660cad\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:49:31.666406 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.666385    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a44c1a0-e2be-4486-9440-07117e660cad-tls-certs\") pod \"rate-test-model-3477ac-kserve-6d74bdfccf-sh77b\" (UID: \"7a44c1a0-e2be-4486-9440-07117e660cad\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:49:31.671724 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.671679    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zvch\" (UniqueName: \"kubernetes.io/projected/7a44c1a0-e2be-4486-9440-07117e660cad-kube-api-access-4zvch\") pod \"rate-test-model-3477ac-kserve-6d74bdfccf-sh77b\" (UID: \"7a44c1a0-e2be-4486-9440-07117e660cad\") " pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:49:31.840801 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:31.840711    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:49:32.111687 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:32.111664    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"]
Sep 10 14:49:32.112830 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:49:32.112795    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a44c1a0_e2be_4486_9440_07117e660cad.slice/crio-06585647e56fd4bb7cc73b769b935d1a499f60c1177dcaaa7843a771ed4db371 WatchSource:0}: Error finding container 06585647e56fd4bb7cc73b769b935d1a499f60c1177dcaaa7843a771ed4db371: Status 404 returned error can't find the container with id 06585647e56fd4bb7cc73b769b935d1a499f60c1177dcaaa7843a771ed4db371
Sep 10 14:49:32.258236 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:32.258202    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b" event={"ID":"7a44c1a0-e2be-4486-9440-07117e660cad","Type":"ContainerStarted","Data":"cd2dcd53164733e2f8bfb145e0326cbfa393b321627b82a8a407f63761dd15df"}
Sep 10 14:49:32.258393 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:32.258247    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b" event={"ID":"7a44c1a0-e2be-4486-9440-07117e660cad","Type":"ContainerStarted","Data":"06585647e56fd4bb7cc73b769b935d1a499f60c1177dcaaa7843a771ed4db371"}
Sep 10 14:49:32.275940 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:32.275888    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b" podStartSLOduration=1.275872681 podStartE2EDuration="1.275872681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:32.273794599 +0000 UTC m=+1784.315669093" watchObservedRunningTime="2026-09-10 14:49:32.275872681 +0000 UTC m=+1784.317747161"
Sep 10 14:49:36.738457 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:36.738415    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:49:41.841607 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:41.841515    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:49:41.841607 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:41.841576    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:49:41.853884 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:41.853860    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:49:42.328105 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:42.328077    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:49:45.683450 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:45.683417    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"]
Sep 10 14:49:45.684000 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:45.683657    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr" podUID="340a1e3b-febd-467d-a31f-c151e896192c" containerName="maas-api" containerID="cri-o://4de4548d16cfb22cf1ae4a53abe25f650f124f5d1c283b7d2750114156daa687" gracePeriod=30
Sep 10 14:49:45.935197 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:45.935138    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"
Sep 10 14:49:46.007175 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:46.007144    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/340a1e3b-febd-467d-a31f-c151e896192c-maas-api-tls\") pod \"340a1e3b-febd-467d-a31f-c151e896192c\" (UID: \"340a1e3b-febd-467d-a31f-c151e896192c\") "
Sep 10 14:49:46.007348 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:46.007189    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/340a1e3b-febd-467d-a31f-c151e896192c-openshift-service-ca\") pod \"340a1e3b-febd-467d-a31f-c151e896192c\" (UID: \"340a1e3b-febd-467d-a31f-c151e896192c\") "
Sep 10 14:49:46.007348 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:46.007255    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7b6m\" (UniqueName: \"kubernetes.io/projected/340a1e3b-febd-467d-a31f-c151e896192c-kube-api-access-d7b6m\") pod \"340a1e3b-febd-467d-a31f-c151e896192c\" (UID: \"340a1e3b-febd-467d-a31f-c151e896192c\") "
Sep 10 14:49:46.007348 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:46.007305    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/340a1e3b-febd-467d-a31f-c151e896192c-metrics-certs\") pod \"340a1e3b-febd-467d-a31f-c151e896192c\" (UID: \"340a1e3b-febd-467d-a31f-c151e896192c\") "
Sep 10 14:49:46.007631 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:46.007598    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/340a1e3b-febd-467d-a31f-c151e896192c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "340a1e3b-febd-467d-a31f-c151e896192c" (UID: "340a1e3b-febd-467d-a31f-c151e896192c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:49:46.009333 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:46.009303    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/340a1e3b-febd-467d-a31f-c151e896192c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "340a1e3b-febd-467d-a31f-c151e896192c" (UID: "340a1e3b-febd-467d-a31f-c151e896192c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:49:46.009440 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:46.009345    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340a1e3b-febd-467d-a31f-c151e896192c-kube-api-access-d7b6m" (OuterVolumeSpecName: "kube-api-access-d7b6m") pod "340a1e3b-febd-467d-a31f-c151e896192c" (UID: "340a1e3b-febd-467d-a31f-c151e896192c"). InnerVolumeSpecName "kube-api-access-d7b6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:49:46.009440 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:46.009403    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/340a1e3b-febd-467d-a31f-c151e896192c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "340a1e3b-febd-467d-a31f-c151e896192c" (UID: "340a1e3b-febd-467d-a31f-c151e896192c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:49:46.108847 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:46.108818    2547 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/340a1e3b-febd-467d-a31f-c151e896192c-maas-api-tls\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:49:46.108847 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:46.108845    2547 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/340a1e3b-febd-467d-a31f-c151e896192c-openshift-service-ca\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:49:46.108847 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:46.108854    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7b6m\" (UniqueName: \"kubernetes.io/projected/340a1e3b-febd-467d-a31f-c151e896192c-kube-api-access-d7b6m\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:49:46.109075 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:46.108863    2547 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/340a1e3b-febd-467d-a31f-c151e896192c-metrics-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:49:46.342163 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:46.342071    2547 generic.go:356] "Generic (PLEG): container finished" podID="340a1e3b-febd-467d-a31f-c151e896192c" containerID="4de4548d16cfb22cf1ae4a53abe25f650f124f5d1c283b7d2750114156daa687" exitCode=0
Sep 10 14:49:46.342163 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:46.342123    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"
Sep 10 14:49:46.342163 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:46.342147    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr" event={"ID":"340a1e3b-febd-467d-a31f-c151e896192c","Type":"ContainerDied","Data":"4de4548d16cfb22cf1ae4a53abe25f650f124f5d1c283b7d2750114156daa687"}
Sep 10 14:49:46.342433 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:46.342199    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr" event={"ID":"340a1e3b-febd-467d-a31f-c151e896192c","Type":"ContainerDied","Data":"43e5da9a60267f6cf4a12990b33ed1531e4c5e3174a49e056e7adb098c6c8bb4"}
Sep 10 14:49:46.342433 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:46.342221    2547 scope.go:117] "RemoveContainer" containerID="4de4548d16cfb22cf1ae4a53abe25f650f124f5d1c283b7d2750114156daa687"
Sep 10 14:49:46.352754 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:46.352733    2547 scope.go:117] "RemoveContainer" containerID="4de4548d16cfb22cf1ae4a53abe25f650f124f5d1c283b7d2750114156daa687"
Sep 10 14:49:46.353069 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:49:46.353047    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4de4548d16cfb22cf1ae4a53abe25f650f124f5d1c283b7d2750114156daa687\": container with ID starting with 4de4548d16cfb22cf1ae4a53abe25f650f124f5d1c283b7d2750114156daa687 not found: ID does not exist" containerID="4de4548d16cfb22cf1ae4a53abe25f650f124f5d1c283b7d2750114156daa687"
Sep 10 14:49:46.353182 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:46.353081    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4de4548d16cfb22cf1ae4a53abe25f650f124f5d1c283b7d2750114156daa687"} err="failed to get container status \"4de4548d16cfb22cf1ae4a53abe25f650f124f5d1c283b7d2750114156daa687\": rpc error: code = NotFound desc = could not find container \"4de4548d16cfb22cf1ae4a53abe25f650f124f5d1c283b7d2750114156daa687\": container with ID starting with 4de4548d16cfb22cf1ae4a53abe25f650f124f5d1c283b7d2750114156daa687 not found: ID does not exist"
Sep 10 14:49:46.363973 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:46.363948    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"]
Sep 10 14:49:46.367393 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:46.367370    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9c318b-6865b997cd-jc9lr"]
Sep 10 14:49:46.424939 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:46.424911    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="340a1e3b-febd-467d-a31f-c151e896192c" path="/var/lib/kubelet/pods/340a1e3b-febd-467d-a31f-c151e896192c/volumes"
Sep 10 14:49:48.474711 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:48.474670    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mg9d6_359d40e9-f428-4cc3-b4e4-991c081d664a/console-operator/1.log"
Sep 10 14:49:48.480280 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:48.480253    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mg9d6_359d40e9-f428-4cc3-b4e4-991c081d664a/console-operator/1.log"
Sep 10 14:49:49.920334 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:49:49.920290    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:50:05.645980 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:05.645940    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:50:08.298906 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:08.298869    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"]
Sep 10 14:50:08.299467 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:08.299138    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5" podUID="501c3d61-510f-4779-9043-d85bad10b23d" containerName="maas-api" containerID="cri-o://89f82d4cc1b7cd09a53456f631b14205e9235a3402f1cf0d7d1b9de88f8d2a6c" gracePeriod=30
Sep 10 14:50:08.473491 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:08.473457    2547 generic.go:356] "Generic (PLEG): container finished" podID="501c3d61-510f-4779-9043-d85bad10b23d" containerID="89f82d4cc1b7cd09a53456f631b14205e9235a3402f1cf0d7d1b9de88f8d2a6c" exitCode=0
Sep 10 14:50:08.473648 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:08.473507    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5" event={"ID":"501c3d61-510f-4779-9043-d85bad10b23d","Type":"ContainerDied","Data":"89f82d4cc1b7cd09a53456f631b14205e9235a3402f1cf0d7d1b9de88f8d2a6c"}
Sep 10 14:50:08.549241 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:08.549184    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"
Sep 10 14:50:08.620768 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:08.620734    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/501c3d61-510f-4779-9043-d85bad10b23d-openshift-service-ca\") pod \"501c3d61-510f-4779-9043-d85bad10b23d\" (UID: \"501c3d61-510f-4779-9043-d85bad10b23d\") "
Sep 10 14:50:08.620931 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:08.620779    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfwlr\" (UniqueName: \"kubernetes.io/projected/501c3d61-510f-4779-9043-d85bad10b23d-kube-api-access-bfwlr\") pod \"501c3d61-510f-4779-9043-d85bad10b23d\" (UID: \"501c3d61-510f-4779-9043-d85bad10b23d\") "
Sep 10 14:50:08.620931 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:08.620846    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/501c3d61-510f-4779-9043-d85bad10b23d-maas-api-tls\") pod \"501c3d61-510f-4779-9043-d85bad10b23d\" (UID: \"501c3d61-510f-4779-9043-d85bad10b23d\") "
Sep 10 14:50:08.620931 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:08.620912    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/501c3d61-510f-4779-9043-d85bad10b23d-metrics-certs\") pod \"501c3d61-510f-4779-9043-d85bad10b23d\" (UID: \"501c3d61-510f-4779-9043-d85bad10b23d\") "
Sep 10 14:50:08.621112 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:08.621066    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/501c3d61-510f-4779-9043-d85bad10b23d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "501c3d61-510f-4779-9043-d85bad10b23d" (UID: "501c3d61-510f-4779-9043-d85bad10b23d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:50:08.621231 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:08.621204    2547 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/501c3d61-510f-4779-9043-d85bad10b23d-openshift-service-ca\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:50:08.622993 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:08.622969    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/501c3d61-510f-4779-9043-d85bad10b23d-kube-api-access-bfwlr" (OuterVolumeSpecName: "kube-api-access-bfwlr") pod "501c3d61-510f-4779-9043-d85bad10b23d" (UID: "501c3d61-510f-4779-9043-d85bad10b23d"). InnerVolumeSpecName "kube-api-access-bfwlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:50:08.623080 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:08.623022    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/501c3d61-510f-4779-9043-d85bad10b23d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "501c3d61-510f-4779-9043-d85bad10b23d" (UID: "501c3d61-510f-4779-9043-d85bad10b23d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:50:08.623080 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:08.623057    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/501c3d61-510f-4779-9043-d85bad10b23d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "501c3d61-510f-4779-9043-d85bad10b23d" (UID: "501c3d61-510f-4779-9043-d85bad10b23d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:50:08.722396 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:08.722357    2547 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/501c3d61-510f-4779-9043-d85bad10b23d-metrics-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:50:08.722396 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:08.722388    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfwlr\" (UniqueName: \"kubernetes.io/projected/501c3d61-510f-4779-9043-d85bad10b23d-kube-api-access-bfwlr\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:50:08.722396 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:08.722402    2547 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/501c3d61-510f-4779-9043-d85bad10b23d-maas-api-tls\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:50:09.480388 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:09.480354    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"
Sep 10 14:50:09.480388 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:09.480368    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5" event={"ID":"501c3d61-510f-4779-9043-d85bad10b23d","Type":"ContainerDied","Data":"9256bd9c70b53e6487cb8f28739516216658945d136b2df012dcf0bcb9b830cc"}
Sep 10 14:50:09.480937 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:09.480423    2547 scope.go:117] "RemoveContainer" containerID="89f82d4cc1b7cd09a53456f631b14205e9235a3402f1cf0d7d1b9de88f8d2a6c"
Sep 10 14:50:09.502485 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:09.502459    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"]
Sep 10 14:50:09.506121 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:09.506098    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-84eb72-59cbdbbc55-gzrh5"]
Sep 10 14:50:10.422405 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:10.422371    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="501c3d61-510f-4779-9043-d85bad10b23d" path="/var/lib/kubelet/pods/501c3d61-510f-4779-9043-d85bad10b23d/volumes"
Sep 10 14:50:15.332892 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:15.332846    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:50:20.901470 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:20.901424    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"]
Sep 10 14:50:21.814788 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:21.814749    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"]
Sep 10 14:50:25.625198 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:25.625167    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"]
Sep 10 14:50:25.625619 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:25.625407    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9" podUID="e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6" containerName="maas-api" containerID="cri-o://880b46dfff67af6b052204c3c55c7c218c7e71765f5e98057857b53bdec12ec8" gracePeriod=30
Sep 10 14:50:25.657966 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:25.657917    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc9af3-29817525-dtfb6"]
Sep 10 14:50:25.660460 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:25.660433    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc9af3-29817525-dtfb6"]
Sep 10 14:50:25.883301 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:25.883231    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"
Sep 10 14:50:25.983714 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:25.983675    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-maas-api-tls\") pod \"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6\" (UID: \"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6\") "
Sep 10 14:50:25.983881 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:25.983755    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-openshift-service-ca\") pod \"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6\" (UID: \"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6\") "
Sep 10 14:50:25.983881 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:25.983810    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snf6w\" (UniqueName: \"kubernetes.io/projected/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-kube-api-access-snf6w\") pod \"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6\" (UID: \"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6\") "
Sep 10 14:50:25.983881 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:25.983866    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-metrics-certs\") pod \"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6\" (UID: \"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6\") "
Sep 10 14:50:25.984185 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:25.984161    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6" (UID: "e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:50:25.986464 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:25.986394    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-kube-api-access-snf6w" (OuterVolumeSpecName: "kube-api-access-snf6w") pod "e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6" (UID: "e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6"). InnerVolumeSpecName "kube-api-access-snf6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:50:25.986666 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:25.986455    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6" (UID: "e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:50:25.986831 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:25.986794    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6" (UID: "e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:50:26.084965 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.084928    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snf6w\" (UniqueName: \"kubernetes.io/projected/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-kube-api-access-snf6w\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:50:26.084965 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.084958    2547 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-metrics-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:50:26.084965 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.084967    2547 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-maas-api-tls\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:50:26.084965 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.084975    2547 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6-openshift-service-ca\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:50:26.421740 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.421679    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c1263a1-00d5-4f5d-94eb-b7cdfbc471d9" path="/var/lib/kubelet/pods/1c1263a1-00d5-4f5d-94eb-b7cdfbc471d9/volumes"
Sep 10 14:50:26.577569 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.577531    2547 generic.go:356] "Generic (PLEG): container finished" podID="e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6" containerID="880b46dfff67af6b052204c3c55c7c218c7e71765f5e98057857b53bdec12ec8" exitCode=0
Sep 10 14:50:26.577754 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.577578    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"
Sep 10 14:50:26.577754 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.577623    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9" event={"ID":"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6","Type":"ContainerDied","Data":"880b46dfff67af6b052204c3c55c7c218c7e71765f5e98057857b53bdec12ec8"}
Sep 10 14:50:26.577754 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.577682    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9" event={"ID":"e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6","Type":"ContainerDied","Data":"f2b3a62ec7227b43191c127378cd5360eaf9fd40bdbaad762e852b2e47ef4e86"}
Sep 10 14:50:26.577754 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.577735    2547 scope.go:117] "RemoveContainer" containerID="880b46dfff67af6b052204c3c55c7c218c7e71765f5e98057857b53bdec12ec8"
Sep 10 14:50:26.586734 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.586707    2547 scope.go:117] "RemoveContainer" containerID="880b46dfff67af6b052204c3c55c7c218c7e71765f5e98057857b53bdec12ec8"
Sep 10 14:50:26.587049 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:50:26.587028    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"880b46dfff67af6b052204c3c55c7c218c7e71765f5e98057857b53bdec12ec8\": container with ID starting with 880b46dfff67af6b052204c3c55c7c218c7e71765f5e98057857b53bdec12ec8 not found: ID does not exist" containerID="880b46dfff67af6b052204c3c55c7c218c7e71765f5e98057857b53bdec12ec8"
Sep 10 14:50:26.587110 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.587061    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880b46dfff67af6b052204c3c55c7c218c7e71765f5e98057857b53bdec12ec8"} err="failed to get container status \"880b46dfff67af6b052204c3c55c7c218c7e71765f5e98057857b53bdec12ec8\": rpc error: code = NotFound desc = could not find container \"880b46dfff67af6b052204c3c55c7c218c7e71765f5e98057857b53bdec12ec8\": container with ID starting with 880b46dfff67af6b052204c3c55c7c218c7e71765f5e98057857b53bdec12ec8 not found: ID does not exist"
Sep 10 14:50:26.604168 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.604130    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"]
Sep 10 14:50:26.612084 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.612055    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bc9af3-694cfc8bff-5kpt9"]
Sep 10 14:50:26.758646 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.758565    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"]
Sep 10 14:50:26.759060 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.759043    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="501c3d61-510f-4779-9043-d85bad10b23d" containerName="maas-api"
Sep 10 14:50:26.759107 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.759063    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="501c3d61-510f-4779-9043-d85bad10b23d" containerName="maas-api"
Sep 10 14:50:26.759107 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.759082    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6" containerName="maas-api"
Sep 10 14:50:26.759107 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.759088    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6" containerName="maas-api"
Sep 10 14:50:26.759107 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.759104    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="340a1e3b-febd-467d-a31f-c151e896192c" containerName="maas-api"
Sep 10 14:50:26.759231 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.759110    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="340a1e3b-febd-467d-a31f-c151e896192c" containerName="maas-api"
Sep 10 14:50:26.759231 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.759167    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="501c3d61-510f-4779-9043-d85bad10b23d" containerName="maas-api"
Sep 10 14:50:26.759231 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.759177    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="340a1e3b-febd-467d-a31f-c151e896192c" containerName="maas-api"
Sep 10 14:50:26.759231 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.759185    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6" containerName="maas-api"
Sep 10 14:50:26.763748 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.763728    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"
Sep 10 14:50:26.766680 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.766656    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-cc50fd-serving-cert\""
Sep 10 14:50:26.774181 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.774149    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"]
Sep 10 14:50:26.892070 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.892032    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e7e381-06c6-443e-8085-2164ffaac00b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8\" (UID: \"75e7e381-06c6-443e-8085-2164ffaac00b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"
Sep 10 14:50:26.892070 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.892073    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e7e381-06c6-443e-8085-2164ffaac00b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8\" (UID: \"75e7e381-06c6-443e-8085-2164ffaac00b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"
Sep 10 14:50:26.892284 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.892208    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e7e381-06c6-443e-8085-2164ffaac00b-metrics-certs\") pod \"maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8\" (UID: \"75e7e381-06c6-443e-8085-2164ffaac00b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"
Sep 10 14:50:26.892284 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.892257    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtwmr\" (UniqueName: \"kubernetes.io/projected/75e7e381-06c6-443e-8085-2164ffaac00b-kube-api-access-dtwmr\") pod \"maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8\" (UID: \"75e7e381-06c6-443e-8085-2164ffaac00b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"
Sep 10 14:50:26.993752 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.993684    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e7e381-06c6-443e-8085-2164ffaac00b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8\" (UID: \"75e7e381-06c6-443e-8085-2164ffaac00b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"
Sep 10 14:50:26.993752 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.993755    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e7e381-06c6-443e-8085-2164ffaac00b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8\" (UID: \"75e7e381-06c6-443e-8085-2164ffaac00b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"
Sep 10 14:50:26.993997 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.993831    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e7e381-06c6-443e-8085-2164ffaac00b-metrics-certs\") pod \"maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8\" (UID: \"75e7e381-06c6-443e-8085-2164ffaac00b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"
Sep 10 14:50:26.993997 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:50:26.993845    2547 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-serving-cert: secret "maas-api-e2e-ipp-a-cc50fd-serving-cert" not found
Sep 10 14:50:26.993997 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.993864    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtwmr\" (UniqueName: \"kubernetes.io/projected/75e7e381-06c6-443e-8085-2164ffaac00b-kube-api-access-dtwmr\") pod \"maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8\" (UID: \"75e7e381-06c6-443e-8085-2164ffaac00b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"
Sep 10 14:50:26.993997 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:50:26.993918    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75e7e381-06c6-443e-8085-2164ffaac00b-maas-api-tls podName:75e7e381-06c6-443e-8085-2164ffaac00b nodeName:}" failed. No retries permitted until 2026-09-10 14:50:27.4938938 +0000 UTC m=+1839.535768266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/75e7e381-06c6-443e-8085-2164ffaac00b-maas-api-tls") pod "maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8" (UID: "75e7e381-06c6-443e-8085-2164ffaac00b") : secret "maas-api-e2e-ipp-a-cc50fd-serving-cert" not found
Sep 10 14:50:26.994326 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.994305    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e7e381-06c6-443e-8085-2164ffaac00b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8\" (UID: \"75e7e381-06c6-443e-8085-2164ffaac00b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"
Sep 10 14:50:26.996220 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:26.996201    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e7e381-06c6-443e-8085-2164ffaac00b-metrics-certs\") pod \"maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8\" (UID: \"75e7e381-06c6-443e-8085-2164ffaac00b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"
Sep 10 14:50:27.004974 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:27.004952    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtwmr\" (UniqueName: \"kubernetes.io/projected/75e7e381-06c6-443e-8085-2164ffaac00b-kube-api-access-dtwmr\") pod \"maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8\" (UID: \"75e7e381-06c6-443e-8085-2164ffaac00b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"
Sep 10 14:50:27.499923 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:27.499838    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e7e381-06c6-443e-8085-2164ffaac00b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8\" (UID: \"75e7e381-06c6-443e-8085-2164ffaac00b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"
Sep 10 14:50:27.502645 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:27.502613    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e7e381-06c6-443e-8085-2164ffaac00b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8\" (UID: \"75e7e381-06c6-443e-8085-2164ffaac00b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"
Sep 10 14:50:27.677049 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:27.677012    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"
Sep 10 14:50:27.957883 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:27.957856    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"]
Sep 10 14:50:27.960051 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:50:27.960012    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e7e381_06c6_443e_8085_2164ffaac00b.slice/crio-bb9265c259f7d3864ffe1f2c51e883ed6b07802b0fddc4eb9e8e6352830f573f WatchSource:0}: Error finding container bb9265c259f7d3864ffe1f2c51e883ed6b07802b0fddc4eb9e8e6352830f573f: Status 404 returned error can't find the container with id bb9265c259f7d3864ffe1f2c51e883ed6b07802b0fddc4eb9e8e6352830f573f
Sep 10 14:50:28.122384 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:28.122346    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:50:28.422273 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:28.422240    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6" path="/var/lib/kubelet/pods/e6188f2c-2c5e-40a5-b9fe-9aaaf4159ec6/volumes"
Sep 10 14:50:28.598461 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:28.598424    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8" event={"ID":"75e7e381-06c6-443e-8085-2164ffaac00b","Type":"ContainerStarted","Data":"bb9265c259f7d3864ffe1f2c51e883ed6b07802b0fddc4eb9e8e6352830f573f"}
Sep 10 14:50:29.605896 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:29.605853    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8" event={"ID":"75e7e381-06c6-443e-8085-2164ffaac00b","Type":"ContainerStarted","Data":"b35b4a1bf600b3b38b2c11b9b214042ab1ed526329cc6caa00df136f0e4c602e"}
Sep 10 14:50:29.606281 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:29.605991    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"
Sep 10 14:50:29.625367 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:29.625311    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8" podStartSLOduration=2.963937565 podStartE2EDuration="3.625295215s" totalImagesPullingTime="661.35765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:50:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:50:29.622623397 +0000 UTC m=+1841.664497880" watchObservedRunningTime="2026-09-10 14:50:29.625295215 +0000 UTC m=+1841.667169695"
Sep 10 14:50:35.616341 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:35.616305    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"
Sep 10 14:50:35.921421 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:35.921377    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk" podUID="f4281eed-bdd5-4b02-878f-3e9670bb1665" containerName="main" containerID="cri-o://026693be601af5ae0dd77f60c8855e273bbe1fb50873c778a72a2d3e41982fa3" gracePeriod=45
Sep 10 14:50:36.177638 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.177575    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:50:36.285284 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.285242    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kwws\" (UniqueName: \"kubernetes.io/projected/f4281eed-bdd5-4b02-878f-3e9670bb1665-kube-api-access-6kwws\") pod \"f4281eed-bdd5-4b02-878f-3e9670bb1665\" (UID: \"f4281eed-bdd5-4b02-878f-3e9670bb1665\") "
Sep 10 14:50:36.285484 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.285317    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4281eed-bdd5-4b02-878f-3e9670bb1665-tmp-dir\") pod \"f4281eed-bdd5-4b02-878f-3e9670bb1665\" (UID: \"f4281eed-bdd5-4b02-878f-3e9670bb1665\") "
Sep 10 14:50:36.285484 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.285358    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4281eed-bdd5-4b02-878f-3e9670bb1665-home\") pod \"f4281eed-bdd5-4b02-878f-3e9670bb1665\" (UID: \"f4281eed-bdd5-4b02-878f-3e9670bb1665\") "
Sep 10 14:50:36.285484 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.285383    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4281eed-bdd5-4b02-878f-3e9670bb1665-tls-certs\") pod \"f4281eed-bdd5-4b02-878f-3e9670bb1665\" (UID: \"f4281eed-bdd5-4b02-878f-3e9670bb1665\") "
Sep 10 14:50:36.285484 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.285406    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4281eed-bdd5-4b02-878f-3e9670bb1665-dshm\") pod \"f4281eed-bdd5-4b02-878f-3e9670bb1665\" (UID: \"f4281eed-bdd5-4b02-878f-3e9670bb1665\") "
Sep 10 14:50:36.285484 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.285453    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4281eed-bdd5-4b02-878f-3e9670bb1665-model-cache\") pod \"f4281eed-bdd5-4b02-878f-3e9670bb1665\" (UID: \"f4281eed-bdd5-4b02-878f-3e9670bb1665\") "
Sep 10 14:50:36.285811 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.285641    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4281eed-bdd5-4b02-878f-3e9670bb1665-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4281eed-bdd5-4b02-878f-3e9670bb1665" (UID: "f4281eed-bdd5-4b02-878f-3e9670bb1665"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:50:36.285811 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.285654    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4281eed-bdd5-4b02-878f-3e9670bb1665-home" (OuterVolumeSpecName: "home") pod "f4281eed-bdd5-4b02-878f-3e9670bb1665" (UID: "f4281eed-bdd5-4b02-878f-3e9670bb1665"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:50:36.285912 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.285803    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4281eed-bdd5-4b02-878f-3e9670bb1665-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4281eed-bdd5-4b02-878f-3e9670bb1665" (UID: "f4281eed-bdd5-4b02-878f-3e9670bb1665"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:50:36.285912 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.285805    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4281eed-bdd5-4b02-878f-3e9670bb1665-tmp-dir\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:50:36.287609 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.287585    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4281eed-bdd5-4b02-878f-3e9670bb1665-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4281eed-bdd5-4b02-878f-3e9670bb1665" (UID: "f4281eed-bdd5-4b02-878f-3e9670bb1665"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:50:36.287609 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.287596    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4281eed-bdd5-4b02-878f-3e9670bb1665-dshm" (OuterVolumeSpecName: "dshm") pod "f4281eed-bdd5-4b02-878f-3e9670bb1665" (UID: "f4281eed-bdd5-4b02-878f-3e9670bb1665"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:50:36.287742 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.287661    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4281eed-bdd5-4b02-878f-3e9670bb1665-kube-api-access-6kwws" (OuterVolumeSpecName: "kube-api-access-6kwws") pod "f4281eed-bdd5-4b02-878f-3e9670bb1665" (UID: "f4281eed-bdd5-4b02-878f-3e9670bb1665"). InnerVolumeSpecName "kube-api-access-6kwws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:50:36.386441 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.386406    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4281eed-bdd5-4b02-878f-3e9670bb1665-home\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:50:36.386441 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.386436    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4281eed-bdd5-4b02-878f-3e9670bb1665-tls-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:50:36.386441 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.386446    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4281eed-bdd5-4b02-878f-3e9670bb1665-dshm\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:50:36.386658 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.386454    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4281eed-bdd5-4b02-878f-3e9670bb1665-model-cache\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:50:36.386658 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.386463    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kwws\" (UniqueName: \"kubernetes.io/projected/f4281eed-bdd5-4b02-878f-3e9670bb1665-kube-api-access-6kwws\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:50:36.646274 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.646238    2547 generic.go:356] "Generic (PLEG): container finished" podID="f4281eed-bdd5-4b02-878f-3e9670bb1665" containerID="026693be601af5ae0dd77f60c8855e273bbe1fb50873c778a72a2d3e41982fa3" exitCode=0
Sep 10 14:50:36.646690 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.646292    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"
Sep 10 14:50:36.646690 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.646323    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk" event={"ID":"f4281eed-bdd5-4b02-878f-3e9670bb1665","Type":"ContainerDied","Data":"026693be601af5ae0dd77f60c8855e273bbe1fb50873c778a72a2d3e41982fa3"}
Sep 10 14:50:36.646690 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.646364    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk" event={"ID":"f4281eed-bdd5-4b02-878f-3e9670bb1665","Type":"ContainerDied","Data":"97a6c6dc6ec6c287c6d7f845573e42982e7018a2e6d4dbda9efacfcdbca45461"}
Sep 10 14:50:36.646690 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.646380    2547 scope.go:117] "RemoveContainer" containerID="026693be601af5ae0dd77f60c8855e273bbe1fb50873c778a72a2d3e41982fa3"
Sep 10 14:50:36.655654 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.655636    2547 scope.go:117] "RemoveContainer" containerID="026693be601af5ae0dd77f60c8855e273bbe1fb50873c778a72a2d3e41982fa3"
Sep 10 14:50:36.655965 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:50:36.655925    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"026693be601af5ae0dd77f60c8855e273bbe1fb50873c778a72a2d3e41982fa3\": container with ID starting with 026693be601af5ae0dd77f60c8855e273bbe1fb50873c778a72a2d3e41982fa3 not found: ID does not exist" containerID="026693be601af5ae0dd77f60c8855e273bbe1fb50873c778a72a2d3e41982fa3"
Sep 10 14:50:36.656057 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.655975    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026693be601af5ae0dd77f60c8855e273bbe1fb50873c778a72a2d3e41982fa3"} err="failed to get container status \"026693be601af5ae0dd77f60c8855e273bbe1fb50873c778a72a2d3e41982fa3\": rpc error: code = NotFound desc = could not find container \"026693be601af5ae0dd77f60c8855e273bbe1fb50873c778a72a2d3e41982fa3\": container with ID starting with 026693be601af5ae0dd77f60c8855e273bbe1fb50873c778a72a2d3e41982fa3 not found: ID does not exist"
Sep 10 14:50:36.663225 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.663202    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"]
Sep 10 14:50:36.667129 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.667108    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-85d86b/rate-test-model-85d86b-kserve-7ff7f5fcbc-4n4mk"]
Sep 10 14:50:36.836101 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:36.836010    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b" podUID="7a44c1a0-e2be-4486-9440-07117e660cad" containerName="main" containerID="cri-o://cd2dcd53164733e2f8bfb145e0326cbfa393b321627b82a8a407f63761dd15df" gracePeriod=45
Sep 10 14:50:37.090631 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.090572    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:50:37.194194 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.194141    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a44c1a0-e2be-4486-9440-07117e660cad-tmp-dir\") pod \"7a44c1a0-e2be-4486-9440-07117e660cad\" (UID: \"7a44c1a0-e2be-4486-9440-07117e660cad\") "
Sep 10 14:50:37.194194 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.194196    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a44c1a0-e2be-4486-9440-07117e660cad-home\") pod \"7a44c1a0-e2be-4486-9440-07117e660cad\" (UID: \"7a44c1a0-e2be-4486-9440-07117e660cad\") "
Sep 10 14:50:37.194431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.194266    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zvch\" (UniqueName: \"kubernetes.io/projected/7a44c1a0-e2be-4486-9440-07117e660cad-kube-api-access-4zvch\") pod \"7a44c1a0-e2be-4486-9440-07117e660cad\" (UID: \"7a44c1a0-e2be-4486-9440-07117e660cad\") "
Sep 10 14:50:37.194431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.194323    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a44c1a0-e2be-4486-9440-07117e660cad-tls-certs\") pod \"7a44c1a0-e2be-4486-9440-07117e660cad\" (UID: \"7a44c1a0-e2be-4486-9440-07117e660cad\") "
Sep 10 14:50:37.194431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.194358    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a44c1a0-e2be-4486-9440-07117e660cad-model-cache\") pod \"7a44c1a0-e2be-4486-9440-07117e660cad\" (UID: \"7a44c1a0-e2be-4486-9440-07117e660cad\") "
Sep 10 14:50:37.194431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.194408    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a44c1a0-e2be-4486-9440-07117e660cad-dshm\") pod \"7a44c1a0-e2be-4486-9440-07117e660cad\" (UID: \"7a44c1a0-e2be-4486-9440-07117e660cad\") "
Sep 10 14:50:37.194641 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.194455    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a44c1a0-e2be-4486-9440-07117e660cad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7a44c1a0-e2be-4486-9440-07117e660cad" (UID: "7a44c1a0-e2be-4486-9440-07117e660cad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:50:37.194641 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.194498    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a44c1a0-e2be-4486-9440-07117e660cad-home" (OuterVolumeSpecName: "home") pod "7a44c1a0-e2be-4486-9440-07117e660cad" (UID: "7a44c1a0-e2be-4486-9440-07117e660cad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:50:37.194641 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.194635    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a44c1a0-e2be-4486-9440-07117e660cad-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a44c1a0-e2be-4486-9440-07117e660cad" (UID: "7a44c1a0-e2be-4486-9440-07117e660cad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:50:37.194837 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.194758    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a44c1a0-e2be-4486-9440-07117e660cad-model-cache\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:50:37.194837 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.194809    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a44c1a0-e2be-4486-9440-07117e660cad-tmp-dir\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:50:37.194837 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.194823    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a44c1a0-e2be-4486-9440-07117e660cad-home\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:50:37.196532 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.196497    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a44c1a0-e2be-4486-9440-07117e660cad-dshm" (OuterVolumeSpecName: "dshm") pod "7a44c1a0-e2be-4486-9440-07117e660cad" (UID: "7a44c1a0-e2be-4486-9440-07117e660cad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:50:37.196659 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.196588    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a44c1a0-e2be-4486-9440-07117e660cad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a44c1a0-e2be-4486-9440-07117e660cad" (UID: "7a44c1a0-e2be-4486-9440-07117e660cad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:50:37.196659 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.196601    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a44c1a0-e2be-4486-9440-07117e660cad-kube-api-access-4zvch" (OuterVolumeSpecName: "kube-api-access-4zvch") pod "7a44c1a0-e2be-4486-9440-07117e660cad" (UID: "7a44c1a0-e2be-4486-9440-07117e660cad"). InnerVolumeSpecName "kube-api-access-4zvch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:50:37.296052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.296002    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a44c1a0-e2be-4486-9440-07117e660cad-tls-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:50:37.296052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.296045    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a44c1a0-e2be-4486-9440-07117e660cad-dshm\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:50:37.296052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.296055    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zvch\" (UniqueName: \"kubernetes.io/projected/7a44c1a0-e2be-4486-9440-07117e660cad-kube-api-access-4zvch\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:50:37.654258 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.654215    2547 generic.go:356] "Generic (PLEG): container finished" podID="7a44c1a0-e2be-4486-9440-07117e660cad" containerID="cd2dcd53164733e2f8bfb145e0326cbfa393b321627b82a8a407f63761dd15df" exitCode=0
Sep 10 14:50:37.654258 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.654257    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"
Sep 10 14:50:37.654846 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.654292    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b" event={"ID":"7a44c1a0-e2be-4486-9440-07117e660cad","Type":"ContainerDied","Data":"cd2dcd53164733e2f8bfb145e0326cbfa393b321627b82a8a407f63761dd15df"}
Sep 10 14:50:37.654846 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.654326    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b" event={"ID":"7a44c1a0-e2be-4486-9440-07117e660cad","Type":"ContainerDied","Data":"06585647e56fd4bb7cc73b769b935d1a499f60c1177dcaaa7843a771ed4db371"}
Sep 10 14:50:37.654846 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.654346    2547 scope.go:117] "RemoveContainer" containerID="cd2dcd53164733e2f8bfb145e0326cbfa393b321627b82a8a407f63761dd15df"
Sep 10 14:50:37.664151 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.664134    2547 scope.go:117] "RemoveContainer" containerID="cd2dcd53164733e2f8bfb145e0326cbfa393b321627b82a8a407f63761dd15df"
Sep 10 14:50:37.664409 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:50:37.664388    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd2dcd53164733e2f8bfb145e0326cbfa393b321627b82a8a407f63761dd15df\": container with ID starting with cd2dcd53164733e2f8bfb145e0326cbfa393b321627b82a8a407f63761dd15df not found: ID does not exist" containerID="cd2dcd53164733e2f8bfb145e0326cbfa393b321627b82a8a407f63761dd15df"
Sep 10 14:50:37.664457 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.664415    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd2dcd53164733e2f8bfb145e0326cbfa393b321627b82a8a407f63761dd15df"} err="failed to get container status \"cd2dcd53164733e2f8bfb145e0326cbfa393b321627b82a8a407f63761dd15df\": rpc error: code = NotFound desc = could not find container \"cd2dcd53164733e2f8bfb145e0326cbfa393b321627b82a8a407f63761dd15df\": container with ID starting with cd2dcd53164733e2f8bfb145e0326cbfa393b321627b82a8a407f63761dd15df not found: ID does not exist"
Sep 10 14:50:37.677569 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.677540    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"]
Sep 10 14:50:37.684205 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:37.684183    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3477ac/rate-test-model-3477ac-kserve-6d74bdfccf-sh77b"]
Sep 10 14:50:38.421371 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:38.421338    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a44c1a0-e2be-4486-9440-07117e660cad" path="/var/lib/kubelet/pods/7a44c1a0-e2be-4486-9440-07117e660cad/volumes"
Sep 10 14:50:38.421821 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:38.421807    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4281eed-bdd5-4b02-878f-3e9670bb1665" path="/var/lib/kubelet/pods/f4281eed-bdd5-4b02-878f-3e9670bb1665/volumes"
Sep 10 14:50:41.820820 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:41.820778    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:50:51.448275 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.448242    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"]
Sep 10 14:50:51.448667 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.448652    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4281eed-bdd5-4b02-878f-3e9670bb1665" containerName="main"
Sep 10 14:50:51.448733 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.448669    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4281eed-bdd5-4b02-878f-3e9670bb1665" containerName="main"
Sep 10 14:50:51.448733 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.448691    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a44c1a0-e2be-4486-9440-07117e660cad" containerName="main"
Sep 10 14:50:51.448733 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.448709    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a44c1a0-e2be-4486-9440-07117e660cad" containerName="main"
Sep 10 14:50:51.448879 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.448779    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a44c1a0-e2be-4486-9440-07117e660cad" containerName="main"
Sep 10 14:50:51.448879 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.448788    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4281eed-bdd5-4b02-878f-3e9670bb1665" containerName="main"
Sep 10 14:50:51.453663 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.453644    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.456627 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.456607    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-eaf326-openshift-default-dockercfg-krxwz\""
Sep 10 14:50:51.465282 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.465260    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"]
Sep 10 14:50:51.524153 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.524120    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-workload-socket\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.524153 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.524159    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c0a8407-94be-44f0-8fb6-6712daf341b8-istio-token\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.524360 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.524186    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-credential-socket\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.524360 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.524274    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjsjj\" (UniqueName: \"kubernetes.io/projected/4c0a8407-94be-44f0-8fb6-6712daf341b8-kube-api-access-wjsjj\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.524438 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.524369    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-istio-data\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.524438 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.524389    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-istio-envoy\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.524438 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.524411    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0a8407-94be-44f0-8fb6-6712daf341b8-istiod-ca-cert\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.524438 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.524428    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0a8407-94be-44f0-8fb6-6712daf341b8-istio-podinfo\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.524628 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.524505    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-workload-certs\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.625407 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.625354    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-istio-data\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.625407 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.625408    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-istio-envoy\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.625684 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.625445    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0a8407-94be-44f0-8fb6-6712daf341b8-istiod-ca-cert\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.625684 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.625472    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0a8407-94be-44f0-8fb6-6712daf341b8-istio-podinfo\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.625684 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.625522    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-workload-certs\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.625902 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.625877    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-istio-data\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.625962 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.625936    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-workload-socket\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.626027 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.625964    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c0a8407-94be-44f0-8fb6-6712daf341b8-istio-token\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.626027 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.625991    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-credential-socket\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.626295 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.626271    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-credential-socket\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.626388 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.626326    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-workload-socket\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.626451 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.626388    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjsjj\" (UniqueName: \"kubernetes.io/projected/4c0a8407-94be-44f0-8fb6-6712daf341b8-kube-api-access-wjsjj\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.626514 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.626390    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0a8407-94be-44f0-8fb6-6712daf341b8-istiod-ca-cert\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.626654 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.626629    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-workload-certs\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.628580 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.628554    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-istio-envoy\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.628750 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.628725    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0a8407-94be-44f0-8fb6-6712daf341b8-istio-podinfo\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.634791 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.634765    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjsjj\" (UniqueName: \"kubernetes.io/projected/4c0a8407-94be-44f0-8fb6-6712daf341b8-kube-api-access-wjsjj\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.634882 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.634794    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c0a8407-94be-44f0-8fb6-6712daf341b8-istio-token\") pod \"e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") " pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:51.765400 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:51.765312    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:52.043111 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:52.043077    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"]
Sep 10 14:50:52.043736 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:50:52.043707    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c0a8407_94be_44f0_8fb6_6712daf341b8.slice/crio-9a46e485d00833476af5ee546e267ca8092f139a9ee33bc357fc86a0327ba161 WatchSource:0}: Error finding container 9a46e485d00833476af5ee546e267ca8092f139a9ee33bc357fc86a0327ba161: Status 404 returned error can't find the container with id 9a46e485d00833476af5ee546e267ca8092f139a9ee33bc357fc86a0327ba161
Sep 10 14:50:52.046137 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:52.046096    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:50:52.046212 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:52.046194    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:50:52.046255 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:52.046240    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:50:52.739112 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:52.739067    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm" event={"ID":"4c0a8407-94be-44f0-8fb6-6712daf341b8","Type":"ContainerStarted","Data":"9b87fb5ed290e2dc2b3428af623093cbb3aff1c5a717b51c584c65708629b2da"}
Sep 10 14:50:52.739491 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:52.739130    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm" event={"ID":"4c0a8407-94be-44f0-8fb6-6712daf341b8","Type":"ContainerStarted","Data":"9a46e485d00833476af5ee546e267ca8092f139a9ee33bc357fc86a0327ba161"}
Sep 10 14:50:52.761439 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:52.761367    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm" podStartSLOduration=1.761345646 podStartE2EDuration="1.761345646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:50:52.7593981 +0000 UTC m=+1864.801272581" watchObservedRunningTime="2026-09-10 14:50:52.761345646 +0000 UTC m=+1864.803220128"
Sep 10 14:50:52.765444 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:52.765422    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:53.423537 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:53.423504    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:50:53.771373 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:53.771298    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:54.750153 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:54.750108    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:50:54.751122 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:50:54.751100    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:51:06.752620 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:06.752576    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:51:10.776393 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:10.775211    2547 patch_prober.go:28] interesting pod/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 14:51:10.776393 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:10.775285    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm" podUID="4c0a8407-94be-44f0-8fb6-6712daf341b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 14:51:10.840200 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:10.840107    2547 generic.go:356] "Generic (PLEG): container finished" podID="4c0a8407-94be-44f0-8fb6-6712daf341b8" containerID="9b87fb5ed290e2dc2b3428af623093cbb3aff1c5a717b51c584c65708629b2da" exitCode=0
Sep 10 14:51:10.840200 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:10.840161    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm" event={"ID":"4c0a8407-94be-44f0-8fb6-6712daf341b8","Type":"ContainerDied","Data":"9b87fb5ed290e2dc2b3428af623093cbb3aff1c5a717b51c584c65708629b2da"}
Sep 10 14:51:10.840599 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:10.840584    2547 scope.go:117] "RemoveContainer" containerID="9b87fb5ed290e2dc2b3428af623093cbb3aff1c5a717b51c584c65708629b2da"
Sep 10 14:51:10.841648 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:10.841617    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:51:10.841731 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:10.841692    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:51:10.841775 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:10.841763    2547 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 14:51:11.846978 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:11.846942    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm" event={"ID":"4c0a8407-94be-44f0-8fb6-6712daf341b8","Type":"ContainerStarted","Data":"ecc1ac0b59f9993cb1880120d087d9078bbbdea8ab86b362f2d8a1730e7bd390"}
Sep 10 14:51:12.766404 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:12.766359    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:51:12.766750 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:12.766532    2547 patch_prober.go:28] interesting pod/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Sep 10 14:51:12.766750 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:12.766580    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm" podUID="4c0a8407-94be-44f0-8fb6-6712daf341b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Sep 10 14:51:13.766545 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:13.766495    2547 patch_prober.go:28] interesting pod/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Sep 10 14:51:13.767058 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:13.766570    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm" podUID="4c0a8407-94be-44f0-8fb6-6712daf341b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Sep 10 14:51:14.765993 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:14.765946    2547 patch_prober.go:28] interesting pod/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Sep 10 14:51:14.766207 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:14.766017    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm" podUID="4c0a8407-94be-44f0-8fb6-6712daf341b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Sep 10 14:51:15.766416 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:15.766376    2547 patch_prober.go:28] interesting pod/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Sep 10 14:51:15.767002 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:15.766969    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm" podUID="4c0a8407-94be-44f0-8fb6-6712daf341b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Sep 10 14:51:16.766764 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:16.766719    2547 patch_prober.go:28] interesting pod/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Sep 10 14:51:16.767376 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:16.767344    2547 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm" podUID="4c0a8407-94be-44f0-8fb6-6712daf341b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Sep 10 14:51:17.769405 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:17.769375    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:51:17.769866 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:17.769635    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:51:17.770379 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:17.770362    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:51:28.923555 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:28.923519    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"]
Sep 10 14:51:28.927243 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:28.927226    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:51:28.929811 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:28.929790    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-cc50fd\"/\"kube-root-ca.crt\""
Sep 10 14:51:28.930958 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:28.930931    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-cc50fd\"/\"ipp-route-cc50fd-kserve-self-signed-certs\""
Sep 10 14:51:28.931078 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:28.930972    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-cc50fd\"/\"default-dockercfg-sb59k\""
Sep 10 14:51:28.931078 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:28.930989    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-cc50fd\"/\"openshift-service-ca.crt\""
Sep 10 14:51:28.937666 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:28.937640    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"]
Sep 10 14:51:29.086598 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:29.086551    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-home\") pod \"ipp-route-cc50fd-kserve-666b546887-f4dn2\" (UID: \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\") " pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:51:29.086856 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:29.086684    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-tls-certs\") pod \"ipp-route-cc50fd-kserve-666b546887-f4dn2\" (UID: \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\") " pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:51:29.086856 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:29.086780    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-tmp-dir\") pod \"ipp-route-cc50fd-kserve-666b546887-f4dn2\" (UID: \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\") " pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:51:29.086856 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:29.086837    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-model-cache\") pod \"ipp-route-cc50fd-kserve-666b546887-f4dn2\" (UID: \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\") " pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:51:29.087022 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:29.086868    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2msdb\" (UniqueName: \"kubernetes.io/projected/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-kube-api-access-2msdb\") pod \"ipp-route-cc50fd-kserve-666b546887-f4dn2\" (UID: \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\") " pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:51:29.087022 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:29.086895    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-dshm\") pod \"ipp-route-cc50fd-kserve-666b546887-f4dn2\" (UID: \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\") " pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:51:29.188417 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:29.188317    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-home\") pod \"ipp-route-cc50fd-kserve-666b546887-f4dn2\" (UID: \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\") " pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:51:29.188417 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:29.188387    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-tls-certs\") pod \"ipp-route-cc50fd-kserve-666b546887-f4dn2\" (UID: \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\") " pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:51:29.188669 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:29.188445    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-tmp-dir\") pod \"ipp-route-cc50fd-kserve-666b546887-f4dn2\" (UID: \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\") " pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:51:29.188669 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:29.188486    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-model-cache\") pod \"ipp-route-cc50fd-kserve-666b546887-f4dn2\" (UID: \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\") " pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:51:29.188669 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:29.188512    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2msdb\" (UniqueName: \"kubernetes.io/projected/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-kube-api-access-2msdb\") pod \"ipp-route-cc50fd-kserve-666b546887-f4dn2\" (UID: \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\") " pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:51:29.188669 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:29.188530    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-dshm\") pod \"ipp-route-cc50fd-kserve-666b546887-f4dn2\" (UID: \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\") " pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:51:29.188908 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:29.188826    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-home\") pod \"ipp-route-cc50fd-kserve-666b546887-f4dn2\" (UID: \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\") " pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:51:29.188908 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:29.188847    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-model-cache\") pod \"ipp-route-cc50fd-kserve-666b546887-f4dn2\" (UID: \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\") " pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:51:29.188908 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:29.188885    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-tmp-dir\") pod \"ipp-route-cc50fd-kserve-666b546887-f4dn2\" (UID: \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\") " pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:51:29.190744 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:29.190722    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-dshm\") pod \"ipp-route-cc50fd-kserve-666b546887-f4dn2\" (UID: \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\") " pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:51:29.191034 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:29.191016    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-tls-certs\") pod \"ipp-route-cc50fd-kserve-666b546887-f4dn2\" (UID: \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\") " pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:51:29.196575 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:29.196552    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2msdb\" (UniqueName: \"kubernetes.io/projected/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-kube-api-access-2msdb\") pod \"ipp-route-cc50fd-kserve-666b546887-f4dn2\" (UID: \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\") " pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:51:29.237650 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:29.237618    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:51:29.507759 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:29.507732    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"]
Sep 10 14:51:29.508735 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:51:29.508684    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94a28834_7eb7_4d6e_81cf_92f1ed51cba9.slice/crio-eb82283252629059e952ecdc4d4f82fd09d7fc8990dd62a30df185b4212e9c59 WatchSource:0}: Error finding container eb82283252629059e952ecdc4d4f82fd09d7fc8990dd62a30df185b4212e9c59: Status 404 returned error can't find the container with id eb82283252629059e952ecdc4d4f82fd09d7fc8990dd62a30df185b4212e9c59
Sep 10 14:51:29.963100 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:29.963064    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2" event={"ID":"94a28834-7eb7-4d6e-81cf-92f1ed51cba9","Type":"ContainerStarted","Data":"45271cad2a0650c8fe4bb89e94a7ad63ceec7d54d95fd244bb9b0fd02c9fea88"}
Sep 10 14:51:29.963477 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:29.963111    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2" event={"ID":"94a28834-7eb7-4d6e-81cf-92f1ed51cba9","Type":"ContainerStarted","Data":"eb82283252629059e952ecdc4d4f82fd09d7fc8990dd62a30df185b4212e9c59"}
Sep 10 14:51:29.980154 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:29.980109    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2" podStartSLOduration=1.980094436 podStartE2EDuration="1.980094436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:51:29.979105225 +0000 UTC m=+1902.020979705" watchObservedRunningTime="2026-09-10 14:51:29.980094436 +0000 UTC m=+1902.021968917"
Sep 10 14:51:32.627036 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:32.627001    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:51:39.238213 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:39.238172    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:51:39.238213 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:39.238223    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:51:39.256979 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:39.251561    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:51:40.031334 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:40.031307    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:51:48.715658 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:48.715623    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:51:48.751859 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:51:48.751829    2547 scope.go:117] "RemoveContainer" containerID="dfe99d48f2c944d626222f4bb947fee02d8e41bd75c0f915071b55b9cdcc6ef1"
Sep 10 14:52:06.423488 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:06.423442    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:52:34.928102 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:34.928063    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:52:54.029481 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:54.029395    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:52:56.604272 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:56.604233    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"]
Sep 10 14:52:56.604687 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:56.604524    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8" podUID="75e7e381-06c6-443e-8085-2164ffaac00b" containerName="maas-api" containerID="cri-o://b35b4a1bf600b3b38b2c11b9b214042ab1ed526329cc6caa00df136f0e4c602e" gracePeriod=30
Sep 10 14:52:56.844596 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:56.844571    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"
Sep 10 14:52:57.007411 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:57.007370    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e7e381-06c6-443e-8085-2164ffaac00b-metrics-certs\") pod \"75e7e381-06c6-443e-8085-2164ffaac00b\" (UID: \"75e7e381-06c6-443e-8085-2164ffaac00b\") "
Sep 10 14:52:57.007594 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:57.007455    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e7e381-06c6-443e-8085-2164ffaac00b-openshift-service-ca\") pod \"75e7e381-06c6-443e-8085-2164ffaac00b\" (UID: \"75e7e381-06c6-443e-8085-2164ffaac00b\") "
Sep 10 14:52:57.007594 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:57.007491    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtwmr\" (UniqueName: \"kubernetes.io/projected/75e7e381-06c6-443e-8085-2164ffaac00b-kube-api-access-dtwmr\") pod \"75e7e381-06c6-443e-8085-2164ffaac00b\" (UID: \"75e7e381-06c6-443e-8085-2164ffaac00b\") "
Sep 10 14:52:57.007594 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:57.007518    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e7e381-06c6-443e-8085-2164ffaac00b-maas-api-tls\") pod \"75e7e381-06c6-443e-8085-2164ffaac00b\" (UID: \"75e7e381-06c6-443e-8085-2164ffaac00b\") "
Sep 10 14:52:57.007869 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:57.007843    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e7e381-06c6-443e-8085-2164ffaac00b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "75e7e381-06c6-443e-8085-2164ffaac00b" (UID: "75e7e381-06c6-443e-8085-2164ffaac00b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:52:57.009609 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:57.009581    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e7e381-06c6-443e-8085-2164ffaac00b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "75e7e381-06c6-443e-8085-2164ffaac00b" (UID: "75e7e381-06c6-443e-8085-2164ffaac00b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:52:57.009738 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:57.009654    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e7e381-06c6-443e-8085-2164ffaac00b-kube-api-access-dtwmr" (OuterVolumeSpecName: "kube-api-access-dtwmr") pod "75e7e381-06c6-443e-8085-2164ffaac00b" (UID: "75e7e381-06c6-443e-8085-2164ffaac00b"). InnerVolumeSpecName "kube-api-access-dtwmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:52:57.009738 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:57.009686    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e7e381-06c6-443e-8085-2164ffaac00b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "75e7e381-06c6-443e-8085-2164ffaac00b" (UID: "75e7e381-06c6-443e-8085-2164ffaac00b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:52:57.108494 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:57.108462    2547 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e7e381-06c6-443e-8085-2164ffaac00b-metrics-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:52:57.108494 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:57.108488    2547 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e7e381-06c6-443e-8085-2164ffaac00b-openshift-service-ca\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:52:57.108494 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:57.108497    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtwmr\" (UniqueName: \"kubernetes.io/projected/75e7e381-06c6-443e-8085-2164ffaac00b-kube-api-access-dtwmr\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:52:57.108753 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:57.108506    2547 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e7e381-06c6-443e-8085-2164ffaac00b-maas-api-tls\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:52:57.442504 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:57.442473    2547 generic.go:356] "Generic (PLEG): container finished" podID="75e7e381-06c6-443e-8085-2164ffaac00b" containerID="b35b4a1bf600b3b38b2c11b9b214042ab1ed526329cc6caa00df136f0e4c602e" exitCode=0
Sep 10 14:52:57.442674 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:57.442510    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8" event={"ID":"75e7e381-06c6-443e-8085-2164ffaac00b","Type":"ContainerDied","Data":"b35b4a1bf600b3b38b2c11b9b214042ab1ed526329cc6caa00df136f0e4c602e"}
Sep 10 14:52:57.442674 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:57.442518    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"
Sep 10 14:52:57.442674 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:57.442552    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8" event={"ID":"75e7e381-06c6-443e-8085-2164ffaac00b","Type":"ContainerDied","Data":"bb9265c259f7d3864ffe1f2c51e883ed6b07802b0fddc4eb9e8e6352830f573f"}
Sep 10 14:52:57.442674 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:57.442576    2547 scope.go:117] "RemoveContainer" containerID="b35b4a1bf600b3b38b2c11b9b214042ab1ed526329cc6caa00df136f0e4c602e"
Sep 10 14:52:57.452410 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:57.452392    2547 scope.go:117] "RemoveContainer" containerID="b35b4a1bf600b3b38b2c11b9b214042ab1ed526329cc6caa00df136f0e4c602e"
Sep 10 14:52:57.452661 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:52:57.452643    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b35b4a1bf600b3b38b2c11b9b214042ab1ed526329cc6caa00df136f0e4c602e\": container with ID starting with b35b4a1bf600b3b38b2c11b9b214042ab1ed526329cc6caa00df136f0e4c602e not found: ID does not exist" containerID="b35b4a1bf600b3b38b2c11b9b214042ab1ed526329cc6caa00df136f0e4c602e"
Sep 10 14:52:57.452725 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:57.452669    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b35b4a1bf600b3b38b2c11b9b214042ab1ed526329cc6caa00df136f0e4c602e"} err="failed to get container status \"b35b4a1bf600b3b38b2c11b9b214042ab1ed526329cc6caa00df136f0e4c602e\": rpc error: code = NotFound desc = could not find container \"b35b4a1bf600b3b38b2c11b9b214042ab1ed526329cc6caa00df136f0e4c602e\": container with ID starting with b35b4a1bf600b3b38b2c11b9b214042ab1ed526329cc6caa00df136f0e4c602e not found: ID does not exist"
Sep 10 14:52:57.464555 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:57.464530    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"]
Sep 10 14:52:57.468025 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:57.468006    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc50fd-6c7877b95b-kgrm8"]
Sep 10 14:52:58.421440 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:52:58.421404    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75e7e381-06c6-443e-8085-2164ffaac00b" path="/var/lib/kubelet/pods/75e7e381-06c6-443e-8085-2164ffaac00b/volumes"
Sep 10 14:53:02.168134 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:02.168094    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"]
Sep 10 14:53:06.515465 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:06.515428    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:53:17.187881 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.187817    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2" podUID="94a28834-7eb7-4d6e-81cf-92f1ed51cba9" containerName="main" containerID="cri-o://45271cad2a0650c8fe4bb89e94a7ad63ceec7d54d95fd244bb9b0fd02c9fea88" gracePeriod=45
Sep 10 14:53:17.436985 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.436961    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:53:17.501422 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.501341    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-tls-certs\") pod \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\" (UID: \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\") "
Sep 10 14:53:17.501422 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.501401    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-home\") pod \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\" (UID: \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\") "
Sep 10 14:53:17.501633 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.501450    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2msdb\" (UniqueName: \"kubernetes.io/projected/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-kube-api-access-2msdb\") pod \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\" (UID: \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\") "
Sep 10 14:53:17.501633 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.501471    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-dshm\") pod \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\" (UID: \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\") "
Sep 10 14:53:17.501633 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.501501    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-tmp-dir\") pod \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\" (UID: \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\") "
Sep 10 14:53:17.501772 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.501721    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-home" (OuterVolumeSpecName: "home") pod "94a28834-7eb7-4d6e-81cf-92f1ed51cba9" (UID: "94a28834-7eb7-4d6e-81cf-92f1ed51cba9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:53:17.501772 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.501764    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "94a28834-7eb7-4d6e-81cf-92f1ed51cba9" (UID: "94a28834-7eb7-4d6e-81cf-92f1ed51cba9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:53:17.501884 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.501777    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-model-cache\") pod \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\" (UID: \"94a28834-7eb7-4d6e-81cf-92f1ed51cba9\") "
Sep 10 14:53:17.502009 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.501988    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-model-cache" (OuterVolumeSpecName: "model-cache") pod "94a28834-7eb7-4d6e-81cf-92f1ed51cba9" (UID: "94a28834-7eb7-4d6e-81cf-92f1ed51cba9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:53:17.502265 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.502209    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-model-cache\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:53:17.502265 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.502229    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-home\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:53:17.502265 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.502243    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-tmp-dir\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:53:17.503768 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.503739    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-kube-api-access-2msdb" (OuterVolumeSpecName: "kube-api-access-2msdb") pod "94a28834-7eb7-4d6e-81cf-92f1ed51cba9" (UID: "94a28834-7eb7-4d6e-81cf-92f1ed51cba9"). InnerVolumeSpecName "kube-api-access-2msdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:53:17.503976 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.503951    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-dshm" (OuterVolumeSpecName: "dshm") pod "94a28834-7eb7-4d6e-81cf-92f1ed51cba9" (UID: "94a28834-7eb7-4d6e-81cf-92f1ed51cba9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:53:17.504056 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.503968    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94a28834-7eb7-4d6e-81cf-92f1ed51cba9" (UID: "94a28834-7eb7-4d6e-81cf-92f1ed51cba9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:53:17.553790 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.553756    2547 generic.go:356] "Generic (PLEG): container finished" podID="94a28834-7eb7-4d6e-81cf-92f1ed51cba9" containerID="45271cad2a0650c8fe4bb89e94a7ad63ceec7d54d95fd244bb9b0fd02c9fea88" exitCode=0
Sep 10 14:53:17.553959 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.553787    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2" event={"ID":"94a28834-7eb7-4d6e-81cf-92f1ed51cba9","Type":"ContainerDied","Data":"45271cad2a0650c8fe4bb89e94a7ad63ceec7d54d95fd244bb9b0fd02c9fea88"}
Sep 10 14:53:17.553959 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.553828    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2" event={"ID":"94a28834-7eb7-4d6e-81cf-92f1ed51cba9","Type":"ContainerDied","Data":"eb82283252629059e952ecdc4d4f82fd09d7fc8990dd62a30df185b4212e9c59"}
Sep 10 14:53:17.553959 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.553798    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"
Sep 10 14:53:17.553959 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.553845    2547 scope.go:117] "RemoveContainer" containerID="45271cad2a0650c8fe4bb89e94a7ad63ceec7d54d95fd244bb9b0fd02c9fea88"
Sep 10 14:53:17.563212 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.563195    2547 scope.go:117] "RemoveContainer" containerID="45271cad2a0650c8fe4bb89e94a7ad63ceec7d54d95fd244bb9b0fd02c9fea88"
Sep 10 14:53:17.563578 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:53:17.563557    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45271cad2a0650c8fe4bb89e94a7ad63ceec7d54d95fd244bb9b0fd02c9fea88\": container with ID starting with 45271cad2a0650c8fe4bb89e94a7ad63ceec7d54d95fd244bb9b0fd02c9fea88 not found: ID does not exist" containerID="45271cad2a0650c8fe4bb89e94a7ad63ceec7d54d95fd244bb9b0fd02c9fea88"
Sep 10 14:53:17.563681 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.563585    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45271cad2a0650c8fe4bb89e94a7ad63ceec7d54d95fd244bb9b0fd02c9fea88"} err="failed to get container status \"45271cad2a0650c8fe4bb89e94a7ad63ceec7d54d95fd244bb9b0fd02c9fea88\": rpc error: code = NotFound desc = could not find container \"45271cad2a0650c8fe4bb89e94a7ad63ceec7d54d95fd244bb9b0fd02c9fea88\": container with ID starting with 45271cad2a0650c8fe4bb89e94a7ad63ceec7d54d95fd244bb9b0fd02c9fea88 not found: ID does not exist"
Sep 10 14:53:17.574973 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.574945    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"]
Sep 10 14:53:17.577599 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.577578    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-cc50fd/ipp-route-cc50fd-kserve-666b546887-f4dn2"]
Sep 10 14:53:17.603431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.603399    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-tls-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:53:17.603431 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.603426    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2msdb\" (UniqueName: \"kubernetes.io/projected/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-kube-api-access-2msdb\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:53:17.603589 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:17.603438    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94a28834-7eb7-4d6e-81cf-92f1ed51cba9-dshm\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:53:18.421711 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:53:18.421663    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94a28834-7eb7-4d6e-81cf-92f1ed51cba9" path="/var/lib/kubelet/pods/94a28834-7eb7-4d6e-81cf-92f1ed51cba9/volumes"
Sep 10 14:54:13.134052 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:13.134012    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:54:16.702783 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:16.702741    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"]
Sep 10 14:54:16.703165 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:16.702969    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm" podUID="4c0a8407-94be-44f0-8fb6-6712daf341b8" containerName="istio-proxy" containerID="cri-o://ecc1ac0b59f9993cb1880120d087d9078bbbdea8ab86b362f2d8a1730e7bd390" gracePeriod=30
Sep 10 14:54:17.772482 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:17.772441    2547 patch_prober.go:28] interesting pod/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 14:54:17.772938 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:17.772502    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm" podUID="4c0a8407-94be-44f0-8fb6-6712daf341b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 14:54:21.909674 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:21.909643    2547 generic.go:356] "Generic (PLEG): container finished" podID="4c0a8407-94be-44f0-8fb6-6712daf341b8" containerID="ecc1ac0b59f9993cb1880120d087d9078bbbdea8ab86b362f2d8a1730e7bd390" exitCode=0
Sep 10 14:54:21.910072 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:21.909726    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm" event={"ID":"4c0a8407-94be-44f0-8fb6-6712daf341b8","Type":"ContainerDied","Data":"ecc1ac0b59f9993cb1880120d087d9078bbbdea8ab86b362f2d8a1730e7bd390"}
Sep 10 14:54:21.910072 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:21.909807    2547 scope.go:117] "RemoveContainer" containerID="9b87fb5ed290e2dc2b3428af623093cbb3aff1c5a717b51c584c65708629b2da"
Sep 10 14:54:21.949994 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:21.949972    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:54:22.087056 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.086970    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0a8407-94be-44f0-8fb6-6712daf341b8-istiod-ca-cert\") pod \"4c0a8407-94be-44f0-8fb6-6712daf341b8\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") "
Sep 10 14:54:22.087056 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.087025    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-workload-socket\") pod \"4c0a8407-94be-44f0-8fb6-6712daf341b8\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") "
Sep 10 14:54:22.087056 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.087043    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c0a8407-94be-44f0-8fb6-6712daf341b8-istio-token\") pod \"4c0a8407-94be-44f0-8fb6-6712daf341b8\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") "
Sep 10 14:54:22.087295 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.087215    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0a8407-94be-44f0-8fb6-6712daf341b8-istio-podinfo\") pod \"4c0a8407-94be-44f0-8fb6-6712daf341b8\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") "
Sep 10 14:54:22.087295 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.087284    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-credential-socket\") pod \"4c0a8407-94be-44f0-8fb6-6712daf341b8\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") "
Sep 10 14:54:22.087372 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.087294    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4c0a8407-94be-44f0-8fb6-6712daf341b8" (UID: "4c0a8407-94be-44f0-8fb6-6712daf341b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:54:22.087372 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.087323    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-istio-envoy\") pod \"4c0a8407-94be-44f0-8fb6-6712daf341b8\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") "
Sep 10 14:54:22.087463 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.087373    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjsjj\" (UniqueName: \"kubernetes.io/projected/4c0a8407-94be-44f0-8fb6-6712daf341b8-kube-api-access-wjsjj\") pod \"4c0a8407-94be-44f0-8fb6-6712daf341b8\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") "
Sep 10 14:54:22.087463 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.087375    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c0a8407-94be-44f0-8fb6-6712daf341b8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4c0a8407-94be-44f0-8fb6-6712daf341b8" (UID: "4c0a8407-94be-44f0-8fb6-6712daf341b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:54:22.087463 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.087430    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-workload-certs\") pod \"4c0a8407-94be-44f0-8fb6-6712daf341b8\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") "
Sep 10 14:54:22.087463 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.087458    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-istio-data\") pod \"4c0a8407-94be-44f0-8fb6-6712daf341b8\" (UID: \"4c0a8407-94be-44f0-8fb6-6712daf341b8\") "
Sep 10 14:54:22.087674 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.087468    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4c0a8407-94be-44f0-8fb6-6712daf341b8" (UID: "4c0a8407-94be-44f0-8fb6-6712daf341b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:54:22.087774 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.087740    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4c0a8407-94be-44f0-8fb6-6712daf341b8" (UID: "4c0a8407-94be-44f0-8fb6-6712daf341b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:54:22.087774 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.087765    2547 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-credential-socket\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:22.087880 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.087785    2547 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0a8407-94be-44f0-8fb6-6712daf341b8-istiod-ca-cert\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:22.087880 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.087799    2547 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-workload-socket\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:22.087880 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.087779    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-istio-data" (OuterVolumeSpecName: "istio-data") pod "4c0a8407-94be-44f0-8fb6-6712daf341b8" (UID: "4c0a8407-94be-44f0-8fb6-6712daf341b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:54:22.089506 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.089485    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4c0a8407-94be-44f0-8fb6-6712daf341b8" (UID: "4c0a8407-94be-44f0-8fb6-6712daf341b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:54:22.089632 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.089530    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0a8407-94be-44f0-8fb6-6712daf341b8-kube-api-access-wjsjj" (OuterVolumeSpecName: "kube-api-access-wjsjj") pod "4c0a8407-94be-44f0-8fb6-6712daf341b8" (UID: "4c0a8407-94be-44f0-8fb6-6712daf341b8"). InnerVolumeSpecName "kube-api-access-wjsjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:54:22.089730 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.089711    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0a8407-94be-44f0-8fb6-6712daf341b8-istio-token" (OuterVolumeSpecName: "istio-token") pod "4c0a8407-94be-44f0-8fb6-6712daf341b8" (UID: "4c0a8407-94be-44f0-8fb6-6712daf341b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:54:22.089730 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.089713    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c0a8407-94be-44f0-8fb6-6712daf341b8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4c0a8407-94be-44f0-8fb6-6712daf341b8" (UID: "4c0a8407-94be-44f0-8fb6-6712daf341b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 14:54:22.188459 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.188421    2547 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-workload-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:22.188459 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.188454    2547 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-istio-data\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:22.188459 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.188463    2547 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c0a8407-94be-44f0-8fb6-6712daf341b8-istio-token\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:22.188680 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.188471    2547 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0a8407-94be-44f0-8fb6-6712daf341b8-istio-podinfo\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:22.188680 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.188479    2547 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0a8407-94be-44f0-8fb6-6712daf341b8-istio-envoy\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:22.188680 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.188488    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjsjj\" (UniqueName: \"kubernetes.io/projected/4c0a8407-94be-44f0-8fb6-6712daf341b8-kube-api-access-wjsjj\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:22.918365 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.918334    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"
Sep 10 14:54:22.918772 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.918376    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm" event={"ID":"4c0a8407-94be-44f0-8fb6-6712daf341b8","Type":"ContainerDied","Data":"9a46e485d00833476af5ee546e267ca8092f139a9ee33bc357fc86a0327ba161"}
Sep 10 14:54:22.918772 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.918438    2547 scope.go:117] "RemoveContainer" containerID="ecc1ac0b59f9993cb1880120d087d9078bbbdea8ab86b362f2d8a1730e7bd390"
Sep 10 14:54:22.937322 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.937295    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"]
Sep 10 14:54:22.942544 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:22.942518    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-eaf326-openshift-default-7b854cd75d-pqlqm"]
Sep 10 14:54:24.421902 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:24.421869    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c0a8407-94be-44f0-8fb6-6712daf341b8" path="/var/lib/kubelet/pods/4c0a8407-94be-44f0-8fb6-6712daf341b8/volumes"
Sep 10 14:54:30.331782 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:30.331735    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:54:34.351446 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:34.351410    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"]
Sep 10 14:54:34.351849 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:34.351666    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2" podUID="641472f0-b228-4074-b0c2-fa5718d9377a" containerName="istio-proxy" containerID="cri-o://719eaa189c7b36fd60fdf29dcd9d4aa8cd6ea958d20ee08db73534d1369b9ecc" gracePeriod=30
Sep 10 14:54:39.600830 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.600800    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:54:39.627804 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.627738    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-workload-certs\") pod \"641472f0-b228-4074-b0c2-fa5718d9377a\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") "
Sep 10 14:54:39.627804 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.627798    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/641472f0-b228-4074-b0c2-fa5718d9377a-istio-podinfo\") pod \"641472f0-b228-4074-b0c2-fa5718d9377a\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") "
Sep 10 14:54:39.627963 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.627833    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96b4f\" (UniqueName: \"kubernetes.io/projected/641472f0-b228-4074-b0c2-fa5718d9377a-kube-api-access-96b4f\") pod \"641472f0-b228-4074-b0c2-fa5718d9377a\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") "
Sep 10 14:54:39.627963 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.627879    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/641472f0-b228-4074-b0c2-fa5718d9377a-istiod-ca-cert\") pod \"641472f0-b228-4074-b0c2-fa5718d9377a\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") "
Sep 10 14:54:39.627963 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.627907    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-credential-socket\") pod \"641472f0-b228-4074-b0c2-fa5718d9377a\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") "
Sep 10 14:54:39.627963 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.627934    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/641472f0-b228-4074-b0c2-fa5718d9377a-istio-token\") pod \"641472f0-b228-4074-b0c2-fa5718d9377a\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") "
Sep 10 14:54:39.628166 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.627982    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-workload-socket\") pod \"641472f0-b228-4074-b0c2-fa5718d9377a\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") "
Sep 10 14:54:39.628433 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.628385    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "641472f0-b228-4074-b0c2-fa5718d9377a" (UID: "641472f0-b228-4074-b0c2-fa5718d9377a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:54:39.629061 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.628525    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-istio-data\") pod \"641472f0-b228-4074-b0c2-fa5718d9377a\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") "
Sep 10 14:54:39.629061 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.628580    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-istio-envoy\") pod \"641472f0-b228-4074-b0c2-fa5718d9377a\" (UID: \"641472f0-b228-4074-b0c2-fa5718d9377a\") "
Sep 10 14:54:39.629061 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.628612    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/641472f0-b228-4074-b0c2-fa5718d9377a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "641472f0-b228-4074-b0c2-fa5718d9377a" (UID: "641472f0-b228-4074-b0c2-fa5718d9377a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:54:39.629061 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.628691    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "641472f0-b228-4074-b0c2-fa5718d9377a" (UID: "641472f0-b228-4074-b0c2-fa5718d9377a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:54:39.629061 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.628892    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-istio-data" (OuterVolumeSpecName: "istio-data") pod "641472f0-b228-4074-b0c2-fa5718d9377a" (UID: "641472f0-b228-4074-b0c2-fa5718d9377a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:54:39.629061 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.628035    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "641472f0-b228-4074-b0c2-fa5718d9377a" (UID: "641472f0-b228-4074-b0c2-fa5718d9377a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:54:39.629380 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.629170    2547 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-istio-data\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:39.629380 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.629188    2547 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-workload-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:39.629380 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.629208    2547 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/641472f0-b228-4074-b0c2-fa5718d9377a-istiod-ca-cert\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:39.629380 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.629221    2547 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-credential-socket\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:39.629380 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.629234    2547 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-workload-socket\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:39.631124 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.631094    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641472f0-b228-4074-b0c2-fa5718d9377a-kube-api-access-96b4f" (OuterVolumeSpecName: "kube-api-access-96b4f") pod "641472f0-b228-4074-b0c2-fa5718d9377a" (UID: "641472f0-b228-4074-b0c2-fa5718d9377a"). InnerVolumeSpecName "kube-api-access-96b4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:54:39.637806 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.637773    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/641472f0-b228-4074-b0c2-fa5718d9377a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "641472f0-b228-4074-b0c2-fa5718d9377a" (UID: "641472f0-b228-4074-b0c2-fa5718d9377a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 14:54:39.637998 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.637793    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641472f0-b228-4074-b0c2-fa5718d9377a-istio-token" (OuterVolumeSpecName: "istio-token") pod "641472f0-b228-4074-b0c2-fa5718d9377a" (UID: "641472f0-b228-4074-b0c2-fa5718d9377a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:54:39.638047 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.638012    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "641472f0-b228-4074-b0c2-fa5718d9377a" (UID: "641472f0-b228-4074-b0c2-fa5718d9377a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:54:39.730342 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.730302    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96b4f\" (UniqueName: \"kubernetes.io/projected/641472f0-b228-4074-b0c2-fa5718d9377a-kube-api-access-96b4f\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:39.730342 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.730337    2547 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/641472f0-b228-4074-b0c2-fa5718d9377a-istio-token\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:39.730342 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.730348    2547 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/641472f0-b228-4074-b0c2-fa5718d9377a-istio-envoy\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:39.730581 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:39.730355    2547 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/641472f0-b228-4074-b0c2-fa5718d9377a-istio-podinfo\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:40.020449 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:40.020410    2547 generic.go:356] "Generic (PLEG): container finished" podID="641472f0-b228-4074-b0c2-fa5718d9377a" containerID="719eaa189c7b36fd60fdf29dcd9d4aa8cd6ea958d20ee08db73534d1369b9ecc" exitCode=0
Sep 10 14:54:40.020613 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:40.020476    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"
Sep 10 14:54:40.020613 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:40.020489    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2" event={"ID":"641472f0-b228-4074-b0c2-fa5718d9377a","Type":"ContainerDied","Data":"719eaa189c7b36fd60fdf29dcd9d4aa8cd6ea958d20ee08db73534d1369b9ecc"}
Sep 10 14:54:40.020613 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:40.020570    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2" event={"ID":"641472f0-b228-4074-b0c2-fa5718d9377a","Type":"ContainerDied","Data":"f3138dec6ecb9e37dc54356b28c94126ff90ec49d6de4d3aeedd9eaf4b222777"}
Sep 10 14:54:40.020613 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:40.020591    2547 scope.go:117] "RemoveContainer" containerID="719eaa189c7b36fd60fdf29dcd9d4aa8cd6ea958d20ee08db73534d1369b9ecc"
Sep 10 14:54:40.029674 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:40.029657    2547 scope.go:117] "RemoveContainer" containerID="a2e9b7cb99f671a82905e800f3039588bed383bd84b2b0b868d89e6c35cd1259"
Sep 10 14:54:40.037642 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:40.037622    2547 scope.go:117] "RemoveContainer" containerID="719eaa189c7b36fd60fdf29dcd9d4aa8cd6ea958d20ee08db73534d1369b9ecc"
Sep 10 14:54:40.037899 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:54:40.037878    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"719eaa189c7b36fd60fdf29dcd9d4aa8cd6ea958d20ee08db73534d1369b9ecc\": container with ID starting with 719eaa189c7b36fd60fdf29dcd9d4aa8cd6ea958d20ee08db73534d1369b9ecc not found: ID does not exist" containerID="719eaa189c7b36fd60fdf29dcd9d4aa8cd6ea958d20ee08db73534d1369b9ecc"
Sep 10 14:54:40.037986 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:40.037911    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"719eaa189c7b36fd60fdf29dcd9d4aa8cd6ea958d20ee08db73534d1369b9ecc"} err="failed to get container status \"719eaa189c7b36fd60fdf29dcd9d4aa8cd6ea958d20ee08db73534d1369b9ecc\": rpc error: code = NotFound desc = could not find container \"719eaa189c7b36fd60fdf29dcd9d4aa8cd6ea958d20ee08db73534d1369b9ecc\": container with ID starting with 719eaa189c7b36fd60fdf29dcd9d4aa8cd6ea958d20ee08db73534d1369b9ecc not found: ID does not exist"
Sep 10 14:54:40.037986 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:40.037934    2547 scope.go:117] "RemoveContainer" containerID="a2e9b7cb99f671a82905e800f3039588bed383bd84b2b0b868d89e6c35cd1259"
Sep 10 14:54:40.038189 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:54:40.038171    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2e9b7cb99f671a82905e800f3039588bed383bd84b2b0b868d89e6c35cd1259\": container with ID starting with a2e9b7cb99f671a82905e800f3039588bed383bd84b2b0b868d89e6c35cd1259 not found: ID does not exist" containerID="a2e9b7cb99f671a82905e800f3039588bed383bd84b2b0b868d89e6c35cd1259"
Sep 10 14:54:40.038233 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:40.038195    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2e9b7cb99f671a82905e800f3039588bed383bd84b2b0b868d89e6c35cd1259"} err="failed to get container status \"a2e9b7cb99f671a82905e800f3039588bed383bd84b2b0b868d89e6c35cd1259\": rpc error: code = NotFound desc = could not find container \"a2e9b7cb99f671a82905e800f3039588bed383bd84b2b0b868d89e6c35cd1259\": container with ID starting with a2e9b7cb99f671a82905e800f3039588bed383bd84b2b0b868d89e6c35cd1259 not found: ID does not exist"
Sep 10 14:54:40.042790 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:40.042767    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"]
Sep 10 14:54:40.046237 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:40.046214    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-85d86b-openshift-default-cd4f7b454-hkzz2"]
Sep 10 14:54:40.422153 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:40.422114    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="641472f0-b228-4074-b0c2-fa5718d9377a" path="/var/lib/kubelet/pods/641472f0-b228-4074-b0c2-fa5718d9377a/volumes"
Sep 10 14:54:45.212664 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:45.212628    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt"]
Sep 10 14:54:45.213244 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:45.212853    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt" podUID="a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd" containerName="maas-api" containerID="cri-o://721d25fae898b0aa42548f819f0add1e10ff99b36d00b7ff141ff13d24a6189b" gracePeriod=30
Sep 10 14:54:45.248313 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:45.248273    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3477ac-29817525-c6k7w"]
Sep 10 14:54:45.255408 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:45.255376    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3477ac-29817525-c6k7w"]
Sep 10 14:54:45.458628 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:45.458603    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt"
Sep 10 14:54:45.577247 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:45.577149    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd-openshift-service-ca\") pod \"a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd\" (UID: \"a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd\") "
Sep 10 14:54:45.577247 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:45.577218    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd-maas-api-tls\") pod \"a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd\" (UID: \"a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd\") "
Sep 10 14:54:45.577484 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:45.577278    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd-metrics-certs\") pod \"a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd\" (UID: \"a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd\") "
Sep 10 14:54:45.577484 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:45.577315    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtdrn\" (UniqueName: \"kubernetes.io/projected/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd-kube-api-access-mtdrn\") pod \"a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd\" (UID: \"a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd\") "
Sep 10 14:54:45.577601 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:45.577572    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd" (UID: "a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:54:45.579374 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:45.579333    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd" (UID: "a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:54:45.579529 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:45.579378    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd" (UID: "a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:54:45.579529 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:45.579435    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd-kube-api-access-mtdrn" (OuterVolumeSpecName: "kube-api-access-mtdrn") pod "a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd" (UID: "a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd"). InnerVolumeSpecName "kube-api-access-mtdrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:54:45.678130 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:45.678098    2547 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd-metrics-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:45.678130 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:45.678125    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtdrn\" (UniqueName: \"kubernetes.io/projected/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd-kube-api-access-mtdrn\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:45.678130 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:45.678136    2547 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd-openshift-service-ca\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:45.678357 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:45.678145    2547 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd-maas-api-tls\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:46.056624 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:46.056588    2547 generic.go:356] "Generic (PLEG): container finished" podID="a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd" containerID="721d25fae898b0aa42548f819f0add1e10ff99b36d00b7ff141ff13d24a6189b" exitCode=0
Sep 10 14:54:46.056821 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:46.056640    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt"
Sep 10 14:54:46.056821 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:46.056666    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt" event={"ID":"a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd","Type":"ContainerDied","Data":"721d25fae898b0aa42548f819f0add1e10ff99b36d00b7ff141ff13d24a6189b"}
Sep 10 14:54:46.056821 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:46.056768    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt" event={"ID":"a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd","Type":"ContainerDied","Data":"5ede1d78541994348da1e9e0e209f98fe4873bbafe2883c1a9eec03176f9983e"}
Sep 10 14:54:46.056821 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:46.056793    2547 scope.go:117] "RemoveContainer" containerID="721d25fae898b0aa42548f819f0add1e10ff99b36d00b7ff141ff13d24a6189b"
Sep 10 14:54:46.066687 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:46.066668    2547 scope.go:117] "RemoveContainer" containerID="721d25fae898b0aa42548f819f0add1e10ff99b36d00b7ff141ff13d24a6189b"
Sep 10 14:54:46.066941 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:54:46.066924    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"721d25fae898b0aa42548f819f0add1e10ff99b36d00b7ff141ff13d24a6189b\": container with ID starting with 721d25fae898b0aa42548f819f0add1e10ff99b36d00b7ff141ff13d24a6189b not found: ID does not exist" containerID="721d25fae898b0aa42548f819f0add1e10ff99b36d00b7ff141ff13d24a6189b"
Sep 10 14:54:46.066988 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:46.066948    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"721d25fae898b0aa42548f819f0add1e10ff99b36d00b7ff141ff13d24a6189b"} err="failed to get container status \"721d25fae898b0aa42548f819f0add1e10ff99b36d00b7ff141ff13d24a6189b\": rpc error: code = NotFound desc = could not find container \"721d25fae898b0aa42548f819f0add1e10ff99b36d00b7ff141ff13d24a6189b\": container with ID starting with 721d25fae898b0aa42548f819f0add1e10ff99b36d00b7ff141ff13d24a6189b not found: ID does not exist"
Sep 10 14:54:46.079539 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:46.079513    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt"]
Sep 10 14:54:46.083152 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:46.083130    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3477ac-6478f6b646-n5cpt"]
Sep 10 14:54:46.424744 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:46.424711    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd" path="/var/lib/kubelet/pods/a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd/volumes"
Sep 10 14:54:46.425112 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:46.425008    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc199420-d1ca-419e-93af-bfcc8c0956ac" path="/var/lib/kubelet/pods/dc199420-d1ca-419e-93af-bfcc8c0956ac/volumes"
Sep 10 14:54:47.824527 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:47.824490    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:54:48.511035 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:48.511004    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mg9d6_359d40e9-f428-4cc3-b4e4-991c081d664a/console-operator/1.log"
Sep 10 14:54:48.521058 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:48.521032    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mg9d6_359d40e9-f428-4cc3-b4e4-991c081d664a/console-operator/1.log"
Sep 10 14:54:48.797005 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:48.796917    2547 scope.go:117] "RemoveContainer" containerID="27724a40f64675ddf4e3c6437f20816f68c793af0db005ad11551fb4a02edfbc"
Sep 10 14:54:52.132449 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:52.132417    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"]
Sep 10 14:54:52.132909 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:52.132622    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq" podUID="2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" containerName="istio-proxy" containerID="cri-o://c78a6d1f80c91afb867db601dbd631909120123860e5d872726830cd7f814545" gracePeriod=30
Sep 10 14:54:56.530761 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:56.530724    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:54:57.377972 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.377944    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:54:57.473368 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.473333    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-credential-socket\") pod \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") "
Sep 10 14:54:57.473368 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.473372    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-workload-certs\") pod \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") "
Sep 10 14:54:57.473621 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.473398    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-workload-socket\") pod \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") "
Sep 10 14:54:57.473621 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.473435    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istiod-ca-cert\") pod \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") "
Sep 10 14:54:57.473621 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.473486    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istio-data\") pod \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") "
Sep 10 14:54:57.473621 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.473538    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istio-envoy\") pod \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") "
Sep 10 14:54:57.473621 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.473573    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istio-token\") pod \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") "
Sep 10 14:54:57.473621 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.473611    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99wwm\" (UniqueName: \"kubernetes.io/projected/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-kube-api-access-99wwm\") pod \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") "
Sep 10 14:54:57.473961 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.473645    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istio-podinfo\") pod \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\" (UID: \"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b\") "
Sep 10 14:54:57.473961 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.473648    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" (UID: "2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:54:57.473961 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.473669    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" (UID: "2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:54:57.473961 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.473733    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" (UID: "2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:54:57.473961 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.473955    2547 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-credential-socket\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:57.474206 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.473974    2547 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-workload-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:57.474206 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.473988    2547 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-workload-socket\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:57.474206 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.474009    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istio-data" (OuterVolumeSpecName: "istio-data") pod "2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" (UID: "2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:54:57.474206 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.474027    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" (UID: "2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:54:57.475843 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.475817    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" (UID: "2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 14:54:57.475947 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.475860    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-kube-api-access-99wwm" (OuterVolumeSpecName: "kube-api-access-99wwm") pod "2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" (UID: "2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b"). InnerVolumeSpecName "kube-api-access-99wwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:54:57.475947 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.475883    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istio-token" (OuterVolumeSpecName: "istio-token") pod "2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" (UID: "2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:54:57.475947 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.475902    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" (UID: "2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:54:57.574758 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.574720    2547 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istio-envoy\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:57.574758 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.574750    2547 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istio-token\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:57.574758 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.574759    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99wwm\" (UniqueName: \"kubernetes.io/projected/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-kube-api-access-99wwm\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:57.574758 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.574769    2547 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istio-podinfo\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:57.575211 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.574778    2547 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istiod-ca-cert\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:57.575211 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:57.574786    2547 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b-istio-data\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 14:54:58.123184 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:58.123147    2547 generic.go:356] "Generic (PLEG): container finished" podID="2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" containerID="c78a6d1f80c91afb867db601dbd631909120123860e5d872726830cd7f814545" exitCode=0
Sep 10 14:54:58.123419 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:58.123201    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"
Sep 10 14:54:58.123419 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:58.123238    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq" event={"ID":"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b","Type":"ContainerDied","Data":"c78a6d1f80c91afb867db601dbd631909120123860e5d872726830cd7f814545"}
Sep 10 14:54:58.123419 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:58.123299    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq" event={"ID":"2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b","Type":"ContainerDied","Data":"dd5891f3258d21c091a1dc13421d954edd4fee3e09528d48a059f1ac4834e83d"}
Sep 10 14:54:58.123419 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:58.123324    2547 scope.go:117] "RemoveContainer" containerID="c78a6d1f80c91afb867db601dbd631909120123860e5d872726830cd7f814545"
Sep 10 14:54:58.134781 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:58.134764    2547 scope.go:117] "RemoveContainer" containerID="a30b0c83bbd04427170aee7c6c498ec94cd839d897c2d7fc55c1b0e94b9f55aa"
Sep 10 14:54:58.142961 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:58.142942    2547 scope.go:117] "RemoveContainer" containerID="c78a6d1f80c91afb867db601dbd631909120123860e5d872726830cd7f814545"
Sep 10 14:54:58.143224 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:54:58.143203    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c78a6d1f80c91afb867db601dbd631909120123860e5d872726830cd7f814545\": container with ID starting with c78a6d1f80c91afb867db601dbd631909120123860e5d872726830cd7f814545 not found: ID does not exist" containerID="c78a6d1f80c91afb867db601dbd631909120123860e5d872726830cd7f814545"
Sep 10 14:54:58.143297 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:58.143235    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c78a6d1f80c91afb867db601dbd631909120123860e5d872726830cd7f814545"} err="failed to get container status \"c78a6d1f80c91afb867db601dbd631909120123860e5d872726830cd7f814545\": rpc error: code = NotFound desc = could not find container \"c78a6d1f80c91afb867db601dbd631909120123860e5d872726830cd7f814545\": container with ID starting with c78a6d1f80c91afb867db601dbd631909120123860e5d872726830cd7f814545 not found: ID does not exist"
Sep 10 14:54:58.143297 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:58.143261    2547 scope.go:117] "RemoveContainer" containerID="a30b0c83bbd04427170aee7c6c498ec94cd839d897c2d7fc55c1b0e94b9f55aa"
Sep 10 14:54:58.143500 ip-10-0-130-249 kubenswrapper[2547]: E0910 14:54:58.143485    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a30b0c83bbd04427170aee7c6c498ec94cd839d897c2d7fc55c1b0e94b9f55aa\": container with ID starting with a30b0c83bbd04427170aee7c6c498ec94cd839d897c2d7fc55c1b0e94b9f55aa not found: ID does not exist" containerID="a30b0c83bbd04427170aee7c6c498ec94cd839d897c2d7fc55c1b0e94b9f55aa"
Sep 10 14:54:58.143556 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:58.143507    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a30b0c83bbd04427170aee7c6c498ec94cd839d897c2d7fc55c1b0e94b9f55aa"} err="failed to get container status \"a30b0c83bbd04427170aee7c6c498ec94cd839d897c2d7fc55c1b0e94b9f55aa\": rpc error: code = NotFound desc = could not find container \"a30b0c83bbd04427170aee7c6c498ec94cd839d897c2d7fc55c1b0e94b9f55aa\": container with ID starting with a30b0c83bbd04427170aee7c6c498ec94cd839d897c2d7fc55c1b0e94b9f55aa not found: ID does not exist"
Sep 10 14:54:58.147898 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:58.147876    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"]
Sep 10 14:54:58.155047 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:58.155028    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3477ac-openshift-default-79ccdb68cb-x4dvq"]
Sep 10 14:54:58.421252 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:54:58.421222    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" path="/var/lib/kubelet/pods/2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b/volumes"
Sep 10 14:56:16.228124 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:56:16.228079    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:56:44.322654 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:56:44.322610    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:56:52.517466 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:56:52.517412    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:56:58.522063 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:56:58.522024    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:57:05.122743 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:57:05.122711    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:57:19.214176 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:57:19.214141    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:57:26.924721 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:57:26.924658    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:58:54.522118 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:58:54.522036    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:59:02.436728 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:02.436682    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:59:08.723921 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:08.723883    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:59:27.187551 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.187508    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"]
Sep 10 14:59:27.188171 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188019    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="641472f0-b228-4074-b0c2-fa5718d9377a" containerName="istio-proxy"
Sep 10 14:59:27.188171 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188041    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="641472f0-b228-4074-b0c2-fa5718d9377a" containerName="istio-proxy"
Sep 10 14:59:27.188171 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188060    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c0a8407-94be-44f0-8fb6-6712daf341b8" containerName="istio-proxy"
Sep 10 14:59:27.188171 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188068    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c0a8407-94be-44f0-8fb6-6712daf341b8" containerName="istio-proxy"
Sep 10 14:59:27.188171 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188082    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c0a8407-94be-44f0-8fb6-6712daf341b8" containerName="istio-proxy"
Sep 10 14:59:27.188171 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188091    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c0a8407-94be-44f0-8fb6-6712daf341b8" containerName="istio-proxy"
Sep 10 14:59:27.188171 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188106    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" containerName="istio-proxy"
Sep 10 14:59:27.188171 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188115    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" containerName="istio-proxy"
Sep 10 14:59:27.188171 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188123    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" containerName="istio-proxy"
Sep 10 14:59:27.188171 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188131    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" containerName="istio-proxy"
Sep 10 14:59:27.188171 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188147    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd" containerName="maas-api"
Sep 10 14:59:27.188171 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188155    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd" containerName="maas-api"
Sep 10 14:59:27.188171 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188163    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e7e381-06c6-443e-8085-2164ffaac00b" containerName="maas-api"
Sep 10 14:59:27.188171 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188171    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e7e381-06c6-443e-8085-2164ffaac00b" containerName="maas-api"
Sep 10 14:59:27.188912 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188199    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94a28834-7eb7-4d6e-81cf-92f1ed51cba9" containerName="main"
Sep 10 14:59:27.188912 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188208    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="94a28834-7eb7-4d6e-81cf-92f1ed51cba9" containerName="main"
Sep 10 14:59:27.188912 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188298    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="75e7e381-06c6-443e-8085-2164ffaac00b" containerName="maas-api"
Sep 10 14:59:27.188912 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188312    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c0a8407-94be-44f0-8fb6-6712daf341b8" containerName="istio-proxy"
Sep 10 14:59:27.188912 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188323    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c0a8407-94be-44f0-8fb6-6712daf341b8" containerName="istio-proxy"
Sep 10 14:59:27.188912 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188335    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="94a28834-7eb7-4d6e-81cf-92f1ed51cba9" containerName="main"
Sep 10 14:59:27.188912 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188345    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="641472f0-b228-4074-b0c2-fa5718d9377a" containerName="istio-proxy"
Sep 10 14:59:27.188912 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188352    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="641472f0-b228-4074-b0c2-fa5718d9377a" containerName="istio-proxy"
Sep 10 14:59:27.188912 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188359    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" containerName="istio-proxy"
Sep 10 14:59:27.188912 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188368    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dc1b82f-dc7c-4eac-bd63-c2c73bfc931b" containerName="istio-proxy"
Sep 10 14:59:27.188912 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.188377    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1e413b9-fa27-4a86-9c4b-1fa6fd56eabd" containerName="maas-api"
Sep 10 14:59:27.191762 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.191741    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 14:59:27.194870 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.194852    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-ac0671f5-kserve-self-signed-certs\""
Sep 10 14:59:27.207157 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.207124    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"]
Sep 10 14:59:27.323395 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.323358    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea558a9-3395-4c55-b19b-4458789b3685-home\") pod \"e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489\" (UID: \"4ea558a9-3395-4c55-b19b-4458789b3685\") " pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 14:59:27.323563 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.323408    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea558a9-3395-4c55-b19b-4458789b3685-tls-certs\") pod \"e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489\" (UID: \"4ea558a9-3395-4c55-b19b-4458789b3685\") " pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 14:59:27.323563 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.323427    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea558a9-3395-4c55-b19b-4458789b3685-dshm\") pod \"e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489\" (UID: \"4ea558a9-3395-4c55-b19b-4458789b3685\") " pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 14:59:27.323563 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.323441    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea558a9-3395-4c55-b19b-4458789b3685-tmp-dir\") pod \"e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489\" (UID: \"4ea558a9-3395-4c55-b19b-4458789b3685\") " pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 14:59:27.323563 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.323482    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea558a9-3395-4c55-b19b-4458789b3685-model-cache\") pod \"e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489\" (UID: \"4ea558a9-3395-4c55-b19b-4458789b3685\") " pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 14:59:27.323736 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.323562    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77p4m\" (UniqueName: \"kubernetes.io/projected/4ea558a9-3395-4c55-b19b-4458789b3685-kube-api-access-77p4m\") pod \"e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489\" (UID: \"4ea558a9-3395-4c55-b19b-4458789b3685\") " pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 14:59:27.424971 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.424933    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea558a9-3395-4c55-b19b-4458789b3685-model-cache\") pod \"e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489\" (UID: \"4ea558a9-3395-4c55-b19b-4458789b3685\") " pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 14:59:27.425173 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.425025    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77p4m\" (UniqueName: \"kubernetes.io/projected/4ea558a9-3395-4c55-b19b-4458789b3685-kube-api-access-77p4m\") pod \"e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489\" (UID: \"4ea558a9-3395-4c55-b19b-4458789b3685\") " pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 14:59:27.425173 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.425059    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea558a9-3395-4c55-b19b-4458789b3685-home\") pod \"e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489\" (UID: \"4ea558a9-3395-4c55-b19b-4458789b3685\") " pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 14:59:27.425173 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.425082    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea558a9-3395-4c55-b19b-4458789b3685-tls-certs\") pod \"e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489\" (UID: \"4ea558a9-3395-4c55-b19b-4458789b3685\") " pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 14:59:27.425173 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.425104    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea558a9-3395-4c55-b19b-4458789b3685-dshm\") pod \"e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489\" (UID: \"4ea558a9-3395-4c55-b19b-4458789b3685\") " pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 14:59:27.425173 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.425127    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea558a9-3395-4c55-b19b-4458789b3685-tmp-dir\") pod \"e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489\" (UID: \"4ea558a9-3395-4c55-b19b-4458789b3685\") " pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 14:59:27.425442 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.425391    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea558a9-3395-4c55-b19b-4458789b3685-model-cache\") pod \"e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489\" (UID: \"4ea558a9-3395-4c55-b19b-4458789b3685\") " pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 14:59:27.425442 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.425402    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea558a9-3395-4c55-b19b-4458789b3685-home\") pod \"e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489\" (UID: \"4ea558a9-3395-4c55-b19b-4458789b3685\") " pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 14:59:27.425555 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.425516    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea558a9-3395-4c55-b19b-4458789b3685-tmp-dir\") pod \"e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489\" (UID: \"4ea558a9-3395-4c55-b19b-4458789b3685\") " pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 14:59:27.427284 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.427264    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea558a9-3395-4c55-b19b-4458789b3685-dshm\") pod \"e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489\" (UID: \"4ea558a9-3395-4c55-b19b-4458789b3685\") " pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 14:59:27.427554 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.427535    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea558a9-3395-4c55-b19b-4458789b3685-tls-certs\") pod \"e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489\" (UID: \"4ea558a9-3395-4c55-b19b-4458789b3685\") " pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 14:59:27.435495 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.435469    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77p4m\" (UniqueName: \"kubernetes.io/projected/4ea558a9-3395-4c55-b19b-4458789b3685-kube-api-access-77p4m\") pod \"e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489\" (UID: \"4ea558a9-3395-4c55-b19b-4458789b3685\") " pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 14:59:27.500759 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.500678    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 14:59:27.783640 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:27.783611    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"]
Sep 10 14:59:27.785012 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:59:27.784986    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ea558a9_3395_4c55_b19b_4458789b3685.slice/crio-8012b02c934f58c6305a26141b467a9c746a2dec7a24910941ef5d35342e9096 WatchSource:0}: Error finding container 8012b02c934f58c6305a26141b467a9c746a2dec7a24910941ef5d35342e9096: Status 404 returned error can't find the container with id 8012b02c934f58c6305a26141b467a9c746a2dec7a24910941ef5d35342e9096
Sep 10 14:59:28.502243 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:28.502205    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489" event={"ID":"4ea558a9-3395-4c55-b19b-4458789b3685","Type":"ContainerStarted","Data":"33ff63419457a8e3cf95eec4972fb90c1dc99fe0cc6863f242ce7caaabbe1b6a"}
Sep 10 14:59:28.502243 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:28.502248    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489" event={"ID":"4ea558a9-3395-4c55-b19b-4458789b3685","Type":"ContainerStarted","Data":"8012b02c934f58c6305a26141b467a9c746a2dec7a24910941ef5d35342e9096"}
Sep 10 14:59:28.524572 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:28.524523    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489" podStartSLOduration=1.524509293 podStartE2EDuration="1.524509293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:59:28.52295893 +0000 UTC m=+2380.564833410" watchObservedRunningTime="2026-09-10 14:59:28.524509293 +0000 UTC m=+2380.566383774"
Sep 10 14:59:29.857019 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:29.856978    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:59:33.649881 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.649848    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"]
Sep 10 14:59:33.650249 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.650182    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="641472f0-b228-4074-b0c2-fa5718d9377a" containerName="istio-proxy"
Sep 10 14:59:33.650249 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.650194    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="641472f0-b228-4074-b0c2-fa5718d9377a" containerName="istio-proxy"
Sep 10 14:59:33.653378 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.653357    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 14:59:33.656165 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.656145    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-ac0671f5-kserve-self-signed-certs\""
Sep 10 14:59:33.748414 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.748367    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"]
Sep 10 14:59:33.777734 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.777677    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-tls-certs\") pod \"e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv\" (UID: \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\") " pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 14:59:33.777872 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.777761    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kngf2\" (UniqueName: \"kubernetes.io/projected/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-kube-api-access-kngf2\") pod \"e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv\" (UID: \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\") " pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 14:59:33.777872 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.777781    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-model-cache\") pod \"e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv\" (UID: \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\") " pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 14:59:33.777872 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.777798    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-dshm\") pod \"e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv\" (UID: \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\") " pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 14:59:33.777872 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.777814    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-tmp-dir\") pod \"e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv\" (UID: \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\") " pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 14:59:33.777872 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.777857    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-home\") pod \"e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv\" (UID: \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\") " pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 14:59:33.878665 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.878632    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kngf2\" (UniqueName: \"kubernetes.io/projected/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-kube-api-access-kngf2\") pod \"e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv\" (UID: \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\") " pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 14:59:33.878665 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.878668    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-model-cache\") pod \"e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv\" (UID: \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\") " pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 14:59:33.878931 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.878689    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-dshm\") pod \"e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv\" (UID: \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\") " pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 14:59:33.878931 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.878734    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-tmp-dir\") pod \"e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv\" (UID: \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\") " pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 14:59:33.878931 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.878752    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-home\") pod \"e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv\" (UID: \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\") " pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 14:59:33.878931 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.878861    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-tls-certs\") pod \"e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv\" (UID: \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\") " pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 14:59:33.879200 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.879173    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-model-cache\") pod \"e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv\" (UID: \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\") " pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 14:59:33.879268 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.879240    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-tmp-dir\") pod \"e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv\" (UID: \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\") " pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 14:59:33.879315 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.879273    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-home\") pod \"e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv\" (UID: \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\") " pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 14:59:33.881122 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.881103    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-dshm\") pod \"e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv\" (UID: \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\") " pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 14:59:33.881425 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.881402    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-tls-certs\") pod \"e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv\" (UID: \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\") " pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 14:59:33.891390 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.891362    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kngf2\" (UniqueName: \"kubernetes.io/projected/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-kube-api-access-kngf2\") pod \"e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv\" (UID: \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\") " pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 14:59:33.963334 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:33.963308    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 14:59:34.258148 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:34.258061    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"]
Sep 10 14:59:34.267731 ip-10-0-130-249 kubenswrapper[2547]: W0910 14:59:34.267688    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad9ff94d_1e2b_4d7e_b21d_a4d93600dc53.slice/crio-654cd81f157d5cf22ebd96f87d1ba7327f5ab56e0346dee0ef8c1cdd93132fe6 WatchSource:0}: Error finding container 654cd81f157d5cf22ebd96f87d1ba7327f5ab56e0346dee0ef8c1cdd93132fe6: Status 404 returned error can't find the container with id 654cd81f157d5cf22ebd96f87d1ba7327f5ab56e0346dee0ef8c1cdd93132fe6
Sep 10 14:59:34.536102 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:34.536001    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv" event={"ID":"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53","Type":"ContainerStarted","Data":"0a3abcb02e816892e27959ff76dc094cdc6a1f7fc48fa7f0246bd2c7b4cee892"}
Sep 10 14:59:34.536102 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:34.536053    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv" event={"ID":"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53","Type":"ContainerStarted","Data":"654cd81f157d5cf22ebd96f87d1ba7327f5ab56e0346dee0ef8c1cdd93132fe6"}
Sep 10 14:59:34.561908 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:34.561859    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv" podStartSLOduration=1.561844717 podStartE2EDuration="1.561844717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:59:34.560784385 +0000 UTC m=+2386.602658866" watchObservedRunningTime="2026-09-10 14:59:34.561844717 +0000 UTC m=+2386.603719197"
Sep 10 14:59:37.501414 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:37.501367    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 14:59:37.501414 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:37.501420    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 14:59:37.515215 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:37.515185    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 14:59:37.563513 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:37.563478    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 14:59:42.946737 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:42.946682    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 14:59:43.963731 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:43.963677    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 14:59:43.963731 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:43.963737    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 14:59:43.975881 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:43.975851    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 14:59:44.600261 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:44.600231    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 14:59:48.551019 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:48.550985    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mg9d6_359d40e9-f428-4cc3-b4e4-991c081d664a/console-operator/1.log"
Sep 10 14:59:48.560526 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:48.560503    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mg9d6_359d40e9-f428-4cc3-b4e4-991c081d664a/console-operator/1.log"
Sep 10 14:59:50.936519 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:50.936487    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"]
Sep 10 14:59:51.822424 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:51.822392    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"]
Sep 10 14:59:54.064033 ip-10-0-130-249 kubenswrapper[2547]: I0910 14:59:54.063993    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 15:00:01.158599 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:01.158559    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 15:00:05.955728 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:05.955669    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489" podUID="4ea558a9-3395-4c55-b19b-4458789b3685" containerName="main" containerID="cri-o://33ff63419457a8e3cf95eec4972fb90c1dc99fe0cc6863f242ce7caaabbe1b6a" gracePeriod=45
Sep 10 15:00:06.212153 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.212088    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 15:00:06.266462 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.266424    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77p4m\" (UniqueName: \"kubernetes.io/projected/4ea558a9-3395-4c55-b19b-4458789b3685-kube-api-access-77p4m\") pod \"4ea558a9-3395-4c55-b19b-4458789b3685\" (UID: \"4ea558a9-3395-4c55-b19b-4458789b3685\") "
Sep 10 15:00:06.266646 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.266483    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea558a9-3395-4c55-b19b-4458789b3685-tls-certs\") pod \"4ea558a9-3395-4c55-b19b-4458789b3685\" (UID: \"4ea558a9-3395-4c55-b19b-4458789b3685\") "
Sep 10 15:00:06.266646 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.266526    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea558a9-3395-4c55-b19b-4458789b3685-tmp-dir\") pod \"4ea558a9-3395-4c55-b19b-4458789b3685\" (UID: \"4ea558a9-3395-4c55-b19b-4458789b3685\") "
Sep 10 15:00:06.266646 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.266570    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea558a9-3395-4c55-b19b-4458789b3685-dshm\") pod \"4ea558a9-3395-4c55-b19b-4458789b3685\" (UID: \"4ea558a9-3395-4c55-b19b-4458789b3685\") "
Sep 10 15:00:06.266646 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.266594    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea558a9-3395-4c55-b19b-4458789b3685-home\") pod \"4ea558a9-3395-4c55-b19b-4458789b3685\" (UID: \"4ea558a9-3395-4c55-b19b-4458789b3685\") "
Sep 10 15:00:06.266646 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.266625    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea558a9-3395-4c55-b19b-4458789b3685-model-cache\") pod \"4ea558a9-3395-4c55-b19b-4458789b3685\" (UID: \"4ea558a9-3395-4c55-b19b-4458789b3685\") "
Sep 10 15:00:06.266932 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.266822    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea558a9-3395-4c55-b19b-4458789b3685-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ea558a9-3395-4c55-b19b-4458789b3685" (UID: "4ea558a9-3395-4c55-b19b-4458789b3685"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:00:06.266996 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.266948    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea558a9-3395-4c55-b19b-4458789b3685-tmp-dir\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 15:00:06.267094 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.267071    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea558a9-3395-4c55-b19b-4458789b3685-home" (OuterVolumeSpecName: "home") pod "4ea558a9-3395-4c55-b19b-4458789b3685" (UID: "4ea558a9-3395-4c55-b19b-4458789b3685"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:00:06.267153 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.267088    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea558a9-3395-4c55-b19b-4458789b3685-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ea558a9-3395-4c55-b19b-4458789b3685" (UID: "4ea558a9-3395-4c55-b19b-4458789b3685"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:00:06.268711 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.268666    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea558a9-3395-4c55-b19b-4458789b3685-dshm" (OuterVolumeSpecName: "dshm") pod "4ea558a9-3395-4c55-b19b-4458789b3685" (UID: "4ea558a9-3395-4c55-b19b-4458789b3685"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:00:06.268812 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.268740    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea558a9-3395-4c55-b19b-4458789b3685-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ea558a9-3395-4c55-b19b-4458789b3685" (UID: "4ea558a9-3395-4c55-b19b-4458789b3685"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:00:06.268812 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.268749    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea558a9-3395-4c55-b19b-4458789b3685-kube-api-access-77p4m" (OuterVolumeSpecName: "kube-api-access-77p4m") pod "4ea558a9-3395-4c55-b19b-4458789b3685" (UID: "4ea558a9-3395-4c55-b19b-4458789b3685"). InnerVolumeSpecName "kube-api-access-77p4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:00:06.367635 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.367597    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea558a9-3395-4c55-b19b-4458789b3685-dshm\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 15:00:06.367635 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.367628    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea558a9-3395-4c55-b19b-4458789b3685-home\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 15:00:06.367635 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.367637    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea558a9-3395-4c55-b19b-4458789b3685-model-cache\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 15:00:06.367932 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.367647    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77p4m\" (UniqueName: \"kubernetes.io/projected/4ea558a9-3395-4c55-b19b-4458789b3685-kube-api-access-77p4m\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 15:00:06.367932 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.367656    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea558a9-3395-4c55-b19b-4458789b3685-tls-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 15:00:06.698982 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.698947    2547 generic.go:356] "Generic (PLEG): container finished" podID="4ea558a9-3395-4c55-b19b-4458789b3685" containerID="33ff63419457a8e3cf95eec4972fb90c1dc99fe0cc6863f242ce7caaabbe1b6a" exitCode=0
Sep 10 15:00:06.699198 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.698997    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"
Sep 10 15:00:06.699198 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.699030    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489" event={"ID":"4ea558a9-3395-4c55-b19b-4458789b3685","Type":"ContainerDied","Data":"33ff63419457a8e3cf95eec4972fb90c1dc99fe0cc6863f242ce7caaabbe1b6a"}
Sep 10 15:00:06.699198 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.699080    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489" event={"ID":"4ea558a9-3395-4c55-b19b-4458789b3685","Type":"ContainerDied","Data":"8012b02c934f58c6305a26141b467a9c746a2dec7a24910941ef5d35342e9096"}
Sep 10 15:00:06.699198 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.699100    2547 scope.go:117] "RemoveContainer" containerID="33ff63419457a8e3cf95eec4972fb90c1dc99fe0cc6863f242ce7caaabbe1b6a"
Sep 10 15:00:06.708198 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.708179    2547 scope.go:117] "RemoveContainer" containerID="33ff63419457a8e3cf95eec4972fb90c1dc99fe0cc6863f242ce7caaabbe1b6a"
Sep 10 15:00:06.708516 ip-10-0-130-249 kubenswrapper[2547]: E0910 15:00:06.708495    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33ff63419457a8e3cf95eec4972fb90c1dc99fe0cc6863f242ce7caaabbe1b6a\": container with ID starting with 33ff63419457a8e3cf95eec4972fb90c1dc99fe0cc6863f242ce7caaabbe1b6a not found: ID does not exist" containerID="33ff63419457a8e3cf95eec4972fb90c1dc99fe0cc6863f242ce7caaabbe1b6a"
Sep 10 15:00:06.708592 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.708525    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33ff63419457a8e3cf95eec4972fb90c1dc99fe0cc6863f242ce7caaabbe1b6a"} err="failed to get container status \"33ff63419457a8e3cf95eec4972fb90c1dc99fe0cc6863f242ce7caaabbe1b6a\": rpc error: code = NotFound desc = could not find container \"33ff63419457a8e3cf95eec4972fb90c1dc99fe0cc6863f242ce7caaabbe1b6a\": container with ID starting with 33ff63419457a8e3cf95eec4972fb90c1dc99fe0cc6863f242ce7caaabbe1b6a not found: ID does not exist"
Sep 10 15:00:06.723781 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.723748    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"]
Sep 10 15:00:06.735024 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.734995    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-ac0671f5-kserve-84fc484cff-9n489"]
Sep 10 15:00:06.841412 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:06.841371    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv" podUID="ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53" containerName="main" containerID="cri-o://0a3abcb02e816892e27959ff76dc094cdc6a1f7fc48fa7f0246bd2c7b4cee892" gracePeriod=45
Sep 10 15:00:07.098680 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.098644    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 15:00:07.175764 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.175722    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-home\") pod \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\" (UID: \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\") "
Sep 10 15:00:07.175973 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.175790    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kngf2\" (UniqueName: \"kubernetes.io/projected/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-kube-api-access-kngf2\") pod \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\" (UID: \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\") "
Sep 10 15:00:07.175973 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.175888    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-tmp-dir\") pod \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\" (UID: \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\") "
Sep 10 15:00:07.176101 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.175990    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-model-cache\") pod \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\" (UID: \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\") "
Sep 10 15:00:07.176101 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.176014    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-tls-certs\") pod \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\" (UID: \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\") "
Sep 10 15:00:07.176101 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.176037    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-dshm\") pod \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\" (UID: \"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53\") "
Sep 10 15:00:07.176101 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.176056    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-home" (OuterVolumeSpecName: "home") pod "ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53" (UID: "ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:00:07.176101 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.176059    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53" (UID: "ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:00:07.176389 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.176254    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53" (UID: "ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:00:07.176389 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.176320    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-home\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 15:00:07.176389 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.176333    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-tmp-dir\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 15:00:07.177963 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.177943    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-kube-api-access-kngf2" (OuterVolumeSpecName: "kube-api-access-kngf2") pod "ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53" (UID: "ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53"). InnerVolumeSpecName "kube-api-access-kngf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:00:07.178087 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.178070    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53" (UID: "ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:00:07.178158 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.178140    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-dshm" (OuterVolumeSpecName: "dshm") pod "ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53" (UID: "ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:00:07.277458 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.277364    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-model-cache\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 15:00:07.277458 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.277396    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-tls-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 15:00:07.277458 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.277405    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-dshm\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 15:00:07.277458 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.277421    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kngf2\" (UniqueName: \"kubernetes.io/projected/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53-kube-api-access-kngf2\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 15:00:07.706539 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.706505    2547 generic.go:356] "Generic (PLEG): container finished" podID="ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53" containerID="0a3abcb02e816892e27959ff76dc094cdc6a1f7fc48fa7f0246bd2c7b4cee892" exitCode=0
Sep 10 15:00:07.706760 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.706552    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"
Sep 10 15:00:07.706760 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.706557    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv" event={"ID":"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53","Type":"ContainerDied","Data":"0a3abcb02e816892e27959ff76dc094cdc6a1f7fc48fa7f0246bd2c7b4cee892"}
Sep 10 15:00:07.706760 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.706587    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv" event={"ID":"ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53","Type":"ContainerDied","Data":"654cd81f157d5cf22ebd96f87d1ba7327f5ab56e0346dee0ef8c1cdd93132fe6"}
Sep 10 15:00:07.706760 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.706603    2547 scope.go:117] "RemoveContainer" containerID="0a3abcb02e816892e27959ff76dc094cdc6a1f7fc48fa7f0246bd2c7b4cee892"
Sep 10 15:00:07.716544 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.716525    2547 scope.go:117] "RemoveContainer" containerID="0a3abcb02e816892e27959ff76dc094cdc6a1f7fc48fa7f0246bd2c7b4cee892"
Sep 10 15:00:07.716862 ip-10-0-130-249 kubenswrapper[2547]: E0910 15:00:07.716842    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3abcb02e816892e27959ff76dc094cdc6a1f7fc48fa7f0246bd2c7b4cee892\": container with ID starting with 0a3abcb02e816892e27959ff76dc094cdc6a1f7fc48fa7f0246bd2c7b4cee892 not found: ID does not exist" containerID="0a3abcb02e816892e27959ff76dc094cdc6a1f7fc48fa7f0246bd2c7b4cee892"
Sep 10 15:00:07.716927 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.716871    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3abcb02e816892e27959ff76dc094cdc6a1f7fc48fa7f0246bd2c7b4cee892"} err="failed to get container status \"0a3abcb02e816892e27959ff76dc094cdc6a1f7fc48fa7f0246bd2c7b4cee892\": rpc error: code = NotFound desc = could not find container \"0a3abcb02e816892e27959ff76dc094cdc6a1f7fc48fa7f0246bd2c7b4cee892\": container with ID starting with 0a3abcb02e816892e27959ff76dc094cdc6a1f7fc48fa7f0246bd2c7b4cee892 not found: ID does not exist"
Sep 10 15:00:07.740312 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.740267    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"]
Sep 10 15:00:07.742333 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:07.742305    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-ac0671f5-kserve-659f49ddc-pb8cv"]
Sep 10 15:00:08.427229 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.427197    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ea558a9-3395-4c55-b19b-4458789b3685" path="/var/lib/kubelet/pods/4ea558a9-3395-4c55-b19b-4458789b3685/volumes"
Sep 10 15:00:08.427601 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.427571    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53" path="/var/lib/kubelet/pods/ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53/volumes"
Sep 10 15:00:08.543120 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.543084    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 15:00:08.837848 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.837767    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m"]
Sep 10 15:00:08.838149 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.838134    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53" containerName="main"
Sep 10 15:00:08.838195 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.838151    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53" containerName="main"
Sep 10 15:00:08.838195 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.838161    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ea558a9-3395-4c55-b19b-4458789b3685" containerName="main"
Sep 10 15:00:08.838195 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.838167    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ea558a9-3395-4c55-b19b-4458789b3685" containerName="main"
Sep 10 15:00:08.838291 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.838232    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ea558a9-3395-4c55-b19b-4458789b3685" containerName="main"
Sep 10 15:00:08.838291 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.838242    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad9ff94d-1e2b-4d7e-b21d-a4d93600dc53" containerName="main"
Sep 10 15:00:08.842742 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.842719    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m"
Sep 10 15:00:08.849116 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.849087    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 15:00:08.849241 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.849118    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9r2v9\""
Sep 10 15:00:08.849311 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.849251    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 15:00:08.849372 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.849314    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 15:00:08.849372 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.849360    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 15:00:08.863117 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.863091    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m"]
Sep 10 15:00:08.875350 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.875323    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-68f46f5b6f-67c57"]
Sep 10 15:00:08.878725 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.878707    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68f46f5b6f-67c57"
Sep 10 15:00:08.889498 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.889466    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa-openshift-service-ca\") pod \"maas-api-6dd849cfd5-tth8m\" (UID: \"e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa\") " pod="odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m"
Sep 10 15:00:08.889618 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.889513    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa-metrics-certs\") pod \"maas-api-6dd849cfd5-tth8m\" (UID: \"e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa\") " pod="odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m"
Sep 10 15:00:08.889618 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.889569    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa-maas-api-tls\") pod \"maas-api-6dd849cfd5-tth8m\" (UID: \"e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa\") " pod="odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m"
Sep 10 15:00:08.889747 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.889718    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb7sk\" (UniqueName: \"kubernetes.io/projected/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa-kube-api-access-rb7sk\") pod \"maas-api-6dd849cfd5-tth8m\" (UID: \"e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa\") " pod="odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m"
Sep 10 15:00:08.919441 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.919404    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68f46f5b6f-67c57"]
Sep 10 15:00:08.990714 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.990658    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa-openshift-service-ca\") pod \"maas-api-6dd849cfd5-tth8m\" (UID: \"e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa\") " pod="odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m"
Sep 10 15:00:08.990888 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.990798    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa-metrics-certs\") pod \"maas-api-6dd849cfd5-tth8m\" (UID: \"e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa\") " pod="odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m"
Sep 10 15:00:08.990888 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.990822    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa-maas-api-tls\") pod \"maas-api-6dd849cfd5-tth8m\" (UID: \"e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa\") " pod="odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m"
Sep 10 15:00:08.990888 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.990871    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfzbg\" (UniqueName: \"kubernetes.io/projected/996e617f-09a0-4fd8-9a5c-10f80be11cc5-kube-api-access-kfzbg\") pod \"maas-api-68f46f5b6f-67c57\" (UID: \"996e617f-09a0-4fd8-9a5c-10f80be11cc5\") " pod="odh-ai-gateway-infra/maas-api-68f46f5b6f-67c57"
Sep 10 15:00:08.991057 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.990894    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996e617f-09a0-4fd8-9a5c-10f80be11cc5-metrics-certs\") pod \"maas-api-68f46f5b6f-67c57\" (UID: \"996e617f-09a0-4fd8-9a5c-10f80be11cc5\") " pod="odh-ai-gateway-infra/maas-api-68f46f5b6f-67c57"
Sep 10 15:00:08.991057 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.991001    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb7sk\" (UniqueName: \"kubernetes.io/projected/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa-kube-api-access-rb7sk\") pod \"maas-api-6dd849cfd5-tth8m\" (UID: \"e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa\") " pod="odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m"
Sep 10 15:00:08.991057 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.991039    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/996e617f-09a0-4fd8-9a5c-10f80be11cc5-maas-api-tls\") pod \"maas-api-68f46f5b6f-67c57\" (UID: \"996e617f-09a0-4fd8-9a5c-10f80be11cc5\") " pod="odh-ai-gateway-infra/maas-api-68f46f5b6f-67c57"
Sep 10 15:00:08.991213 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.991055    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/996e617f-09a0-4fd8-9a5c-10f80be11cc5-openshift-service-ca\") pod \"maas-api-68f46f5b6f-67c57\" (UID: \"996e617f-09a0-4fd8-9a5c-10f80be11cc5\") " pod="odh-ai-gateway-infra/maas-api-68f46f5b6f-67c57"
Sep 10 15:00:08.991488 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.991457    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa-openshift-service-ca\") pod \"maas-api-6dd849cfd5-tth8m\" (UID: \"e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa\") " pod="odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m"
Sep 10 15:00:08.993281 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.993263    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa-metrics-certs\") pod \"maas-api-6dd849cfd5-tth8m\" (UID: \"e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa\") " pod="odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m"
Sep 10 15:00:08.993428 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:08.993407    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa-maas-api-tls\") pod \"maas-api-6dd849cfd5-tth8m\" (UID: \"e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa\") " pod="odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m"
Sep 10 15:00:09.003494 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:09.003468    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb7sk\" (UniqueName: \"kubernetes.io/projected/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa-kube-api-access-rb7sk\") pod \"maas-api-6dd849cfd5-tth8m\" (UID: \"e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa\") " pod="odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m"
Sep 10 15:00:09.091906 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:09.091818    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfzbg\" (UniqueName: \"kubernetes.io/projected/996e617f-09a0-4fd8-9a5c-10f80be11cc5-kube-api-access-kfzbg\") pod \"maas-api-68f46f5b6f-67c57\" (UID: \"996e617f-09a0-4fd8-9a5c-10f80be11cc5\") " pod="odh-ai-gateway-infra/maas-api-68f46f5b6f-67c57"
Sep 10 15:00:09.091906 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:09.091862    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996e617f-09a0-4fd8-9a5c-10f80be11cc5-metrics-certs\") pod \"maas-api-68f46f5b6f-67c57\" (UID: \"996e617f-09a0-4fd8-9a5c-10f80be11cc5\") " pod="odh-ai-gateway-infra/maas-api-68f46f5b6f-67c57"
Sep 10 15:00:09.092120 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:09.091911    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/996e617f-09a0-4fd8-9a5c-10f80be11cc5-maas-api-tls\") pod \"maas-api-68f46f5b6f-67c57\" (UID: \"996e617f-09a0-4fd8-9a5c-10f80be11cc5\") " pod="odh-ai-gateway-infra/maas-api-68f46f5b6f-67c57"
Sep 10 15:00:09.092120 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:09.091934    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/996e617f-09a0-4fd8-9a5c-10f80be11cc5-openshift-service-ca\") pod \"maas-api-68f46f5b6f-67c57\" (UID: \"996e617f-09a0-4fd8-9a5c-10f80be11cc5\") " pod="odh-ai-gateway-infra/maas-api-68f46f5b6f-67c57"
Sep 10 15:00:09.092669 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:09.092641    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/996e617f-09a0-4fd8-9a5c-10f80be11cc5-openshift-service-ca\") pod \"maas-api-68f46f5b6f-67c57\" (UID: \"996e617f-09a0-4fd8-9a5c-10f80be11cc5\") " pod="odh-ai-gateway-infra/maas-api-68f46f5b6f-67c57"
Sep 10 15:00:09.094348 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:09.094328    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996e617f-09a0-4fd8-9a5c-10f80be11cc5-metrics-certs\") pod \"maas-api-68f46f5b6f-67c57\" (UID: \"996e617f-09a0-4fd8-9a5c-10f80be11cc5\") " pod="odh-ai-gateway-infra/maas-api-68f46f5b6f-67c57"
Sep 10 15:00:09.094510 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:09.094487    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/996e617f-09a0-4fd8-9a5c-10f80be11cc5-maas-api-tls\") pod \"maas-api-68f46f5b6f-67c57\" (UID: \"996e617f-09a0-4fd8-9a5c-10f80be11cc5\") " pod="odh-ai-gateway-infra/maas-api-68f46f5b6f-67c57"
Sep 10 15:00:09.108251 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:09.108219    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfzbg\" (UniqueName: \"kubernetes.io/projected/996e617f-09a0-4fd8-9a5c-10f80be11cc5-kube-api-access-kfzbg\") pod \"maas-api-68f46f5b6f-67c57\" (UID: \"996e617f-09a0-4fd8-9a5c-10f80be11cc5\") " pod="odh-ai-gateway-infra/maas-api-68f46f5b6f-67c57"
Sep 10 15:00:09.153858 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:09.153822    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m"
Sep 10 15:00:09.190648 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:09.190614    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68f46f5b6f-67c57"
Sep 10 15:00:09.439384 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:09.439358    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m"]
Sep 10 15:00:09.440844 ip-10-0-130-249 kubenswrapper[2547]: W0910 15:00:09.440817    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2dc2861_2c9e_43be_a5f4_ff0d134cb7fa.slice/crio-7d5e592aa7414347fd2e515e172c9f071efd57261a78ece6ab6e477bd7368cff WatchSource:0}: Error finding container 7d5e592aa7414347fd2e515e172c9f071efd57261a78ece6ab6e477bd7368cff: Status 404 returned error can't find the container with id 7d5e592aa7414347fd2e515e172c9f071efd57261a78ece6ab6e477bd7368cff
Sep 10 15:00:09.442194 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:09.442174    2547 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:00:09.483976 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:09.483950    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68f46f5b6f-67c57"]
Sep 10 15:00:09.485814 ip-10-0-130-249 kubenswrapper[2547]: W0910 15:00:09.485790    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996e617f_09a0_4fd8_9a5c_10f80be11cc5.slice/crio-49a8f416e55976df3bea552fd4bafeff3983cceaf928a60bdd41e5b02ff01983 WatchSource:0}: Error finding container 49a8f416e55976df3bea552fd4bafeff3983cceaf928a60bdd41e5b02ff01983: Status 404 returned error can't find the container with id 49a8f416e55976df3bea552fd4bafeff3983cceaf928a60bdd41e5b02ff01983
Sep 10 15:00:09.720652 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:09.720548    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68f46f5b6f-67c57" event={"ID":"996e617f-09a0-4fd8-9a5c-10f80be11cc5","Type":"ContainerStarted","Data":"49a8f416e55976df3bea552fd4bafeff3983cceaf928a60bdd41e5b02ff01983"}
Sep 10 15:00:09.721740 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:09.721715    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m" event={"ID":"e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa","Type":"ContainerStarted","Data":"7d5e592aa7414347fd2e515e172c9f071efd57261a78ece6ab6e477bd7368cff"}
Sep 10 15:00:11.733313 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:11.733275    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68f46f5b6f-67c57" event={"ID":"996e617f-09a0-4fd8-9a5c-10f80be11cc5","Type":"ContainerStarted","Data":"2154702d3d017c875280d41d8d5faf14afc9049f64f953b91ded4958ab85d34a"}
Sep 10 15:00:11.733783 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:11.733384    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-68f46f5b6f-67c57"
Sep 10 15:00:11.734996 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:11.734968    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m" event={"ID":"e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa","Type":"ContainerStarted","Data":"eb1dc190dbe8821bb75492c33fc3f6a4ee710c910d2722ecd27628c467afd2bb"}
Sep 10 15:00:11.735177 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:11.735156    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m"
Sep 10 15:00:11.756207 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:11.756167    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-68f46f5b6f-67c57" podStartSLOduration=2.488119759 podStartE2EDuration="3.756155719s" totalImagesPullingTime="1.26803596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:00:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:00:11.75411202 +0000 UTC m=+2423.795986503" watchObservedRunningTime="2026-09-10 15:00:11.756155719 +0000 UTC m=+2423.798030199"
Sep 10 15:00:11.776733 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:11.776668    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m" podStartSLOduration=2.258411866 podStartE2EDuration="3.776653147s" totalImagesPullingTime="1.518241281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:00:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:00:11.774647127 +0000 UTC m=+2423.816521610" watchObservedRunningTime="2026-09-10 15:00:11.776653147 +0000 UTC m=+2423.818527627"
Sep 10 15:00:16.343628 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:16.343593    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 15:00:17.745526 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:17.745491    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m"
Sep 10 15:00:17.745971 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:17.745923    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-68f46f5b6f-67c57"
Sep 10 15:00:17.873590 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:17.873560    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m"]
Sep 10 15:00:17.873857 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:17.873833    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m" podUID="e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa" containerName="maas-api" containerID="cri-o://eb1dc190dbe8821bb75492c33fc3f6a4ee710c910d2722ecd27628c467afd2bb" gracePeriod=30
Sep 10 15:00:18.120552 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:18.120527    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m"
Sep 10 15:00:18.271476 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:18.271445    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa-metrics-certs\") pod \"e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa\" (UID: \"e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa\") "
Sep 10 15:00:18.271671 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:18.271488    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rb7sk\" (UniqueName: \"kubernetes.io/projected/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa-kube-api-access-rb7sk\") pod \"e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa\" (UID: \"e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa\") "
Sep 10 15:00:18.271671 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:18.271537    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa-maas-api-tls\") pod \"e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa\" (UID: \"e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa\") "
Sep 10 15:00:18.271840 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:18.271686    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa-openshift-service-ca\") pod \"e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa\" (UID: \"e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa\") "
Sep 10 15:00:18.272064 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:18.272030    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa" (UID: "e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:00:18.273731 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:18.273681    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa-kube-api-access-rb7sk" (OuterVolumeSpecName: "kube-api-access-rb7sk") pod "e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa" (UID: "e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa"). InnerVolumeSpecName "kube-api-access-rb7sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:00:18.273731 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:18.273722    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa" (UID: "e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:00:18.273854 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:18.273744    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa" (UID: "e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:00:18.373059 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:18.372983    2547 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa-openshift-service-ca\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 15:00:18.373059 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:18.373012    2547 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa-metrics-certs\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 15:00:18.373059 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:18.373021    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb7sk\" (UniqueName: \"kubernetes.io/projected/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa-kube-api-access-rb7sk\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 15:00:18.373059 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:18.373030    2547 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa-maas-api-tls\") on node \"ip-10-0-130-249.ec2.internal\" DevicePath \"\""
Sep 10 15:00:18.770237 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:18.770201    2547 generic.go:356] "Generic (PLEG): container finished" podID="e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa" containerID="eb1dc190dbe8821bb75492c33fc3f6a4ee710c910d2722ecd27628c467afd2bb" exitCode=0
Sep 10 15:00:18.770690 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:18.770243    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m"
Sep 10 15:00:18.770690 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:18.770281    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m" event={"ID":"e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa","Type":"ContainerDied","Data":"eb1dc190dbe8821bb75492c33fc3f6a4ee710c910d2722ecd27628c467afd2bb"}
Sep 10 15:00:18.770690 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:18.770335    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m" event={"ID":"e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa","Type":"ContainerDied","Data":"7d5e592aa7414347fd2e515e172c9f071efd57261a78ece6ab6e477bd7368cff"}
Sep 10 15:00:18.770690 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:18.770360    2547 scope.go:117] "RemoveContainer" containerID="eb1dc190dbe8821bb75492c33fc3f6a4ee710c910d2722ecd27628c467afd2bb"
Sep 10 15:00:18.779359 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:18.779341    2547 scope.go:117] "RemoveContainer" containerID="eb1dc190dbe8821bb75492c33fc3f6a4ee710c910d2722ecd27628c467afd2bb"
Sep 10 15:00:18.779658 ip-10-0-130-249 kubenswrapper[2547]: E0910 15:00:18.779634    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb1dc190dbe8821bb75492c33fc3f6a4ee710c910d2722ecd27628c467afd2bb\": container with ID starting with eb1dc190dbe8821bb75492c33fc3f6a4ee710c910d2722ecd27628c467afd2bb not found: ID does not exist" containerID="eb1dc190dbe8821bb75492c33fc3f6a4ee710c910d2722ecd27628c467afd2bb"
Sep 10 15:00:18.779771 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:18.779667    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb1dc190dbe8821bb75492c33fc3f6a4ee710c910d2722ecd27628c467afd2bb"} err="failed to get container status \"eb1dc190dbe8821bb75492c33fc3f6a4ee710c910d2722ecd27628c467afd2bb\": rpc error: code = NotFound desc = could not find container \"eb1dc190dbe8821bb75492c33fc3f6a4ee710c910d2722ecd27628c467afd2bb\": container with ID starting with eb1dc190dbe8821bb75492c33fc3f6a4ee710c910d2722ecd27628c467afd2bb not found: ID does not exist"
Sep 10 15:00:18.799509 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:18.799475    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m"]
Sep 10 15:00:18.803808 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:18.803782    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6dd849cfd5-tth8m"]
Sep 10 15:00:20.421018 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:20.420982    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa" path="/var/lib/kubelet/pods/e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa/volumes"
Sep 10 15:00:23.336769 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:23.336731    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 15:00:31.764939 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:31.764900    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 15:00:37.185308 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:37.185270    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c6mlk"]
Sep 10 15:00:39.771968 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:39.771935    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-666d6cbfc5-6shpd_7adf78ec-2887-4fe9-97f9-75ed6e67a05b/authorino/0.log"
Sep 10 15:00:45.756981 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:45.756948    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-hrq7m_9db08a11-abdf-415a-8f96-2c363bb4d1e9/server/0.log"
Sep 10 15:00:45.878224 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:45.878191    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-gcjhf_447019ec-0133-4f4b-a530-b4fd4d693651/manager/0.log"
Sep 10 15:00:46.125210 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:46.125133    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-755bd7f47-vrmm7_2e2642b2-98c3-43ff-b568-4a0932c2eb6f/copy-manifests/0.log"
Sep 10 15:00:46.133332 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:46.133304    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-755bd7f47-vrmm7_2e2642b2-98c3-43ff-b568-4a0932c2eb6f/manager/0.log"
Sep 10 15:00:46.597091 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:46.597045    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-68f46f5b6f-67c57_996e617f-09a0-4fd8-9a5c-10f80be11cc5/maas-api/0.log"
Sep 10 15:00:47.807598 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:47.807570    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-666d6cbfc5-6shpd_7adf78ec-2887-4fe9-97f9-75ed6e67a05b/authorino/0.log"
Sep 10 15:00:48.482520 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:48.482485    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-c6mlk_9f409f9a-6846-4ea2-874e-401a2d71f247/limitador/0.log"
Sep 10 15:00:49.235784 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:49.235750    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f69648866-w6nq7_2891fd11-751e-473b-aba3-ff3570a980bc/kube-auth-proxy/0.log"
Sep 10 15:00:49.868785 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:49.868752    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76f86fcc44-d46qd_47ae46ed-3dac-4491-9f8c-098e66424ebc/router/0.log"
Sep 10 15:00:50.953715 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:50.953671    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-sjt57_5e81a218-b560-4d8f-892e-1ff9a5ffb2b4/main/0.log"
Sep 10 15:00:56.738373 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:56.738341    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b6dn7/must-gather-mb9tz"]
Sep 10 15:00:56.738847 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:56.738688    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa" containerName="maas-api"
Sep 10 15:00:56.738847 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:56.738714    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa" containerName="maas-api"
Sep 10 15:00:56.738847 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:56.738775    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2dc2861-2c9e-43be-a5f4-ff0d134cb7fa" containerName="maas-api"
Sep 10 15:00:56.741910 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:56.741892    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b6dn7/must-gather-mb9tz"
Sep 10 15:00:56.746095 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:56.746078    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b6dn7\"/\"default-dockercfg-86sqw\""
Sep 10 15:00:56.746629 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:56.746613    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b6dn7\"/\"kube-root-ca.crt\""
Sep 10 15:00:56.746751 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:56.746734    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b6dn7\"/\"openshift-service-ca.crt\""
Sep 10 15:00:56.760540 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:56.760518    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b6dn7/must-gather-mb9tz"]
Sep 10 15:00:56.797959 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:56.797934    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxqcm\" (UniqueName: \"kubernetes.io/projected/96bb77b3-6a39-463f-b09c-07db7b560676-kube-api-access-wxqcm\") pod \"must-gather-mb9tz\" (UID: \"96bb77b3-6a39-463f-b09c-07db7b560676\") " pod="openshift-must-gather-b6dn7/must-gather-mb9tz"
Sep 10 15:00:56.798097 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:56.797971    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/96bb77b3-6a39-463f-b09c-07db7b560676-must-gather-output\") pod \"must-gather-mb9tz\" (UID: \"96bb77b3-6a39-463f-b09c-07db7b560676\") " pod="openshift-must-gather-b6dn7/must-gather-mb9tz"
Sep 10 15:00:56.898915 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:56.898883    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxqcm\" (UniqueName: \"kubernetes.io/projected/96bb77b3-6a39-463f-b09c-07db7b560676-kube-api-access-wxqcm\") pod \"must-gather-mb9tz\" (UID: \"96bb77b3-6a39-463f-b09c-07db7b560676\") " pod="openshift-must-gather-b6dn7/must-gather-mb9tz"
Sep 10 15:00:56.899073 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:56.898928    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/96bb77b3-6a39-463f-b09c-07db7b560676-must-gather-output\") pod \"must-gather-mb9tz\" (UID: \"96bb77b3-6a39-463f-b09c-07db7b560676\") " pod="openshift-must-gather-b6dn7/must-gather-mb9tz"
Sep 10 15:00:56.899227 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:56.899210    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/96bb77b3-6a39-463f-b09c-07db7b560676-must-gather-output\") pod \"must-gather-mb9tz\" (UID: \"96bb77b3-6a39-463f-b09c-07db7b560676\") " pod="openshift-must-gather-b6dn7/must-gather-mb9tz"
Sep 10 15:00:56.911886 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:56.911867    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxqcm\" (UniqueName: \"kubernetes.io/projected/96bb77b3-6a39-463f-b09c-07db7b560676-kube-api-access-wxqcm\") pod \"must-gather-mb9tz\" (UID: \"96bb77b3-6a39-463f-b09c-07db7b560676\") " pod="openshift-must-gather-b6dn7/must-gather-mb9tz"
Sep 10 15:00:57.050804 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:57.050674    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b6dn7/must-gather-mb9tz"
Sep 10 15:00:57.343886 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:57.343861    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b6dn7/must-gather-mb9tz"]
Sep 10 15:00:57.346135 ip-10-0-130-249 kubenswrapper[2547]: W0910 15:00:57.346108    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96bb77b3_6a39_463f_b09c_07db7b560676.slice/crio-6932dacf836f5734b0b1061eacee8c5be4253e036bcf982b14e0749a71b5b2dd WatchSource:0}: Error finding container 6932dacf836f5734b0b1061eacee8c5be4253e036bcf982b14e0749a71b5b2dd: Status 404 returned error can't find the container with id 6932dacf836f5734b0b1061eacee8c5be4253e036bcf982b14e0749a71b5b2dd
Sep 10 15:00:57.975427 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:57.975392    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b6dn7/must-gather-mb9tz" event={"ID":"96bb77b3-6a39-463f-b09c-07db7b560676","Type":"ContainerStarted","Data":"6932dacf836f5734b0b1061eacee8c5be4253e036bcf982b14e0749a71b5b2dd"}
Sep 10 15:00:58.984622 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:58.984581    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b6dn7/must-gather-mb9tz" event={"ID":"96bb77b3-6a39-463f-b09c-07db7b560676","Type":"ContainerStarted","Data":"88c216071d7314da2ad6ec4fc13f6d54371abed19e610e00f3f0cf15846029f1"}
Sep 10 15:00:58.985262 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:58.985232    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b6dn7/must-gather-mb9tz" event={"ID":"96bb77b3-6a39-463f-b09c-07db7b560676","Type":"ContainerStarted","Data":"07d666d8a09c595de2340202574450337109186735941193ad77480c6a00dfcf"}
Sep 10 15:00:59.030749 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:00:59.030659    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-b6dn7/must-gather-mb9tz" podStartSLOduration=2.086372551 podStartE2EDuration="3.03063645s" totalImagesPullingTime="944.263899ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:00:59.030598185 +0000 UTC m=+2471.072472666" watchObservedRunningTime="2026-09-10 15:00:59.03063645 +0000 UTC m=+2471.072510929"
Sep 10 15:01:00.430970 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:00.430927    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m2v46_4e33264c-464d-41c9-8d74-b98e7b92b510/global-pull-secret-syncer/0.log"
Sep 10 15:01:00.682414 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:00.682175    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vp8vn_922dcf42-4b45-45fd-a011-f8a7eede8400/konnectivity-agent/0.log"
Sep 10 15:01:00.728601 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:00.728562    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-249.ec2.internal_7f4b9a9d29461bb847438bd99b7b31d4/haproxy/0.log"
Sep 10 15:01:05.116434 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:05.116390    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-666d6cbfc5-6shpd_7adf78ec-2887-4fe9-97f9-75ed6e67a05b/authorino/0.log"
Sep 10 15:01:05.618160 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:05.618122    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-c6mlk_9f409f9a-6846-4ea2-874e-401a2d71f247/limitador/0.log"
Sep 10 15:01:07.982295 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:07.982260    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-btcc8_d3d95df4-9fab-4b3c-a604-c856fd795d00/node-exporter/0.log"
Sep 10 15:01:08.014610 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:08.014585    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-btcc8_d3d95df4-9fab-4b3c-a604-c856fd795d00/kube-rbac-proxy/0.log"
Sep 10 15:01:08.051296 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:08.051264    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-btcc8_d3d95df4-9fab-4b3c-a604-c856fd795d00/init-textfile/0.log"
Sep 10 15:01:08.395012 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:08.394932    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-fxvnj_12b839b8-01d2-4eb6-ab72-67f26377a6d2/kube-rbac-proxy-main/0.log"
Sep 10 15:01:08.442094 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:08.442058    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-fxvnj_12b839b8-01d2-4eb6-ab72-67f26377a6d2/kube-rbac-proxy-self/0.log"
Sep 10 15:01:08.471482 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:08.471429    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-fxvnj_12b839b8-01d2-4eb6-ab72-67f26377a6d2/openshift-state-metrics/0.log"
Sep 10 15:01:08.979768 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:08.979734    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-ffd5c7b6b-4csxv_8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19/thanos-query/0.log"
Sep 10 15:01:09.023737 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:09.023685    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-ffd5c7b6b-4csxv_8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19/kube-rbac-proxy-web/0.log"
Sep 10 15:01:09.058963 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:09.058932    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-ffd5c7b6b-4csxv_8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19/kube-rbac-proxy/0.log"
Sep 10 15:01:09.089338 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:09.089307    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-ffd5c7b6b-4csxv_8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19/prom-label-proxy/0.log"
Sep 10 15:01:09.117900 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:09.117853    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-ffd5c7b6b-4csxv_8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19/kube-rbac-proxy-rules/0.log"
Sep 10 15:01:09.141084 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:09.141043    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-ffd5c7b6b-4csxv_8e764f9d-c7bb-4fb1-b293-ecd77ebb4a19/kube-rbac-proxy-metrics/0.log"
Sep 10 15:01:10.219684 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:10.219646    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv"]
Sep 10 15:01:10.223027 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:10.223004    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv"
Sep 10 15:01:10.238639 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:10.238609    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv"]
Sep 10 15:01:10.333204 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:10.333167    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e1c848db-6d04-40bc-8546-8795ce17158f-podres\") pod \"perf-node-gather-daemonset-jz9gv\" (UID: \"e1c848db-6d04-40bc-8546-8795ce17158f\") " pod="openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv"
Sep 10 15:01:10.333462 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:10.333441    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1c848db-6d04-40bc-8546-8795ce17158f-sys\") pod \"perf-node-gather-daemonset-jz9gv\" (UID: \"e1c848db-6d04-40bc-8546-8795ce17158f\") " pod="openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv"
Sep 10 15:01:10.333613 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:10.333598    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1c848db-6d04-40bc-8546-8795ce17158f-lib-modules\") pod \"perf-node-gather-daemonset-jz9gv\" (UID: \"e1c848db-6d04-40bc-8546-8795ce17158f\") " pod="openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv"
Sep 10 15:01:10.333816 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:10.333798    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e1c848db-6d04-40bc-8546-8795ce17158f-proc\") pod \"perf-node-gather-daemonset-jz9gv\" (UID: \"e1c848db-6d04-40bc-8546-8795ce17158f\") " pod="openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv"
Sep 10 15:01:10.333937 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:10.333923    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8bfw\" (UniqueName: \"kubernetes.io/projected/e1c848db-6d04-40bc-8546-8795ce17158f-kube-api-access-q8bfw\") pod \"perf-node-gather-daemonset-jz9gv\" (UID: \"e1c848db-6d04-40bc-8546-8795ce17158f\") " pod="openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv"
Sep 10 15:01:10.434514 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:10.434476    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1c848db-6d04-40bc-8546-8795ce17158f-lib-modules\") pod \"perf-node-gather-daemonset-jz9gv\" (UID: \"e1c848db-6d04-40bc-8546-8795ce17158f\") " pod="openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv"
Sep 10 15:01:10.434728 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:10.434542    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e1c848db-6d04-40bc-8546-8795ce17158f-proc\") pod \"perf-node-gather-daemonset-jz9gv\" (UID: \"e1c848db-6d04-40bc-8546-8795ce17158f\") " pod="openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv"
Sep 10 15:01:10.434728 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:10.434563    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8bfw\" (UniqueName: \"kubernetes.io/projected/e1c848db-6d04-40bc-8546-8795ce17158f-kube-api-access-q8bfw\") pod \"perf-node-gather-daemonset-jz9gv\" (UID: \"e1c848db-6d04-40bc-8546-8795ce17158f\") " pod="openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv"
Sep 10 15:01:10.434728 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:10.434594    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e1c848db-6d04-40bc-8546-8795ce17158f-podres\") pod \"perf-node-gather-daemonset-jz9gv\" (UID: \"e1c848db-6d04-40bc-8546-8795ce17158f\") " pod="openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv"
Sep 10 15:01:10.434728 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:10.434616    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1c848db-6d04-40bc-8546-8795ce17158f-sys\") pod \"perf-node-gather-daemonset-jz9gv\" (UID: \"e1c848db-6d04-40bc-8546-8795ce17158f\") " pod="openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv"
Sep 10 15:01:10.434728 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:10.434615    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e1c848db-6d04-40bc-8546-8795ce17158f-proc\") pod \"perf-node-gather-daemonset-jz9gv\" (UID: \"e1c848db-6d04-40bc-8546-8795ce17158f\") " pod="openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv"
Sep 10 15:01:10.434728 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:10.434691    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1c848db-6d04-40bc-8546-8795ce17158f-lib-modules\") pod \"perf-node-gather-daemonset-jz9gv\" (UID: \"e1c848db-6d04-40bc-8546-8795ce17158f\") " pod="openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv"
Sep 10 15:01:10.434728 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:10.434715    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1c848db-6d04-40bc-8546-8795ce17158f-sys\") pod \"perf-node-gather-daemonset-jz9gv\" (UID: \"e1c848db-6d04-40bc-8546-8795ce17158f\") " pod="openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv"
Sep 10 15:01:10.435149 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:10.434737    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e1c848db-6d04-40bc-8546-8795ce17158f-podres\") pod \"perf-node-gather-daemonset-jz9gv\" (UID: \"e1c848db-6d04-40bc-8546-8795ce17158f\") " pod="openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv"
Sep 10 15:01:10.445327 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:10.445299    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8bfw\" (UniqueName: \"kubernetes.io/projected/e1c848db-6d04-40bc-8546-8795ce17158f-kube-api-access-q8bfw\") pod \"perf-node-gather-daemonset-jz9gv\" (UID: \"e1c848db-6d04-40bc-8546-8795ce17158f\") " pod="openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv"
Sep 10 15:01:10.537406 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:10.537313    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv"
Sep 10 15:01:10.861893 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:10.861821    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv"]
Sep 10 15:01:10.867260 ip-10-0-130-249 kubenswrapper[2547]: W0910 15:01:10.866607    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode1c848db_6d04_40bc_8546_8795ce17158f.slice/crio-9cd809b43d2bf3948d56f695705791870c4fc822fdb1159f3539d00023125027 WatchSource:0}: Error finding container 9cd809b43d2bf3948d56f695705791870c4fc822fdb1159f3539d00023125027: Status 404 returned error can't find the container with id 9cd809b43d2bf3948d56f695705791870c4fc822fdb1159f3539d00023125027
Sep 10 15:01:10.956279 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:10.956246    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mg9d6_359d40e9-f428-4cc3-b4e4-991c081d664a/console-operator/1.log"
Sep 10 15:01:10.964159 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:10.964128    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mg9d6_359d40e9-f428-4cc3-b4e4-991c081d664a/console-operator/2.log"
Sep 10 15:01:11.084914 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:11.084871    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv" event={"ID":"e1c848db-6d04-40bc-8546-8795ce17158f","Type":"ContainerStarted","Data":"426ebf226265b24df564869c55c5bf7e8706f05d22548f1fbdb9652d412f3aac"}
Sep 10 15:01:11.085101 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:11.084942    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv" event={"ID":"e1c848db-6d04-40bc-8546-8795ce17158f","Type":"ContainerStarted","Data":"9cd809b43d2bf3948d56f695705791870c4fc822fdb1159f3539d00023125027"}
Sep 10 15:01:11.085792 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:11.085763    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv"
Sep 10 15:01:11.126355 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:11.126253    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv" podStartSLOduration=1.126237073 podStartE2EDuration="1.126237073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:01:11.125413813 +0000 UTC m=+2483.167288294" watchObservedRunningTime="2026-09-10 15:01:11.126237073 +0000 UTC m=+2483.168111547"
Sep 10 15:01:11.562073 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:11.562042    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-65df4d6d55-f2ncz_28fd9ffc-32ef-49c2-870b-c0b40ca95592/console/0.log"
Sep 10 15:01:11.605045 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:11.604984    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-dsgb8_af9ef9d2-bc7a-4d77-b0ac-ed3531249fcc/download-server/0.log"
Sep 10 15:01:13.303742 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:13.303688    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-957xl_54e8b08a-0bfa-4e3f-87e3-d6a5e886c798/dns/0.log"
Sep 10 15:01:13.326210 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:13.326169    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-957xl_54e8b08a-0bfa-4e3f-87e3-d6a5e886c798/kube-rbac-proxy/0.log"
Sep 10 15:01:13.513221 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:13.513188    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vz9vp_5a663e3c-6895-4085-aa41-1efd5d6ab208/dns-node-resolver/0.log"
Sep 10 15:01:14.101215 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:14.101181    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-46jxd_fe41a3f3-0ada-4163-9f76-74d8d30809ed/node-ca/0.log"
Sep 10 15:01:15.432056 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:15.432022    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f69648866-w6nq7_2891fd11-751e-473b-aba3-ff3570a980bc/kube-auth-proxy/0.log"
Sep 10 15:01:15.617866 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:15.617833    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76f86fcc44-d46qd_47ae46ed-3dac-4491-9f8c-098e66424ebc/router/0.log"
Sep 10 15:01:16.511619 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:16.511584    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2c9kh_d6415697-570a-4f1f-9317-7e774d60cc04/serve-healthcheck-canary/0.log"
Sep 10 15:01:17.274137 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:17.274098    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c2xkx_233305e5-df37-4854-bddb-a383fe612b09/kube-rbac-proxy/0.log"
Sep 10 15:01:17.308738 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:17.308685    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c2xkx_233305e5-df37-4854-bddb-a383fe612b09/exporter/0.log"
Sep 10 15:01:17.342309 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:17.342249    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c2xkx_233305e5-df37-4854-bddb-a383fe612b09/extractor/0.log"
Sep 10 15:01:18.110117 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:18.110088    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b6dn7/perf-node-gather-daemonset-jz9gv"
Sep 10 15:01:20.236467 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:20.236438    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-hrq7m_9db08a11-abdf-415a-8f96-2c363bb4d1e9/server/0.log"
Sep 10 15:01:20.363045 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:20.363012    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-gcjhf_447019ec-0133-4f4b-a530-b4fd4d693651/manager/0.log"
Sep 10 15:01:20.447200 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:20.447167    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-755bd7f47-vrmm7_2e2642b2-98c3-43ff-b568-4a0932c2eb6f/manager/0.log"
Sep 10 15:01:20.468631 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:20.468598    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-755bd7f47-vrmm7_2e2642b2-98c3-43ff-b568-4a0932c2eb6f/copy-manifests/0.log"
Sep 10 15:01:20.841880 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:20.841845    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f69648866-w6nq7_2891fd11-751e-473b-aba3-ff3570a980bc/kube-auth-proxy/0.log"
Sep 10 15:01:20.978303 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:20.978266    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76f86fcc44-d46qd_47ae46ed-3dac-4491-9f8c-098e66424ebc/router/0.log"
Sep 10 15:01:21.951541 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:21.951508    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8f5db98bf-rjb2s_e2c33dd1-671a-4509-9c50-205b60962d1c/manager/0.log"
Sep 10 15:01:22.743562 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:22.743526    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f69648866-w6nq7_2891fd11-751e-473b-aba3-ff3570a980bc/kube-auth-proxy/0.log"
Sep 10 15:01:22.881781 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:22.881752    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76f86fcc44-d46qd_47ae46ed-3dac-4491-9f8c-098e66424ebc/router/0.log"
Sep 10 15:01:27.075225 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:27.075164    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-gxwmg_6e2e1ff5-7d0f-4de4-8b6c-83d8942f9ea6/migrator/0.log"
Sep 10 15:01:27.102182 ip-10-0-130-249 kubenswrapper[2547]: I0910 15:01:27.102154    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-gxwmg_6e2e1ff5-7d0f-4de4-8b6c-83d8942f9ea6/graceful-termination/0.log"