Sep 10 20:56:54.555336 ip-10-0-135-205 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 20:56:54.555349 ip-10-0-135-205 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 20:56:54.555358 ip-10-0-135-205 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 20:56:54.555587 ip-10-0-135-205 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 20:57:04.804578 ip-10-0-135-205 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 10 20:57:04.804595 ip-10-0-135-205 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e5d9e8fb00df4676b28bd137c0526b61 --
Sep 10 20:59:00.219746 ip-10-0-135-205 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 20:59:00.617122 ip-10-0-135-205 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 20:59:00.617122 ip-10-0-135-205 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 20:59:00.617122 ip-10-0-135-205 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 20:59:00.617122 ip-10-0-135-205 kubenswrapper[2584]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 20:59:00.617122 ip-10-0-135-205 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 20:59:00.618521 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.618438    2584 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 20:59:00.622144 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622129    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:59:00.622144 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622143    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:59:00.622209 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622148    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:59:00.622209 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622151    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:59:00.622209 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622154    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:59:00.622209 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622157    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:59:00.622209 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622160    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:59:00.622209 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622163    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:59:00.622209 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622166    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:59:00.622209 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622169    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:59:00.622209 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622172    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:59:00.622209 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622175    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:59:00.622209 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622177    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:59:00.622209 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622180    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:59:00.622209 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622183    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:59:00.622209 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622186    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:59:00.622209 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622188    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:59:00.622209 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622191    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:59:00.622209 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622194    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:59:00.622209 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622196    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:59:00.622209 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622199    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:59:00.622209 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622202    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:59:00.622705 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622204    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:59:00.622705 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622207    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:59:00.622705 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622210    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:59:00.622705 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622212    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:59:00.622705 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622216    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:59:00.622705 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622219    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:59:00.622705 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622222    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:59:00.622705 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622225    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:59:00.622705 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622227    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:59:00.622705 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622230    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:59:00.622705 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622232    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:59:00.622705 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622237    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:59:00.622705 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622241    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:59:00.622705 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622244    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:59:00.622705 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622247    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:59:00.622705 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622249    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:59:00.622705 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622252    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:59:00.622705 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622254    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:59:00.622705 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622257    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:59:00.623175 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622259    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:59:00.623175 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622262    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:59:00.623175 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622264    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:59:00.623175 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622267    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:59:00.623175 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622269    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:59:00.623175 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622272    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:59:00.623175 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622275    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:59:00.623175 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622279    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:59:00.623175 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622283    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:59:00.623175 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622285    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:59:00.623175 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622288    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:59:00.623175 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622290    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:59:00.623175 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622293    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:59:00.623175 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622295    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:59:00.623175 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622299    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:59:00.623175 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622301    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:59:00.623175 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622303    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:59:00.623175 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622306    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:59:00.623175 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622309    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:59:00.623674 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622311    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:59:00.623674 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622315    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:59:00.623674 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622317    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:59:00.623674 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622320    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:59:00.623674 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622322    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:59:00.623674 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622325    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:59:00.623674 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622327    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:59:00.623674 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622330    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:59:00.623674 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622333    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:59:00.623674 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622336    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:59:00.623674 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622339    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:59:00.623674 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622341    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:59:00.623674 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622343    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:59:00.623674 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622346    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:59:00.623674 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622348    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:59:00.623674 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622351    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:59:00.623674 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622353    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:59:00.623674 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622356    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:59:00.623674 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622358    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:59:00.623674 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622361    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:59:00.624157 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622363    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:59:00.624157 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622367    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:59:00.624157 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622370    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:59:00.624157 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622373    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:59:00.624157 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622376    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:59:00.624157 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622379    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:59:00.624157 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622382    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:59:00.624157 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622388    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:59:00.624157 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622390    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:59:00.624157 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622394    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:59:00.624157 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622396    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:59:00.624157 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622399    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:59:00.624157 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622402    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:59:00.624157 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622405    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:59:00.624157 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622407    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:59:00.624157 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622410    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:59:00.624157 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622412    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:59:00.624157 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622415    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:59:00.624157 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.622417    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:59:00.624157 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.623992    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:59:00.624662 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624000    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:59:00.624662 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624003    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:59:00.624662 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624007    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:59:00.624662 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624010    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:59:00.624662 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624013    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:59:00.624662 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624016    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:59:00.624662 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624019    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:59:00.624662 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624021    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:59:00.624662 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624024    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:59:00.624662 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624027    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:59:00.624662 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624030    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:59:00.624662 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624033    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:59:00.624662 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624035    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:59:00.624662 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624038    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:59:00.624662 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624040    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:59:00.624662 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624043    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:59:00.624662 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624046    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:59:00.624662 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624049    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:59:00.624662 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624051    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:59:00.624662 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624055    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:59:00.625158 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624059    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:59:00.625158 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624062    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:59:00.625158 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624065    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:59:00.625158 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624067    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:59:00.625158 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624070    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:59:00.625158 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624073    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:59:00.625158 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624076    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:59:00.625158 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624078    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:59:00.625158 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624081    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:59:00.625158 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624083    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:59:00.625158 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624086    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:59:00.625158 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624088    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:59:00.625158 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624091    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:59:00.625158 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624094    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:59:00.625158 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624097    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:59:00.625158 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624100    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:59:00.625158 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624102    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:59:00.625158 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624105    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:59:00.625158 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624107    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:59:00.625158 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624110    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:59:00.625679 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624112    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:59:00.625679 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624115    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:59:00.625679 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624117    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:59:00.625679 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624120    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:59:00.625679 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624123    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:59:00.625679 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624125    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:59:00.625679 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624128    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:59:00.625679 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624131    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:59:00.625679 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624141    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:59:00.625679 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624144    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:59:00.625679 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624147    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:59:00.625679 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624150    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:59:00.625679 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624153    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:59:00.625679 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624156    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:59:00.625679 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624158    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:59:00.625679 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624161    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:59:00.625679 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624165    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:59:00.625679 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624167    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:59:00.625679 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624170    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:59:00.625679 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624173    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:59:00.626167 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624176    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:59:00.626167 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624178    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:59:00.626167 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624181    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:59:00.626167 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624183    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:59:00.626167 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624186    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:59:00.626167 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624188    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:59:00.626167 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624191    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:59:00.626167 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624194    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:59:00.626167 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624197    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:59:00.626167 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624199    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:59:00.626167 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624202    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:59:00.626167 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624205    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:59:00.626167 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624208    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:59:00.626167 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624210    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:59:00.626167 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624213    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:59:00.626167 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624215    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:59:00.626167 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624218    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:59:00.626167 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624221    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:59:00.626167 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624223    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:59:00.626657 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624228    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:59:00.626657 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624231    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:59:00.626657 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624234    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:59:00.626657 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624237    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:59:00.626657 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624241    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:59:00.626657 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624244    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:59:00.626657 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624246    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:59:00.626657 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624249    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:59:00.626657 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624252    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:59:00.626657 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624255    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:59:00.626657 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624257    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:59:00.626657 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624260    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:59:00.626657 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624275    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:59:00.626657 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624279    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:59:00.626657 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624282    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:59:00.626657 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624285    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:59:00.626657 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624288    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:59:00.626657 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624292    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:59:00.627108 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624296    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:59:00.627108 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624676    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:59:00.627108 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624682    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:59:00.627108 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624685    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:59:00.627108 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624690    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:59:00.627108 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624694    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:59:00.627108 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624697    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:59:00.627108 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624700    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:59:00.627108 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624703    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:59:00.627108 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624705    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:59:00.627108 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624708    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:59:00.627108 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624711    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:59:00.627108 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624713    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:59:00.627108 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624716    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:59:00.627108 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624718    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:59:00.627108 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624721    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:59:00.627108 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624723    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:59:00.627108 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624726    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:59:00.627108 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624729    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:59:00.627108 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624732    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:59:00.627627 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624735    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:59:00.627627 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624737    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:59:00.627627 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624740    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:59:00.627627 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624742    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:59:00.627627 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624745    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:59:00.627627 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624747    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:59:00.627627 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624750    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:59:00.627627 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624752    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:59:00.627627 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624755    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:59:00.627627 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624757    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:59:00.627627 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624760    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:59:00.627627 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624762    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:59:00.627627 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624765    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:59:00.627627 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624767    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:59:00.627627 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624775    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:59:00.627627 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624778    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:59:00.627627 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624780    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:59:00.627627 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624783    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:59:00.627627 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624785    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:59:00.628087 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624797    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:59:00.628087 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624801    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:59:00.628087 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624804    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:59:00.628087 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624807    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:59:00.628087 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624810    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:59:00.628087 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624813    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:59:00.628087 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624816    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:59:00.628087 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624819    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:59:00.628087 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624821    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:59:00.628087 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624824    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:59:00.628087 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624826    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:59:00.628087 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624829    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:59:00.628087 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624832    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:59:00.628087 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624835    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:59:00.628087 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624837    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:59:00.628087 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624840    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:59:00.628087 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624842    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:59:00.628087 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624845    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:59:00.628087 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624847    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:59:00.628087 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624851    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:59:00.628595 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624853    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:59:00.628595 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624856    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:59:00.628595 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624858    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:59:00.628595 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624861    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:59:00.628595 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624863    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:59:00.628595 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624866    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:59:00.628595 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624868    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:59:00.628595 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624870    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:59:00.628595 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624875    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:59:00.628595 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624878    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:59:00.628595 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624881    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:59:00.628595 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624885    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:59:00.628595 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624888    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:59:00.628595 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624891    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:59:00.628595 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624893    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:59:00.628595 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624896    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:59:00.628595 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624899    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:59:00.628595 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624902    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:59:00.628595 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624904    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:59:00.629091 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624907    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:59:00.629091 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624909    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:59:00.629091 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624912    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:59:00.629091 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624915    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:59:00.629091 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624917    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:59:00.629091 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624920    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:59:00.629091 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624922    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:59:00.629091 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624925    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:59:00.629091 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624927    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:59:00.629091 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624930    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:59:00.629091 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624932    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:59:00.629091 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624934    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:59:00.629091 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624937    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:59:00.629091 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624940    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:59:00.629091 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624942    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:59:00.629091 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624946    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:59:00.629091 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624948    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:59:00.629091 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624951    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:59:00.629091 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624953    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:59:00.629091 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624955    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:59:00.629584 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624958    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:59:00.629584 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.624961    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:59:00.629584 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625029    2584 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 20:59:00.629584 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625060    2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 20:59:00.629584 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625066    2584 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 20:59:00.629584 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625070    2584 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 20:59:00.629584 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625074    2584 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 20:59:00.629584 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625077    2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 20:59:00.629584 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625081    2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 20:59:00.629584 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625085    2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 20:59:00.629584 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625125    2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 20:59:00.629584 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625133    2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 20:59:00.629584 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625138    2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 20:59:00.629584 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625142    2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 20:59:00.629584 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625146    2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 20:59:00.629584 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625150    2584 flags.go:64] FLAG: --cgroup-root=""
Sep 10 20:59:00.629584 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625153    2584 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 20:59:00.629584 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625157    2584 flags.go:64] FLAG: --client-ca-file=""
Sep 10 20:59:00.629584 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625161    2584 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 20:59:00.629584 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625164    2584 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 20:59:00.629584 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625170    2584 flags.go:64] FLAG: --cluster-domain=""
Sep 10 20:59:00.629584 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625173    2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 20:59:00.629584 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625176    2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 20:59:00.630164 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625180    2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 20:59:00.630164 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625183    2584 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 20:59:00.630164 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625188    2584 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 20:59:00.630164 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625191    2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 20:59:00.630164 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625194    2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 20:59:00.630164 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625198    2584 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 20:59:00.630164 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625201    2584 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 20:59:00.630164 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625204    2584 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 20:59:00.630164 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625207    2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 20:59:00.630164 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625211    2584 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 20:59:00.630164 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625214    2584 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 20:59:00.630164 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625218    2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 20:59:00.630164 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625221    2584 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 20:59:00.630164 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625225    2584 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 20:59:00.630164 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625228    2584 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 20:59:00.630164 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625231    2584 flags.go:64] FLAG: --enable-server="true"
Sep 10 20:59:00.630164 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625233    2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 20:59:00.630164 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625239    2584 flags.go:64] FLAG: --event-burst="100"
Sep 10 20:59:00.630164 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625242    2584 flags.go:64] FLAG: --event-qps="50"
Sep 10 20:59:00.630164 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625245    2584 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 20:59:00.630164 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625250    2584 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 20:59:00.630164 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625253    2584 flags.go:64] FLAG: --eviction-hard=""
Sep 10 20:59:00.630164 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625257    2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 20:59:00.630164 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625260    2584 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 20:59:00.630753 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625263    2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 20:59:00.630753 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625266    2584 flags.go:64] FLAG: --eviction-soft=""
Sep 10 20:59:00.630753 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625269    2584 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 20:59:00.630753 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625272    2584 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 20:59:00.630753 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625275    2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 20:59:00.630753 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625278    2584 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 20:59:00.630753 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625281    2584 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 20:59:00.630753 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625284    2584 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 20:59:00.630753 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625287    2584 flags.go:64] FLAG: --feature-gates=""
Sep 10 20:59:00.630753 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625290    2584 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 20:59:00.630753 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625294    2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 20:59:00.630753 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625297    2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 20:59:00.630753 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625300    2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 20:59:00.630753 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625306    2584 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 20:59:00.630753 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625309    2584 flags.go:64] FLAG: --help="false"
Sep 10 20:59:00.630753 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625313    2584 flags.go:64] FLAG: --hostname-override="ip-10-0-135-205.ec2.internal"
Sep 10 20:59:00.630753 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625316    2584 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 20:59:00.630753 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625319    2584 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 20:59:00.630753 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625322    2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 20:59:00.630753 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625325    2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 20:59:00.630753 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625329    2584 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 20:59:00.630753 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625332    2584 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 20:59:00.630753 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625335    2584 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625338    2584 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625341    2584 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625344    2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625347    2584 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625350    2584 flags.go:64] FLAG: --kube-reserved=""
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625353    2584 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625356    2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625360    2584 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625362    2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625365    2584 flags.go:64] FLAG: --lock-file=""
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625368    2584 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625371    2584 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625374    2584 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625380    2584 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625382    2584 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625385    2584 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625388    2584 flags.go:64] FLAG: --logging-format="text"
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625391    2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625394    2584 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625397    2584 flags.go:64] FLAG: --manifest-url=""
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625401    2584 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625405    2584 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625410    2584 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625414    2584 flags.go:64] FLAG: --max-pods="110"
Sep 10 20:59:00.631346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625417    2584 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 20:59:00.631993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625431    2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 20:59:00.631993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625435    2584 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 20:59:00.631993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625438    2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 20:59:00.631993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625441    2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 20:59:00.631993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625444    2584 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 20:59:00.631993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625447    2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 20:59:00.631993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625461    2584 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 20:59:00.631993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625463    2584 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 20:59:00.631993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625467    2584 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 20:59:00.631993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625470    2584 flags.go:64] FLAG: --pod-cidr=""
Sep 10 20:59:00.631993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625473    2584 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 10 20:59:00.631993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625479    2584 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 20:59:00.631993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625482    2584 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 20:59:00.631993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625485    2584 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 20:59:00.631993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625488    2584 flags.go:64] FLAG: --port="10250"
Sep 10 20:59:00.631993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625492    2584 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 20:59:00.631993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625495    2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02e1d0830cef6d969"
Sep 10 20:59:00.631993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625498    2584 flags.go:64] FLAG: --qos-reserved=""
Sep 10 20:59:00.631993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625501    2584 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 20:59:00.631993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625504    2584 flags.go:64] FLAG: --register-node="true"
Sep 10 20:59:00.631993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625507    2584 flags.go:64] FLAG: --register-with-taints=""
Sep 10 20:59:00.631993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625511    2584 flags.go:64] FLAG: --registry-burst="10"
Sep 10 20:59:00.631993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625514    2584 flags.go:64] FLAG: --registry-qps="5"
Sep 10 20:59:00.631993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625516    2584 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625520    2584 flags.go:64] FLAG: --reserved-memory=""
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625524    2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625527    2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625530    2584 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625533    2584 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625538    2584 flags.go:64] FLAG: --runonce="false"
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625541    2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625544    2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625547    2584 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625550    2584 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625553    2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625556    2584 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625559    2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625562    2584 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625567    2584 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625570    2584 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625573    2584 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625576    2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625579    2584 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625582    2584 flags.go:64] FLAG: --system-cgroups=""
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625584    2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625591    2584 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625594    2584 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625597    2584 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 20:59:00.632674 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625602    2584 flags.go:64] FLAG: --tls-min-version=""
Sep 10 20:59:00.633663 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625605    2584 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 20:59:00.633663 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625608    2584 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 20:59:00.633663 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625611    2584 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 20:59:00.633663 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625614    2584 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 20:59:00.633663 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625617    2584 flags.go:64] FLAG: --v="2"
Sep 10 20:59:00.633663 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625621    2584 flags.go:64] FLAG: --version="false"
Sep 10 20:59:00.633663 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625625    2584 flags.go:64] FLAG: --vmodule=""
Sep 10 20:59:00.633663 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625629    2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 20:59:00.633663 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.625632    2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 20:59:00.633663 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625754    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:59:00.633663 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625758    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:59:00.633663 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625761    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:59:00.633663 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625766    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:59:00.633663 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625769    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:59:00.633663 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625772    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:59:00.633663 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625774    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:59:00.633663 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625778    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:59:00.633663 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625780    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:59:00.633663 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625783    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:59:00.633663 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625786    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:59:00.633663 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625788    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:59:00.633663 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625792    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:59:00.634514 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625795    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:59:00.634514 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625798    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:59:00.634514 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625801    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:59:00.634514 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625804    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:59:00.634514 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625806    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:59:00.634514 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625809    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:59:00.634514 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625812    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:59:00.634514 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625814    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:59:00.634514 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625819    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:59:00.634514 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625823    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:59:00.634514 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625826    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:59:00.634514 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625829    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:59:00.634514 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625832    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:59:00.634514 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625834    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:59:00.634514 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625837    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:59:00.634514 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625840    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:59:00.634514 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625842    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:59:00.634514 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625845    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:59:00.634514 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625848    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:59:00.634990 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625850    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:59:00.634990 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625853    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:59:00.634990 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625856    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:59:00.634990 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625859    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:59:00.634990 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625862    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:59:00.634990 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625865    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:59:00.634990 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625868    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:59:00.634990 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625870    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:59:00.634990 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625873    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:59:00.634990 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625875    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:59:00.634990 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625878    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:59:00.634990 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625881    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:59:00.634990 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625885    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:59:00.634990 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625888    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:59:00.634990 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625892    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:59:00.634990 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625896    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:59:00.634990 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625899    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:59:00.634990 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625902    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:59:00.634990 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625904    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:59:00.635486 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625907    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:59:00.635486 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625910    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:59:00.635486 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625912    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:59:00.635486 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625915    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:59:00.635486 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625918    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:59:00.635486 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625921    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:59:00.635486 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625923    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:59:00.635486 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625926    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:59:00.635486 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625928    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:59:00.635486 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625931    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:59:00.635486 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625933    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:59:00.635486 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625936    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:59:00.635486 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625939    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:59:00.635486 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625941    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:59:00.635486 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625944    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:59:00.635486 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625947    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:59:00.635486 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625951    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:59:00.635486 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625954    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:59:00.635486 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625956    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:59:00.635486 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625959    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:59:00.635982 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625962    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:59:00.635982 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625965    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:59:00.635982 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625967    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:59:00.635982 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625970    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:59:00.635982 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625972    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:59:00.635982 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625976    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:59:00.635982 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625980    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:59:00.635982 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625984    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:59:00.635982 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625987    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:59:00.635982 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625990    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:59:00.635982 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625993    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:59:00.635982 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.625996    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:59:00.635982 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.626000    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:59:00.635982 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.626004    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:59:00.635982 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.626008    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:59:00.635982 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.626012    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:59:00.635982 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.626016    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:59:00.635982 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.626020    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:59:00.635982 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.626024    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:59:00.635982 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.626027    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:59:00.636494 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.626031    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:59:00.636494 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.626035    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:59:00.636494 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.626039    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:59:00.636494 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.626042    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:59:00.636494 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.626045    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:59:00.636494 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.626048    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:59:00.636494 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.626051    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:59:00.636494 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.626053    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:59:00.636494 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.626625    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 20:59:00.636494 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.634616    2584 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 20:59:00.636494 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.634631    2584 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 20:59:00.636494 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.635512    2584 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 20:59:00.636494 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.635521    2584 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 20:59:00.636494 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635576    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:59:00.636494 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635580    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:59:00.636872 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635584    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:59:00.636872 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635588    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:59:00.636872 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635591    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:59:00.636872 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635594    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:59:00.636872 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635597    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:59:00.636872 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635600    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:59:00.636872 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635603    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:59:00.636872 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635606    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:59:00.636872 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635608    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:59:00.636872 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635611    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:59:00.636872 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635614    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:59:00.636872 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635616    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:59:00.636872 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635619    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:59:00.636872 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635622    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:59:00.636872 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635624    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:59:00.636872 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635627    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:59:00.636872 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635630    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:59:00.636872 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635632    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:59:00.636872 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635635    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:59:00.636872 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635638    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:59:00.637348 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635642    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:59:00.637348 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635646    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:59:00.637348 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635649    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:59:00.637348 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635652    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:59:00.637348 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635656    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:59:00.637348 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635659    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:59:00.637348 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635662    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:59:00.637348 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635665    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:59:00.637348 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635668    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:59:00.637348 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635671    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:59:00.637348 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635674    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:59:00.637348 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635676    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:59:00.637348 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635679    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:59:00.637348 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635682    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:59:00.637348 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635684    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:59:00.637348 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635687    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:59:00.637348 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635690    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:59:00.637348 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635692    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:59:00.637348 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635695    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:59:00.637917 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635699    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:59:00.637917 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635702    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:59:00.637917 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635705    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:59:00.637917 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635708    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:59:00.637917 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635710    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:59:00.637917 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635713    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:59:00.637917 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635716    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:59:00.637917 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635719    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:59:00.637917 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635721    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:59:00.637917 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635724    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:59:00.637917 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635727    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:59:00.637917 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635729    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:59:00.637917 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635732    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:59:00.637917 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635735    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:59:00.637917 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635737    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:59:00.637917 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635740    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:59:00.637917 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635743    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:59:00.637917 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635745    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:59:00.637917 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635748    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:59:00.637917 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635750    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:59:00.638399 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635754    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:59:00.638399 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635757    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:59:00.638399 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635759    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:59:00.638399 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635762    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:59:00.638399 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635764    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:59:00.638399 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635767    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:59:00.638399 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635770    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:59:00.638399 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635772    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:59:00.638399 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635775    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:59:00.638399 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635777    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:59:00.638399 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635780    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:59:00.638399 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635782    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:59:00.638399 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635785    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:59:00.638399 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635787    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:59:00.638399 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635790    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:59:00.638399 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635792    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:59:00.638399 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635795    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:59:00.638399 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635799    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:59:00.638399 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635803    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:59:00.638882 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635806    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:59:00.638882 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635809    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:59:00.638882 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635812    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:59:00.638882 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635815    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:59:00.638882 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635818    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:59:00.638882 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635821    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:59:00.638882 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635823    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:59:00.638882 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635826    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:59:00.638882 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635829    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:59:00.638882 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635831    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:59:00.638882 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635834    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:59:00.638882 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635836    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:59:00.638882 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635839    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:59:00.638882 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635842    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:59:00.638882 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635844    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:59:00.638882 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635847    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:59:00.638882 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635850    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:59:00.638882 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635852    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:59:00.638882 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635855    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:59:00.639439 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.635861    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 20:59:00.639439 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635967    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:59:00.639439 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635972    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:59:00.639439 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635976    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:59:00.639439 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635979    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:59:00.639439 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635982    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:59:00.639439 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635985    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:59:00.639439 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635988    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:59:00.639439 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635990    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:59:00.639439 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635993    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:59:00.639439 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635996    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:59:00.639439 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.635999    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:59:00.639439 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636001    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:59:00.639439 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636004    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:59:00.639439 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636007    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:59:00.639822 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636010    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:59:00.639822 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636012    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:59:00.639822 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636015    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:59:00.639822 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636018    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:59:00.639822 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636021    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:59:00.639822 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636023    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:59:00.639822 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636026    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:59:00.639822 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636029    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:59:00.639822 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636031    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:59:00.639822 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636034    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:59:00.639822 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636037    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:59:00.639822 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636039    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:59:00.639822 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636042    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:59:00.639822 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636044    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:59:00.639822 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636047    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:59:00.639822 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636050    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:59:00.639822 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636052    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:59:00.639822 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636055    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:59:00.639822 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636057    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:59:00.639822 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636060    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:59:00.640413 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636062    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:59:00.640413 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636065    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:59:00.640413 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636068    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:59:00.640413 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636070    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:59:00.640413 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636073    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:59:00.640413 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636076    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:59:00.640413 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636080    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:59:00.640413 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636083    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:59:00.640413 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636086    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:59:00.640413 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636088    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:59:00.640413 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636091    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:59:00.640413 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636095    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:59:00.640413 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636099    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:59:00.640413 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636101    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:59:00.640413 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636104    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:59:00.640413 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636107    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:59:00.640413 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636110    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:59:00.640413 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636113    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:59:00.640413 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636116    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:59:00.640413 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636119    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:59:00.640942 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636121    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:59:00.640942 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636124    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:59:00.640942 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636127    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:59:00.640942 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636130    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:59:00.640942 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636139    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:59:00.640942 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636142    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:59:00.640942 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636145    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:59:00.640942 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636148    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:59:00.640942 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636151    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:59:00.640942 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636153    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:59:00.640942 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636156    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:59:00.640942 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636159    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:59:00.640942 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636161    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:59:00.640942 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636164    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:59:00.640942 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636166    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:59:00.640942 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636169    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:59:00.640942 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636171    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:59:00.640942 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636174    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:59:00.640942 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636176    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:59:00.641388 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636179    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:59:00.641388 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636182    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:59:00.641388 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636184    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:59:00.641388 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636188    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:59:00.641388 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636191    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:59:00.641388 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636196    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:59:00.641388 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636198    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:59:00.641388 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636201    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:59:00.641388 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636204    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:59:00.641388 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636207    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:59:00.641388 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636209    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:59:00.641388 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636212    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:59:00.641388 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636214    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:59:00.641388 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636217    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:59:00.641388 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636220    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:59:00.641388 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636222    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:59:00.641388 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636225    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:59:00.641388 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636228    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:59:00.641388 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636230    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:59:00.642137 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636233    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:59:00.642137 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636235    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:59:00.642137 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636238    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:59:00.642137 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636240    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:59:00.642137 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636243    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:59:00.642137 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636245    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:59:00.642137 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:00.636248    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:59:00.642137 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.636253    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 20:59:00.642137 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.636345    2584 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 20:59:00.642137 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.638878    2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 20:59:00.642137 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.639810    2584 server.go:1019] "Starting client certificate rotation"
Sep 10 20:59:00.642137 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.639928    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 20:59:00.642137 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.639978    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 20:59:00.642694 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.642247    2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 20:59:00.643405 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.643386    2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 20:59:00.658565 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.658550    2584 log.go:25] "Validated CRI v1 runtime API"
Sep 10 20:59:00.661326 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.661307    2584 log.go:25] "Validated CRI v1 image API"
Sep 10 20:59:00.662119 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.662105    2584 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 20:59:00.664521 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.664507    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 20:59:00.665541 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.665524    2584 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c32d4c26-a877-4d56-a71a-2d959f56b980:/dev/nvme0n1p3 fcf57b83-f872-4d09-9cf0-987695869d62:/dev/nvme0n1p4]
Sep 10 20:59:00.665605 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.665540    2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 20:59:00.672397 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.672282    2584 manager.go:216] Machine: {Timestamp:2026-09-10 20:59:00.669692404 +0000 UTC m=+0.334717198 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105699 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2182add30c1390eef23f4f160519aa SystemUUID:ec2182ad-d30c-1390-eef2-3f4f160519aa BootID:e5d9e8fb-00df-4676-b28b-d137c0526b61 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:96:93:47:4d:b1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:96:93:47:4d:b1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:4f:af:10:bb:0d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 20:59:00.672397 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.672393    2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 20:59:00.672513 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.672501    2584 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 20:59:00.675834 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.675809    2584 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 20:59:00.675963 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.675836    2584 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-205.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 20:59:00.676519 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.676508    2584 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 20:59:00.676558 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.676522    2584 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 20:59:00.676558 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.676529    2584 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 20:59:00.677536 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.677526    2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 20:59:00.678259 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.678249    2584 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 20:59:00.681959 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.681947    2584 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 20:59:00.682067 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.682058    2584 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 20:59:00.682095 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.682087    2584 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 20:59:00.682125 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.682100    2584 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 20:59:00.682153 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.682126    2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 20:59:00.682153 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.682144    2584 kubelet.go:398] "Adding apiserver pod source"
Sep 10 20:59:00.682214 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.682161    2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 20:59:00.683347 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.683328    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 20:59:00.683442 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.683361    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 20:59:00.686869 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.686852    2584 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 20:59:00.688458 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.688439    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2vd5l"
Sep 10 20:59:00.688851 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.688836    2584 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 20:59:00.688912 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.688865    2584 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 20:59:00.690496 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.690485    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 20:59:00.690546 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.690501    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 20:59:00.690546 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.690507    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 20:59:00.690546 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.690513    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 20:59:00.690546 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.690522    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 20:59:00.690546 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.690527    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 20:59:00.690546 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.690540    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 20:59:00.690546 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.690544    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 20:59:00.690724 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.690551    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 20:59:00.690724 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.690556    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 20:59:00.690724 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.690564    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 20:59:00.690724 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.690570    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 20:59:00.691315 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.691307    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 20:59:00.691315 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.691315    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 20:59:00.694016 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.693998    2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-205.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 20:59:00.694596 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:00.694572    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-205.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 20:59:00.694667 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:00.694618    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 20:59:00.694910 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.694896    2584 server.go:1268] "Started kubelet"
Sep 10 20:59:00.694988 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.694962    2584 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 20:59:00.695037 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.694973    2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 20:59:00.695037 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.695028    2584 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 20:59:00.695700 ip-10-0-135-205 systemd[1]: Started Kubernetes Kubelet.
Sep 10 20:59:00.695975 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.695958    2584 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 20:59:00.696392 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.696371    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2vd5l"
Sep 10 20:59:00.696654 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.696642    2584 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 20:59:00.700301 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:00.699651    2584 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 20:59:00.701513 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:00.700462    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-205.ec2.internal.18d411452806a0a6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-205.ec2.internal,UID:ip-10-0-135-205.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-205.ec2.internal,},FirstTimestamp:2026-09-10 20:59:00.694868134 +0000 UTC m=+0.359892926,LastTimestamp:2026-09-10 20:59:00.694868134 +0000 UTC m=+0.359892926,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-205.ec2.internal,}"
Sep 10 20:59:00.701651 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.701635    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 20:59:00.702223 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.702207    2584 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 20:59:00.702345 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:00.702327    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 10 20:59:00.702793 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.702778    2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 20:59:00.702793 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.702779    2584 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 20:59:00.702918 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.702805    2584 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 20:59:00.702918 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.702906    2584 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 20:59:00.702918 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.702913    2584 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 20:59:00.703196 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.703179    2584 factory.go:55] Registering systemd factory
Sep 10 20:59:00.703239 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.703200    2584 factory.go:223] Registration of the systemd container factory successfully
Sep 10 20:59:00.703393 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.703379    2584 factory.go:153] Registering CRI-O factory
Sep 10 20:59:00.703450 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.703394    2584 factory.go:223] Registration of the crio container factory successfully
Sep 10 20:59:00.703491 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.703453    2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 20:59:00.703491 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.703468    2584 factory.go:103] Registering Raw factory
Sep 10 20:59:00.703491 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.703477    2584 manager.go:1196] Started watching for new ooms in manager
Sep 10 20:59:00.703898 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.703883    2584 manager.go:349] Starting recovery of all containers
Sep 10 20:59:00.715527 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.715494    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:59:00.716160 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.716130    2584 manager.go:354] Recovery completed
Sep 10 20:59:00.719308 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:00.719286    2584 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-205.ec2.internal\" not found" node="ip-10-0-135-205.ec2.internal"
Sep 10 20:59:00.721407 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.721393    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:59:00.724337 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.724321    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:59:00.724417 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.724349    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:59:00.724417 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.724362    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:59:00.724882 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.724863    2584 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 20:59:00.724882 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.724875    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:59:00.724882 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.724880    2584 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 20:59:00.725044 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.724898    2584 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 20:59:00.725621 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.725607    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:59:00.725686 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.725630    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:59:00.725686 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.725643    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:59:00.727146 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.727129    2584 policy_none.go:49] "None policy: Start"
Sep 10 20:59:00.727146 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.727148    2584 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 20:59:00.727286 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.727161    2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 20:59:00.729113 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.729101    2584 policy_none.go:47] "Start"
Sep 10 20:59:00.765012 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.764993    2584 manager.go:337] "Starting Device Plugin manager"
Sep 10 20:59:00.780817 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:00.765032    2584 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 20:59:00.780817 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.765045    2584 server.go:87] "Starting device plugin registration server"
Sep 10 20:59:00.780817 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.765276    2584 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 20:59:00.780817 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.765288    2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 20:59:00.780817 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.765444    2584 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 20:59:00.780817 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.765509    2584 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 20:59:00.780817 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.765517    2584 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 20:59:00.780817 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:00.767523    2584 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 20:59:00.780817 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:00.767562    2584 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 10 20:59:00.832505 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.832478    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 20:59:00.833618 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.833595    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 20:59:00.833692 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.833653    2584 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 20:59:00.833692 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.833684    2584 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 20:59:00.833788 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:00.833733    2584 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 20:59:00.836360 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.836343    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:59:00.866231 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.866212    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:59:00.867106 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.867090    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:59:00.867219 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.867125    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:59:00.867219 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.867140    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:59:00.867219 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.867168    2584 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-205.ec2.internal"
Sep 10 20:59:00.873822 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.873807    2584 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-205.ec2.internal"
Sep 10 20:59:00.873893 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:00.873828    2584 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-205.ec2.internal\": node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 10 20:59:00.892116 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:00.892092    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 10 20:59:00.934368 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.934337    2584 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-205.ec2.internal"]
Sep 10 20:59:00.934465 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.934406    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:59:00.935215 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.935200    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:59:00.935290 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.935230    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:59:00.935290 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.935244    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:59:00.937665 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.937650    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:59:00.937792 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.937777    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal"
Sep 10 20:59:00.937837 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.937809    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:59:00.938326 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.938311    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:59:00.938408 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.938337    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:59:00.938408 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.938356    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:59:00.938408 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.938370    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:59:00.938408 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.938392    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:59:00.938408 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.938403    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:59:00.940718 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.940700    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-205.ec2.internal"
Sep 10 20:59:00.940807 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.940726    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:59:00.941365 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.941350    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:59:00.941445 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.941381    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:59:00.941445 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:00.941397    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:59:00.963984 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:00.963968    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-205.ec2.internal\" not found" node="ip-10-0-135-205.ec2.internal"
Sep 10 20:59:00.968246 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:00.968233    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-205.ec2.internal\" not found" node="ip-10-0-135-205.ec2.internal"
Sep 10 20:59:00.993104 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:00.993085    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 10 20:59:01.004920 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:01.004906    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/77a4c4cd902dc458728fadabb7f791eb-config\") pod \"kube-apiserver-proxy-ip-10-0-135-205.ec2.internal\" (UID: \"77a4c4cd902dc458728fadabb7f791eb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-205.ec2.internal"
Sep 10 20:59:01.004986 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:01.004927    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0f127f626c5949a577288cc12509aa1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal\" (UID: \"f0f127f626c5949a577288cc12509aa1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal"
Sep 10 20:59:01.004986 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:01.004944    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0f127f626c5949a577288cc12509aa1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal\" (UID: \"f0f127f626c5949a577288cc12509aa1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal"
Sep 10 20:59:01.093975 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:01.093945    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 10 20:59:01.105284 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:01.105268    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0f127f626c5949a577288cc12509aa1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal\" (UID: \"f0f127f626c5949a577288cc12509aa1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal"
Sep 10 20:59:01.105335 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:01.105290    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0f127f626c5949a577288cc12509aa1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal\" (UID: \"f0f127f626c5949a577288cc12509aa1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal"
Sep 10 20:59:01.105335 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:01.105308    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/77a4c4cd902dc458728fadabb7f791eb-config\") pod \"kube-apiserver-proxy-ip-10-0-135-205.ec2.internal\" (UID: \"77a4c4cd902dc458728fadabb7f791eb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-205.ec2.internal"
Sep 10 20:59:01.105396 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:01.105343    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0f127f626c5949a577288cc12509aa1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal\" (UID: \"f0f127f626c5949a577288cc12509aa1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal"
Sep 10 20:59:01.105396 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:01.105350    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0f127f626c5949a577288cc12509aa1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal\" (UID: \"f0f127f626c5949a577288cc12509aa1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal"
Sep 10 20:59:01.105516 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:01.105415    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/77a4c4cd902dc458728fadabb7f791eb-config\") pod \"kube-apiserver-proxy-ip-10-0-135-205.ec2.internal\" (UID: \"77a4c4cd902dc458728fadabb7f791eb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-205.ec2.internal"
Sep 10 20:59:01.194794 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:01.194702    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 10 20:59:01.266139 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:01.266103    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal"
Sep 10 20:59:01.271198 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:01.270806    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-205.ec2.internal"
Sep 10 20:59:01.295554 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:01.295524    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 10 20:59:01.396222 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:01.396188    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 10 20:59:01.500114 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:01.500026    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 10 20:59:01.601405 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:01.601363    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 10 20:59:01.640765 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:01.640734    2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 20:59:01.641187 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:01.640878    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 20:59:01.641187 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:01.640927    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 20:59:01.701003 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:01.700964    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 20:54:01 +0000 UTC" deadline="2028-03-05 05:10:12.232957376 +0000 UTC"
Sep 10 20:59:01.701003 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:01.700995    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12992h11m10.531967374s"
Sep 10 20:59:01.702584 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:01.702565    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 10 20:59:01.742577 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:01.742549    2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:59:01.803431 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:01.803399    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal"
Sep 10 20:59:01.822100 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:01.822072    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 20:59:01.824106 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:01.824088    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-205.ec2.internal"
Sep 10 20:59:01.833620 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:01.833600    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 20:59:01.947063 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:01.947016    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f127f626c5949a577288cc12509aa1.slice/crio-272bd805f9299902de6b013ca0edfb985b41c21aaf573a4f368972bdbf79bd4d WatchSource:0}: Error finding container 272bd805f9299902de6b013ca0edfb985b41c21aaf573a4f368972bdbf79bd4d: Status 404 returned error can't find the container with id 272bd805f9299902de6b013ca0edfb985b41c21aaf573a4f368972bdbf79bd4d
Sep 10 20:59:01.947659 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:01.947625    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77a4c4cd902dc458728fadabb7f791eb.slice/crio-b50097d2c34901a936f0f3c3dca42c4cfb737abb20a8715e89e59f2fbbf9d314 WatchSource:0}: Error finding container b50097d2c34901a936f0f3c3dca42c4cfb737abb20a8715e89e59f2fbbf9d314: Status 404 returned error can't find the container with id b50097d2c34901a936f0f3c3dca42c4cfb737abb20a8715e89e59f2fbbf9d314
Sep 10 20:59:01.951630 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:01.951615    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 20:59:02.007925 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.007899    2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:59:02.684078 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.684045    2584 apiserver.go:52] "Watching apiserver"
Sep 10 20:59:02.693978 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.693952    2584 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 20:59:02.696023 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.695978    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-jskzm","openshift-network-diagnostics/network-check-target-6hc86","openshift-network-operator/iptables-alerter-6tbjl","kube-system/konnectivity-agent-r282d","kube-system/kube-apiserver-proxy-ip-10-0-135-205.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76","openshift-cluster-node-tuning-operator/tuned-vmvb8","openshift-image-registry/node-ca-xswnp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal","openshift-multus/multus-nrl66","openshift-ovn-kubernetes/ovnkube-node-qbwqs","kube-system/global-pull-secret-syncer-k8v5s","openshift-dns/node-resolver-wprl7","openshift-multus/multus-additional-cni-plugins-c6cwt"]
Sep 10 20:59:02.698758 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.698733    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.702707 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.701758    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.702707 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.702675    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 20:59:02.703339 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.703317    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 20:59:02.703464 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.703415    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xslb9\""
Sep 10 20:59:02.703586 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.703467    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:59:02.704298 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.704017    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 20:59:02.704298 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.704059    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 20:59:02.704298 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.704135    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 20:59:02.704298 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.704263    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2sptm\""
Sep 10 20:59:02.704739 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.704330    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 20:59:02.704849 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.704827    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 20:59:02.704990 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.704968    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6tbjl"
Sep 10 20:59:02.705426 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.705403    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 20:59:02.707019 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.706993    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 20:59:02.707317 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.707295    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r282d"
Sep 10 20:59:02.707416 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.707368    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4gzfv\""
Sep 10 20:59:02.707672 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.707651    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:59:02.707756 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.707727    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 20:59:02.709581 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.709383    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 20:59:02.709650 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.709639    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 20:59:02.709650 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.709645    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cfp2h\""
Sep 10 20:59:02.710980 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.710960    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c6cwt"
Sep 10 20:59:02.713338 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713321    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76"
Sep 10 20:59:02.713338 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713325    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hdc4\" (UniqueName: \"kubernetes.io/projected/dbed4e47-f58b-451c-8205-283e0392aec5-kube-api-access-6hdc4\") pod \"iptables-alerter-6tbjl\" (UID: \"dbed4e47-f58b-451c-8205-283e0392aec5\") " pod="openshift-network-operator/iptables-alerter-6tbjl"
Sep 10 20:59:02.713516 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713359    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-etc-modprobe-d\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.713516 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713377    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-etc-openvswitch\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.713516 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713393    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-run-ovn\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.713516 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713409    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-host-run-ovn-kubernetes\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.713516 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713425    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-host-cni-netd\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.713516 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713439    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c1477d2e-55a6-415f-85d0-b325df3980f1-ovnkube-config\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.713516 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713477    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1477d2e-55a6-415f-85d0-b325df3980f1-ovn-node-metrics-cert\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.713516 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713514    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dbed4e47-f58b-451c-8205-283e0392aec5-host-slash\") pod \"iptables-alerter-6tbjl\" (UID: \"dbed4e47-f58b-451c-8205-283e0392aec5\") " pod="openshift-network-operator/iptables-alerter-6tbjl"
Sep 10 20:59:02.713920 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713539    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5fn8\" (UniqueName: \"kubernetes.io/projected/22f578b7-ad26-41b3-b9df-8b61f1770171-kube-api-access-m5fn8\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.713920 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713563    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-systemd-units\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.713920 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713587    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-var-lib-openvswitch\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.713920 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713613    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c1477d2e-55a6-415f-85d0-b325df3980f1-env-overrides\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.713920 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713639    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-etc-kubernetes\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.713920 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713665    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-var-lib-kubelet\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.713920 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713718    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.713920 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713326    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 20:59:02.713920 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713781    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7zzq\" (UniqueName: \"kubernetes.io/projected/c1477d2e-55a6-415f-85d0-b325df3980f1-kube-api-access-p7zzq\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.713920 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713802    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-etc-sysctl-d\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.713920 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713817    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-etc-sysctl-conf\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.713920 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713854    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-run\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.713920 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713890    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-run-systemd\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.713920 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713917    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-host-cni-bin\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.714642 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713954    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-etc-systemd\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.714642 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.713982    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-host\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.714642 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.714015    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-host-run-netns\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.714642 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.714039    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-node-log\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.714642 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.714065    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-log-socket\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.714642 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.714089    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-sys\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.714642 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.714116    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dbed4e47-f58b-451c-8205-283e0392aec5-iptables-alerter-script\") pod \"iptables-alerter-6tbjl\" (UID: \"dbed4e47-f58b-451c-8205-283e0392aec5\") " pod="openshift-network-operator/iptables-alerter-6tbjl"
Sep 10 20:59:02.714642 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.714154    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-etc-sysconfig\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.714642 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.714184    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6cthj\""
Sep 10 20:59:02.714642 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.714196    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-lib-modules\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.714642 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.714195    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 20:59:02.714642 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.714232    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22f578b7-ad26-41b3-b9df-8b61f1770171-tmp\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.714642 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.714244    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 20:59:02.714642 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.714284    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-host-slash\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.714642 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.714316    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-run-openvswitch\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.714642 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.714324    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 20:59:02.714642 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.714340    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/22f578b7-ad26-41b3-b9df-8b61f1770171-etc-tuned\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.714642 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.714365    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-host-kubelet\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.714642 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.714389    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c1477d2e-55a6-415f-85d0-b325df3980f1-ovnkube-script-lib\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.715372 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.714486    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 20:59:02.715372 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.714596    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 20:59:02.715741 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.715718    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ddn26\""
Sep 10 20:59:02.715900 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.715878    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 20:59:02.716026 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.715982    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 20:59:02.716117 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.716094    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 20:59:02.716391 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.716373    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xswnp"
Sep 10 20:59:02.718700 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.718436    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qkcgh\""
Sep 10 20:59:02.718788 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.718487    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 20:59:02.718856 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.718623    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 20:59:02.718950 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.718668    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 20:59:02.719222 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.719199    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wprl7"
Sep 10 20:59:02.721077 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.721053    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 20:59:02.723511 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.723306    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 20:59:02.723511 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.723387    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d7mp8\""
Sep 10 20:59:02.724508 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.724491    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:02.724733 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:02.724702    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6hc86" podUID="c12d6b27-9952-4214-8b9d-2864f13966f9"
Sep 10 20:59:02.727079 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.727058    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.729120 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.729098    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 20:59:02.729220 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.729157    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pzvmp\""
Sep 10 20:59:02.729662 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.729547    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:02.729662 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:02.729626    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jskzm" podUID="b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e"
Sep 10 20:59:02.731999 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.731980    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:02.732089 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:02.732046    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8v5s" podUID="eb76c9c7-abc6-4d93-b8e8-f68110f905a9"
Sep 10 20:59:02.748749 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.748726    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gwbqd"
Sep 10 20:59:02.757805 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.757782    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gwbqd"
Sep 10 20:59:02.804439 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.804411    2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 20:59:02.815216 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815185    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-lib-modules\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.815330 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815217    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22f578b7-ad26-41b3-b9df-8b61f1770171-tmp\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.815330 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815245    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-host-slash\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.815330 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815272    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-run-openvswitch\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.815330 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815319    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-dbus\") pod \"global-pull-secret-syncer-k8v5s\" (UID: \"eb76c9c7-abc6-4d93-b8e8-f68110f905a9\") " pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:02.815572 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815331    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-host-slash\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.815572 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815344    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/22f578b7-ad26-41b3-b9df-8b61f1770171-etc-tuned\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.815572 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815369    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5c065834-094d-4fef-8f9c-ad8e42c34084-registration-dir\") pod \"aws-ebs-csi-driver-node-vct76\" (UID: \"5c065834-094d-4fef-8f9c-ad8e42c34084\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76"
Sep 10 20:59:02.815572 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815391    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-run-openvswitch\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.815572 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815377    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-lib-modules\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.815572 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815446    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/841ad25d-3c20-4383-a1c2-31b401b6a6d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c6cwt\" (UID: \"841ad25d-3c20-4383-a1c2-31b401b6a6d1\") " pod="openshift-multus/multus-additional-cni-plugins-c6cwt"
Sep 10 20:59:02.815572 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815495    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6aa58f1a-77a8-4479-a7a8-a68b6a4673cc-hosts-file\") pod \"node-resolver-wprl7\" (UID: \"6aa58f1a-77a8-4479-a7a8-a68b6a4673cc\") " pod="openshift-dns/node-resolver-wprl7"
Sep 10 20:59:02.815572 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815526    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-multus-cni-dir\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.815572 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815556    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-host-run-multus-certs\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.816020 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815596    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-etc-modprobe-d\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.816020 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815625    2584 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 20:59:02.816020 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815655    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-etc-openvswitch\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.816020 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815683    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-run-ovn\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.816020 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815740    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-etc-modprobe-d\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.816020 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815744    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c1477d2e-55a6-415f-85d0-b325df3980f1-ovnkube-config\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.816020 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815761    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-etc-openvswitch\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.816020 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815792    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-cnibin\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.816020 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815830    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-run-ovn\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.816020 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815839    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-multus-conf-dir\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.816020 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815868    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-etc-kubernetes\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.816020 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815914    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk5ml\" (UniqueName: \"kubernetes.io/projected/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-kube-api-access-zk5ml\") pod \"network-metrics-daemon-jskzm\" (UID: \"b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e\") " pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:02.816020 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.815972    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-var-lib-openvswitch\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.816020 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816024    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/841ad25d-3c20-4383-a1c2-31b401b6a6d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-c6cwt\" (UID: \"841ad25d-3c20-4383-a1c2-31b401b6a6d1\") " pod="openshift-multus/multus-additional-cni-plugins-c6cwt"
Sep 10 20:59:02.816815 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816054    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/841ad25d-3c20-4383-a1c2-31b401b6a6d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c6cwt\" (UID: \"841ad25d-3c20-4383-a1c2-31b401b6a6d1\") " pod="openshift-multus/multus-additional-cni-plugins-c6cwt"
Sep 10 20:59:02.816815 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816093    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmzxv\" (UniqueName: \"kubernetes.io/projected/6aa58f1a-77a8-4479-a7a8-a68b6a4673cc-kube-api-access-zmzxv\") pod \"node-resolver-wprl7\" (UID: \"6aa58f1a-77a8-4479-a7a8-a68b6a4673cc\") " pod="openshift-dns/node-resolver-wprl7"
Sep 10 20:59:02.816815 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816107    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-var-lib-openvswitch\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.816815 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816121    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-etc-kubernetes\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.816815 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816159    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-var-lib-kubelet\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.816815 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816186    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/841ad25d-3c20-4383-a1c2-31b401b6a6d1-cnibin\") pod \"multus-additional-cni-plugins-c6cwt\" (UID: \"841ad25d-3c20-4383-a1c2-31b401b6a6d1\") " pod="openshift-multus/multus-additional-cni-plugins-c6cwt"
Sep 10 20:59:02.816815 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816218    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-host-var-lib-cni-multus\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.816815 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816245    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cdafa9fe-0815-45f4-b40b-d41b196f2735-konnectivity-ca\") pod \"konnectivity-agent-r282d\" (UID: \"cdafa9fe-0815-45f4-b40b-d41b196f2735\") " pod="kube-system/konnectivity-agent-r282d"
Sep 10 20:59:02.816815 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816257    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-var-lib-kubelet\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.816815 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816305    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-etc-sysctl-conf\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.816815 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816331    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-run-systemd\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.816815 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816356    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-host-cni-bin\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.816815 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816380    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-etc-systemd\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.816815 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816403    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-node-log\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.816815 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816423    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-etc-sysctl-conf\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.816815 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816442    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/445e9438-de21-4264-86b9-3fad26c5ebb3-cni-binary-copy\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.816815 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816481    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-host-cni-bin\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.817653 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816499    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dbed4e47-f58b-451c-8205-283e0392aec5-iptables-alerter-script\") pod \"iptables-alerter-6tbjl\" (UID: \"dbed4e47-f58b-451c-8205-283e0392aec5\") " pod="openshift-network-operator/iptables-alerter-6tbjl"
Sep 10 20:59:02.817653 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816526    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-run-systemd\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.817653 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816526    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbt8t\" (UniqueName: \"kubernetes.io/projected/5c065834-094d-4fef-8f9c-ad8e42c34084-kube-api-access-vbt8t\") pod \"aws-ebs-csi-driver-node-vct76\" (UID: \"5c065834-094d-4fef-8f9c-ad8e42c34084\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76"
Sep 10 20:59:02.817653 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816540    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c1477d2e-55a6-415f-85d0-b325df3980f1-ovnkube-config\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.817653 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816614    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-system-cni-dir\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.817653 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816653    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-etc-sysconfig\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.817653 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816674    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-node-log\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.817653 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816704    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5c065834-094d-4fef-8f9c-ad8e42c34084-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vct76\" (UID: \"5c065834-094d-4fef-8f9c-ad8e42c34084\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76"
Sep 10 20:59:02.817653 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816737    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5c065834-094d-4fef-8f9c-ad8e42c34084-socket-dir\") pod \"aws-ebs-csi-driver-node-vct76\" (UID: \"5c065834-094d-4fef-8f9c-ad8e42c34084\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76"
Sep 10 20:59:02.817653 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816243    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-etc-kubernetes\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.817653 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816765    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-etc-sysconfig\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.817653 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816791    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl4sh\" (UniqueName: \"kubernetes.io/projected/841ad25d-3c20-4383-a1c2-31b401b6a6d1-kube-api-access-xl4sh\") pod \"multus-additional-cni-plugins-c6cwt\" (UID: \"841ad25d-3c20-4383-a1c2-31b401b6a6d1\") " pod="openshift-multus/multus-additional-cni-plugins-c6cwt"
Sep 10 20:59:02.817653 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816833    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-hostroot\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.817653 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816859    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-etc-systemd\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.817653 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816881    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-host-kubelet\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.817653 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816914    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c1477d2e-55a6-415f-85d0-b325df3980f1-ovnkube-script-lib\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.817653 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816943    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-host-kubelet\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.818480 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.816988    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hdc4\" (UniqueName: \"kubernetes.io/projected/dbed4e47-f58b-451c-8205-283e0392aec5-kube-api-access-6hdc4\") pod \"iptables-alerter-6tbjl\" (UID: \"dbed4e47-f58b-451c-8205-283e0392aec5\") " pod="openshift-network-operator/iptables-alerter-6tbjl"
Sep 10 20:59:02.818480 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.817015    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-host-run-ovn-kubernetes\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.818480 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.817075    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-host-cni-netd\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.818480 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.817108    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1477d2e-55a6-415f-85d0-b325df3980f1-ovn-node-metrics-cert\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.818480 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.817160    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dbed4e47-f58b-451c-8205-283e0392aec5-host-slash\") pod \"iptables-alerter-6tbjl\" (UID: \"dbed4e47-f58b-451c-8205-283e0392aec5\") " pod="openshift-network-operator/iptables-alerter-6tbjl"
Sep 10 20:59:02.818480 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.817228    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5c065834-094d-4fef-8f9c-ad8e42c34084-device-dir\") pod \"aws-ebs-csi-driver-node-vct76\" (UID: \"5c065834-094d-4fef-8f9c-ad8e42c34084\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76"
Sep 10 20:59:02.818480 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.817261    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvdnw\" (UniqueName: \"kubernetes.io/projected/445e9438-de21-4264-86b9-3fad26c5ebb3-kube-api-access-lvdnw\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.818480 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.817294    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5fn8\" (UniqueName: \"kubernetes.io/projected/22f578b7-ad26-41b3-b9df-8b61f1770171-kube-api-access-m5fn8\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.818480 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.817298    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-host-run-ovn-kubernetes\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.818480 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.817262    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dbed4e47-f58b-451c-8205-283e0392aec5-host-slash\") pod \"iptables-alerter-6tbjl\" (UID: \"dbed4e47-f58b-451c-8205-283e0392aec5\") " pod="openshift-network-operator/iptables-alerter-6tbjl"
Sep 10 20:59:02.818480 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.817369    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-systemd-units\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.818480 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.817424    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c1477d2e-55a6-415f-85d0-b325df3980f1-env-overrides\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.818480 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.817490    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8032963a-4ac1-4d7b-a4de-e9bac1e2c9a1-serviceca\") pod \"node-ca-xswnp\" (UID: \"8032963a-4ac1-4d7b-a4de-e9bac1e2c9a1\") " pod="openshift-image-registry/node-ca-xswnp"
Sep 10 20:59:02.818480 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.817541    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/841ad25d-3c20-4383-a1c2-31b401b6a6d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c6cwt\" (UID: \"841ad25d-3c20-4383-a1c2-31b401b6a6d1\") " pod="openshift-multus/multus-additional-cni-plugins-c6cwt"
Sep 10 20:59:02.818480 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.817572    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.818480 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.817633    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7zzq\" (UniqueName: \"kubernetes.io/projected/c1477d2e-55a6-415f-85d0-b325df3980f1-kube-api-access-p7zzq\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.818480 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.817654    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-host-cni-netd\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.819367 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.817700    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-original-pull-secret\") pod \"global-pull-secret-syncer-k8v5s\" (UID: \"eb76c9c7-abc6-4d93-b8e8-f68110f905a9\") " pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:02.819367 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.817757    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-systemd-units\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.819367 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.817750    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.819367 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.817800    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dbed4e47-f58b-451c-8205-283e0392aec5-iptables-alerter-script\") pod \"iptables-alerter-6tbjl\" (UID: \"dbed4e47-f58b-451c-8205-283e0392aec5\") " pod="openshift-network-operator/iptables-alerter-6tbjl"
Sep 10 20:59:02.819367 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.817878    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/841ad25d-3c20-4383-a1c2-31b401b6a6d1-system-cni-dir\") pod \"multus-additional-cni-plugins-c6cwt\" (UID: \"841ad25d-3c20-4383-a1c2-31b401b6a6d1\") " pod="openshift-multus/multus-additional-cni-plugins-c6cwt"
Sep 10 20:59:02.819367 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.817897    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c1477d2e-55a6-415f-85d0-b325df3980f1-ovnkube-script-lib\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.819367 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.817921    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6aa58f1a-77a8-4479-a7a8-a68b6a4673cc-tmp-dir\") pod \"node-resolver-wprl7\" (UID: \"6aa58f1a-77a8-4479-a7a8-a68b6a4673cc\") " pod="openshift-dns/node-resolver-wprl7"
Sep 10 20:59:02.819367 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.817968    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cdafa9fe-0815-45f4-b40b-d41b196f2735-agent-certs\") pod \"konnectivity-agent-r282d\" (UID: \"cdafa9fe-0815-45f4-b40b-d41b196f2735\") " pod="kube-system/konnectivity-agent-r282d"
Sep 10 20:59:02.819367 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818021    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-etc-sysctl-d\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.819367 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818090    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-run\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.819367 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818198    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-etc-sysctl-d\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.819367 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818201    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c1477d2e-55a6-415f-85d0-b325df3980f1-env-overrides\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.819367 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818233    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5c065834-094d-4fef-8f9c-ad8e42c34084-sys-fs\") pod \"aws-ebs-csi-driver-node-vct76\" (UID: \"5c065834-094d-4fef-8f9c-ad8e42c34084\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76"
Sep 10 20:59:02.819367 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818266    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8032963a-4ac1-4d7b-a4de-e9bac1e2c9a1-host\") pod \"node-ca-xswnp\" (UID: \"8032963a-4ac1-4d7b-a4de-e9bac1e2c9a1\") " pod="openshift-image-registry/node-ca-xswnp"
Sep 10 20:59:02.819367 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818293    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-multus-socket-dir-parent\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.819367 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818323    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-host-var-lib-cni-bin\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.819367 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818326    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-run\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.820255 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818366    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-host\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.820255 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818404    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-host-run-netns\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.820255 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818428    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-host\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.820255 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818448    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-log-socket\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.820255 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818476    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5c065834-094d-4fef-8f9c-ad8e42c34084-etc-selinux\") pod \"aws-ebs-csi-driver-node-vct76\" (UID: \"5c065834-094d-4fef-8f9c-ad8e42c34084\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76"
Sep 10 20:59:02.820255 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818504    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-host-run-netns\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.820255 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818541    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvtcx\" (UniqueName: \"kubernetes.io/projected/8032963a-4ac1-4d7b-a4de-e9bac1e2c9a1-kube-api-access-hvtcx\") pod \"node-ca-xswnp\" (UID: \"8032963a-4ac1-4d7b-a4de-e9bac1e2c9a1\") " pod="openshift-image-registry/node-ca-xswnp"
Sep 10 20:59:02.820255 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818599    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c1477d2e-55a6-415f-85d0-b325df3980f1-log-socket\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.820255 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818645    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-kubelet-config\") pod \"global-pull-secret-syncer-k8v5s\" (UID: \"eb76c9c7-abc6-4d93-b8e8-f68110f905a9\") " pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:02.820255 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818691    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-host-run-k8s-cni-cncf-io\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.820255 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818728    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/445e9438-de21-4264-86b9-3fad26c5ebb3-multus-daemon-config\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.820255 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818764    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-sys\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.820255 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818789    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfzk5\" (UniqueName: \"kubernetes.io/projected/c12d6b27-9952-4214-8b9d-2864f13966f9-kube-api-access-lfzk5\") pod \"network-check-target-6hc86\" (UID: \"c12d6b27-9952-4214-8b9d-2864f13966f9\") " pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:02.820255 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818818    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-host-run-netns\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.820255 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818839    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-host-var-lib-kubelet\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.820255 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818843    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22f578b7-ad26-41b3-b9df-8b61f1770171-sys\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.820255 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.818912    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-metrics-certs\") pod \"network-metrics-daemon-jskzm\" (UID: \"b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e\") " pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:02.820976 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.820517    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/22f578b7-ad26-41b3-b9df-8b61f1770171-etc-tuned\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.821855 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.821784    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22f578b7-ad26-41b3-b9df-8b61f1770171-tmp\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.822363 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.822085    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1477d2e-55a6-415f-85d0-b325df3980f1-ovn-node-metrics-cert\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.840454 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.839911    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hdc4\" (UniqueName: \"kubernetes.io/projected/dbed4e47-f58b-451c-8205-283e0392aec5-kube-api-access-6hdc4\") pod \"iptables-alerter-6tbjl\" (UID: \"dbed4e47-f58b-451c-8205-283e0392aec5\") " pod="openshift-network-operator/iptables-alerter-6tbjl"
Sep 10 20:59:02.841036 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.840977    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7zzq\" (UniqueName: \"kubernetes.io/projected/c1477d2e-55a6-415f-85d0-b325df3980f1-kube-api-access-p7zzq\") pod \"ovnkube-node-qbwqs\" (UID: \"c1477d2e-55a6-415f-85d0-b325df3980f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:02.841258 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.841227    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5fn8\" (UniqueName: \"kubernetes.io/projected/22f578b7-ad26-41b3-b9df-8b61f1770171-kube-api-access-m5fn8\") pod \"tuned-vmvb8\" (UID: \"22f578b7-ad26-41b3-b9df-8b61f1770171\") " pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:02.842870 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.842481    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-205.ec2.internal" event={"ID":"77a4c4cd902dc458728fadabb7f791eb","Type":"ContainerStarted","Data":"b50097d2c34901a936f0f3c3dca42c4cfb737abb20a8715e89e59f2fbbf9d314"}
Sep 10 20:59:02.844761 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.844408    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal" event={"ID":"f0f127f626c5949a577288cc12509aa1","Type":"ContainerStarted","Data":"272bd805f9299902de6b013ca0edfb985b41c21aaf573a4f368972bdbf79bd4d"}
Sep 10 20:59:02.920122 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.919520    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-dbus\") pod \"global-pull-secret-syncer-k8v5s\" (UID: \"eb76c9c7-abc6-4d93-b8e8-f68110f905a9\") " pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:02.920122 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.919555    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5c065834-094d-4fef-8f9c-ad8e42c34084-registration-dir\") pod \"aws-ebs-csi-driver-node-vct76\" (UID: \"5c065834-094d-4fef-8f9c-ad8e42c34084\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76"
Sep 10 20:59:02.920122 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.919587    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/841ad25d-3c20-4383-a1c2-31b401b6a6d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c6cwt\" (UID: \"841ad25d-3c20-4383-a1c2-31b401b6a6d1\") " pod="openshift-multus/multus-additional-cni-plugins-c6cwt"
Sep 10 20:59:02.920122 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.919612    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6aa58f1a-77a8-4479-a7a8-a68b6a4673cc-hosts-file\") pod \"node-resolver-wprl7\" (UID: \"6aa58f1a-77a8-4479-a7a8-a68b6a4673cc\") " pod="openshift-dns/node-resolver-wprl7"
Sep 10 20:59:02.920122 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.919634    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-multus-cni-dir\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.920122 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.919655    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-host-run-multus-certs\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.920122 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.919681    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-cnibin\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.920122 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.919707    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-multus-conf-dir\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.920122 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.919721    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-dbus\") pod \"global-pull-secret-syncer-k8v5s\" (UID: \"eb76c9c7-abc6-4d93-b8e8-f68110f905a9\") " pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:02.920122 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.919743    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-etc-kubernetes\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.920122 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.919772    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk5ml\" (UniqueName: \"kubernetes.io/projected/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-kube-api-access-zk5ml\") pod \"network-metrics-daemon-jskzm\" (UID: \"b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e\") " pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:02.920122 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.919800    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5c065834-094d-4fef-8f9c-ad8e42c34084-registration-dir\") pod \"aws-ebs-csi-driver-node-vct76\" (UID: \"5c065834-094d-4fef-8f9c-ad8e42c34084\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76"
Sep 10 20:59:02.920122 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.919802    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/841ad25d-3c20-4383-a1c2-31b401b6a6d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-c6cwt\" (UID: \"841ad25d-3c20-4383-a1c2-31b401b6a6d1\") " pod="openshift-multus/multus-additional-cni-plugins-c6cwt"
Sep 10 20:59:02.920122 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.919844    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/841ad25d-3c20-4383-a1c2-31b401b6a6d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c6cwt\" (UID: \"841ad25d-3c20-4383-a1c2-31b401b6a6d1\") " pod="openshift-multus/multus-additional-cni-plugins-c6cwt"
Sep 10 20:59:02.920122 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.919873    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmzxv\" (UniqueName: \"kubernetes.io/projected/6aa58f1a-77a8-4479-a7a8-a68b6a4673cc-kube-api-access-zmzxv\") pod \"node-resolver-wprl7\" (UID: \"6aa58f1a-77a8-4479-a7a8-a68b6a4673cc\") " pod="openshift-dns/node-resolver-wprl7"
Sep 10 20:59:02.920122 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.919904    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/841ad25d-3c20-4383-a1c2-31b401b6a6d1-cnibin\") pod \"multus-additional-cni-plugins-c6cwt\" (UID: \"841ad25d-3c20-4383-a1c2-31b401b6a6d1\") " pod="openshift-multus/multus-additional-cni-plugins-c6cwt"
Sep 10 20:59:02.920122 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.919928    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-host-var-lib-cni-multus\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.921176 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.919954    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cdafa9fe-0815-45f4-b40b-d41b196f2735-konnectivity-ca\") pod \"konnectivity-agent-r282d\" (UID: \"cdafa9fe-0815-45f4-b40b-d41b196f2735\") " pod="kube-system/konnectivity-agent-r282d"
Sep 10 20:59:02.921176 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.919983    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/445e9438-de21-4264-86b9-3fad26c5ebb3-cni-binary-copy\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.921176 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920025    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbt8t\" (UniqueName: \"kubernetes.io/projected/5c065834-094d-4fef-8f9c-ad8e42c34084-kube-api-access-vbt8t\") pod \"aws-ebs-csi-driver-node-vct76\" (UID: \"5c065834-094d-4fef-8f9c-ad8e42c34084\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76"
Sep 10 20:59:02.921176 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920053    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-system-cni-dir\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.921176 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920080    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5c065834-094d-4fef-8f9c-ad8e42c34084-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vct76\" (UID: \"5c065834-094d-4fef-8f9c-ad8e42c34084\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76"
Sep 10 20:59:02.921176 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920107    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5c065834-094d-4fef-8f9c-ad8e42c34084-socket-dir\") pod \"aws-ebs-csi-driver-node-vct76\" (UID: \"5c065834-094d-4fef-8f9c-ad8e42c34084\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76"
Sep 10 20:59:02.921176 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920138    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl4sh\" (UniqueName: \"kubernetes.io/projected/841ad25d-3c20-4383-a1c2-31b401b6a6d1-kube-api-access-xl4sh\") pod \"multus-additional-cni-plugins-c6cwt\" (UID: \"841ad25d-3c20-4383-a1c2-31b401b6a6d1\") " pod="openshift-multus/multus-additional-cni-plugins-c6cwt"
Sep 10 20:59:02.921176 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920193    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-host-run-multus-certs\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.921176 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920192    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-host-var-lib-cni-multus\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.921176 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920250    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-hostroot\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.921176 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920286    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5c065834-094d-4fef-8f9c-ad8e42c34084-device-dir\") pod \"aws-ebs-csi-driver-node-vct76\" (UID: \"5c065834-094d-4fef-8f9c-ad8e42c34084\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76"
Sep 10 20:59:02.921176 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920315    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvdnw\" (UniqueName: \"kubernetes.io/projected/445e9438-de21-4264-86b9-3fad26c5ebb3-kube-api-access-lvdnw\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.921176 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920370    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/841ad25d-3c20-4383-a1c2-31b401b6a6d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c6cwt\" (UID: \"841ad25d-3c20-4383-a1c2-31b401b6a6d1\") " pod="openshift-multus/multus-additional-cni-plugins-c6cwt"
Sep 10 20:59:02.921176 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920413    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8032963a-4ac1-4d7b-a4de-e9bac1e2c9a1-serviceca\") pod \"node-ca-xswnp\" (UID: \"8032963a-4ac1-4d7b-a4de-e9bac1e2c9a1\") " pod="openshift-image-registry/node-ca-xswnp"
Sep 10 20:59:02.921176 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920432    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6aa58f1a-77a8-4479-a7a8-a68b6a4673cc-hosts-file\") pod \"node-resolver-wprl7\" (UID: \"6aa58f1a-77a8-4479-a7a8-a68b6a4673cc\") " pod="openshift-dns/node-resolver-wprl7"
Sep 10 20:59:02.921176 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920442    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/841ad25d-3c20-4383-a1c2-31b401b6a6d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c6cwt\" (UID: \"841ad25d-3c20-4383-a1c2-31b401b6a6d1\") " pod="openshift-multus/multus-additional-cni-plugins-c6cwt"
Sep 10 20:59:02.921176 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920437    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/841ad25d-3c20-4383-a1c2-31b401b6a6d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-c6cwt\" (UID: \"841ad25d-3c20-4383-a1c2-31b401b6a6d1\") " pod="openshift-multus/multus-additional-cni-plugins-c6cwt"
Sep 10 20:59:02.922016 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920474    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-multus-conf-dir\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.922016 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920481    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-etc-kubernetes\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.922016 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920506    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-original-pull-secret\") pod \"global-pull-secret-syncer-k8v5s\" (UID: \"eb76c9c7-abc6-4d93-b8e8-f68110f905a9\") " pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:02.922016 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920536    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/841ad25d-3c20-4383-a1c2-31b401b6a6d1-system-cni-dir\") pod \"multus-additional-cni-plugins-c6cwt\" (UID: \"841ad25d-3c20-4383-a1c2-31b401b6a6d1\") " pod="openshift-multus/multus-additional-cni-plugins-c6cwt"
Sep 10 20:59:02.922016 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920550    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-multus-cni-dir\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.922016 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920563    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6aa58f1a-77a8-4479-a7a8-a68b6a4673cc-tmp-dir\") pod \"node-resolver-wprl7\" (UID: \"6aa58f1a-77a8-4479-a7a8-a68b6a4673cc\") " pod="openshift-dns/node-resolver-wprl7"
Sep 10 20:59:02.922016 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920437    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5c065834-094d-4fef-8f9c-ad8e42c34084-device-dir\") pod \"aws-ebs-csi-driver-node-vct76\" (UID: \"5c065834-094d-4fef-8f9c-ad8e42c34084\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76"
Sep 10 20:59:02.922016 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920638    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-cnibin\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.922016 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:02.920720    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:59:02.922016 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:02.920809    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-original-pull-secret podName:eb76c9c7-abc6-4d93-b8e8-f68110f905a9 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:03.42077439 +0000 UTC m=+3.085799175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-original-pull-secret") pod "global-pull-secret-syncer-k8v5s" (UID: "eb76c9c7-abc6-4d93-b8e8-f68110f905a9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:59:02.922016 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920851    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cdafa9fe-0815-45f4-b40b-d41b196f2735-agent-certs\") pod \"konnectivity-agent-r282d\" (UID: \"cdafa9fe-0815-45f4-b40b-d41b196f2735\") " pod="kube-system/konnectivity-agent-r282d"
Sep 10 20:59:02.922016 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920888    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5c065834-094d-4fef-8f9c-ad8e42c34084-sys-fs\") pod \"aws-ebs-csi-driver-node-vct76\" (UID: \"5c065834-094d-4fef-8f9c-ad8e42c34084\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76"
Sep 10 20:59:02.922016 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920923    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8032963a-4ac1-4d7b-a4de-e9bac1e2c9a1-host\") pod \"node-ca-xswnp\" (UID: \"8032963a-4ac1-4d7b-a4de-e9bac1e2c9a1\") " pod="openshift-image-registry/node-ca-xswnp"
Sep 10 20:59:02.922016 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920952    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-multus-socket-dir-parent\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.922016 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.920983    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-host-var-lib-cni-bin\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.922016 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.921064    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cdafa9fe-0815-45f4-b40b-d41b196f2735-konnectivity-ca\") pod \"konnectivity-agent-r282d\" (UID: \"cdafa9fe-0815-45f4-b40b-d41b196f2735\") " pod="kube-system/konnectivity-agent-r282d"
Sep 10 20:59:02.922016 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.921078    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-system-cni-dir\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.922944 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.921067    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-multus-socket-dir-parent\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.922944 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.921127    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/841ad25d-3c20-4383-a1c2-31b401b6a6d1-cnibin\") pod \"multus-additional-cni-plugins-c6cwt\" (UID: \"841ad25d-3c20-4383-a1c2-31b401b6a6d1\") " pod="openshift-multus/multus-additional-cni-plugins-c6cwt"
Sep 10 20:59:02.922944 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.921164    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5c065834-094d-4fef-8f9c-ad8e42c34084-etc-selinux\") pod \"aws-ebs-csi-driver-node-vct76\" (UID: \"5c065834-094d-4fef-8f9c-ad8e42c34084\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76"
Sep 10 20:59:02.922944 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.921167    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5c065834-094d-4fef-8f9c-ad8e42c34084-socket-dir\") pod \"aws-ebs-csi-driver-node-vct76\" (UID: \"5c065834-094d-4fef-8f9c-ad8e42c34084\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76"
Sep 10 20:59:02.922944 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.921192    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvtcx\" (UniqueName: \"kubernetes.io/projected/8032963a-4ac1-4d7b-a4de-e9bac1e2c9a1-kube-api-access-hvtcx\") pod \"node-ca-xswnp\" (UID: \"8032963a-4ac1-4d7b-a4de-e9bac1e2c9a1\") " pod="openshift-image-registry/node-ca-xswnp"
Sep 10 20:59:02.922944 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.921206    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/841ad25d-3c20-4383-a1c2-31b401b6a6d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c6cwt\" (UID: \"841ad25d-3c20-4383-a1c2-31b401b6a6d1\") " pod="openshift-multus/multus-additional-cni-plugins-c6cwt"
Sep 10 20:59:02.922944 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.921221    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-kubelet-config\") pod \"global-pull-secret-syncer-k8v5s\" (UID: \"eb76c9c7-abc6-4d93-b8e8-f68110f905a9\") " pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:02.922944 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.921249    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-host-run-k8s-cni-cncf-io\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.922944 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.921271    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-hostroot\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.922944 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.921292    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/445e9438-de21-4264-86b9-3fad26c5ebb3-multus-daemon-config\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.922944 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.921647    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6aa58f1a-77a8-4479-a7a8-a68b6a4673cc-tmp-dir\") pod \"node-resolver-wprl7\" (UID: \"6aa58f1a-77a8-4479-a7a8-a68b6a4673cc\") " pod="openshift-dns/node-resolver-wprl7"
Sep 10 20:59:02.922944 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.921692    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/841ad25d-3c20-4383-a1c2-31b401b6a6d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c6cwt\" (UID: \"841ad25d-3c20-4383-a1c2-31b401b6a6d1\") " pod="openshift-multus/multus-additional-cni-plugins-c6cwt"
Sep 10 20:59:02.922944 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.921712    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-kubelet-config\") pod \"global-pull-secret-syncer-k8v5s\" (UID: \"eb76c9c7-abc6-4d93-b8e8-f68110f905a9\") " pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:02.922944 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.921725    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5c065834-094d-4fef-8f9c-ad8e42c34084-sys-fs\") pod \"aws-ebs-csi-driver-node-vct76\" (UID: \"5c065834-094d-4fef-8f9c-ad8e42c34084\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76"
Sep 10 20:59:02.922944 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.921746    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8032963a-4ac1-4d7b-a4de-e9bac1e2c9a1-serviceca\") pod \"node-ca-xswnp\" (UID: \"8032963a-4ac1-4d7b-a4de-e9bac1e2c9a1\") " pod="openshift-image-registry/node-ca-xswnp"
Sep 10 20:59:02.922944 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.921773    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfzk5\" (UniqueName: \"kubernetes.io/projected/c12d6b27-9952-4214-8b9d-2864f13966f9-kube-api-access-lfzk5\") pod \"network-check-target-6hc86\" (UID: \"c12d6b27-9952-4214-8b9d-2864f13966f9\") " pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:02.922944 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.921840    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5c065834-094d-4fef-8f9c-ad8e42c34084-etc-selinux\") pod \"aws-ebs-csi-driver-node-vct76\" (UID: \"5c065834-094d-4fef-8f9c-ad8e42c34084\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76"
Sep 10 20:59:02.923816 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.921889    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-host-run-k8s-cni-cncf-io\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.923816 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.921945    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/841ad25d-3c20-4383-a1c2-31b401b6a6d1-system-cni-dir\") pod \"multus-additional-cni-plugins-c6cwt\" (UID: \"841ad25d-3c20-4383-a1c2-31b401b6a6d1\") " pod="openshift-multus/multus-additional-cni-plugins-c6cwt"
Sep 10 20:59:02.923816 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.921985    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-host-var-lib-cni-bin\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.923816 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.922023    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-host-run-netns\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.923816 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.922053    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-host-var-lib-kubelet\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.923816 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.922109    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-host-var-lib-kubelet\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.923816 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.922113    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-metrics-certs\") pod \"network-metrics-daemon-jskzm\" (UID: \"b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e\") " pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:02.923816 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:02.922203    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:59:02.923816 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:02.922272    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-metrics-certs podName:b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e nodeName:}" failed. No retries permitted until 2026-09-10 20:59:03.422252597 +0000 UTC m=+3.087277395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-metrics-certs") pod "network-metrics-daemon-jskzm" (UID: "b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:59:02.923816 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.922281    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/445e9438-de21-4264-86b9-3fad26c5ebb3-cni-binary-copy\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.923816 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.922310    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/445e9438-de21-4264-86b9-3fad26c5ebb3-multus-daemon-config\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.923816 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.922332    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8032963a-4ac1-4d7b-a4de-e9bac1e2c9a1-host\") pod \"node-ca-xswnp\" (UID: \"8032963a-4ac1-4d7b-a4de-e9bac1e2c9a1\") " pod="openshift-image-registry/node-ca-xswnp"
Sep 10 20:59:02.923816 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.922341    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/445e9438-de21-4264-86b9-3fad26c5ebb3-host-run-netns\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.923816 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.922369    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5c065834-094d-4fef-8f9c-ad8e42c34084-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vct76\" (UID: \"5c065834-094d-4fef-8f9c-ad8e42c34084\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76"
Sep 10 20:59:02.925168 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.925143    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cdafa9fe-0815-45f4-b40b-d41b196f2735-agent-certs\") pod \"konnectivity-agent-r282d\" (UID: \"cdafa9fe-0815-45f4-b40b-d41b196f2735\") " pod="kube-system/konnectivity-agent-r282d"
Sep 10 20:59:02.936871 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.936780    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl4sh\" (UniqueName: \"kubernetes.io/projected/841ad25d-3c20-4383-a1c2-31b401b6a6d1-kube-api-access-xl4sh\") pod \"multus-additional-cni-plugins-c6cwt\" (UID: \"841ad25d-3c20-4383-a1c2-31b401b6a6d1\") " pod="openshift-multus/multus-additional-cni-plugins-c6cwt"
Sep 10 20:59:02.937325 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.937249    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk5ml\" (UniqueName: \"kubernetes.io/projected/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-kube-api-access-zk5ml\") pod \"network-metrics-daemon-jskzm\" (UID: \"b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e\") " pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:02.939712 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.939665    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvdnw\" (UniqueName: \"kubernetes.io/projected/445e9438-de21-4264-86b9-3fad26c5ebb3-kube-api-access-lvdnw\") pod \"multus-nrl66\" (UID: \"445e9438-de21-4264-86b9-3fad26c5ebb3\") " pod="openshift-multus/multus-nrl66"
Sep 10 20:59:02.939807 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.939738    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmzxv\" (UniqueName: \"kubernetes.io/projected/6aa58f1a-77a8-4479-a7a8-a68b6a4673cc-kube-api-access-zmzxv\") pod \"node-resolver-wprl7\" (UID: \"6aa58f1a-77a8-4479-a7a8-a68b6a4673cc\") " pod="openshift-dns/node-resolver-wprl7"
Sep 10 20:59:02.940694 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.940661    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvtcx\" (UniqueName: \"kubernetes.io/projected/8032963a-4ac1-4d7b-a4de-e9bac1e2c9a1-kube-api-access-hvtcx\") pod \"node-ca-xswnp\" (UID: \"8032963a-4ac1-4d7b-a4de-e9bac1e2c9a1\") " pod="openshift-image-registry/node-ca-xswnp"
Sep 10 20:59:02.940694 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.940686    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbt8t\" (UniqueName: \"kubernetes.io/projected/5c065834-094d-4fef-8f9c-ad8e42c34084-kube-api-access-vbt8t\") pod \"aws-ebs-csi-driver-node-vct76\" (UID: \"5c065834-094d-4fef-8f9c-ad8e42c34084\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76"
Sep 10 20:59:02.943253 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:02.943131    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:59:02.943253 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:02.943165    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:59:02.943253 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:02.943179    2584 projected.go:196] Error preparing data for projected volume kube-api-access-lfzk5 for pod openshift-network-diagnostics/network-check-target-6hc86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:59:02.943253 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:02.943234    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c12d6b27-9952-4214-8b9d-2864f13966f9-kube-api-access-lfzk5 podName:c12d6b27-9952-4214-8b9d-2864f13966f9 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:03.443220399 +0000 UTC m=+3.108245178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lfzk5" (UniqueName: "kubernetes.io/projected/c12d6b27-9952-4214-8b9d-2864f13966f9-kube-api-access-lfzk5") pod "network-check-target-6hc86" (UID: "c12d6b27-9952-4214-8b9d-2864f13966f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:59:02.955818 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:02.955797    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:59:03.012008 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.011861    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:59:03.013040 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.012970    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vmvb8"
Sep 10 20:59:03.022952 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.022927    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:03.025038 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:03.024963    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f578b7_ad26_41b3_b9df_8b61f1770171.slice/crio-85ed4a1db6d73a9388405800af6fc469692deeae89e2232329fd1102ea0dac91 WatchSource:0}: Error finding container 85ed4a1db6d73a9388405800af6fc469692deeae89e2232329fd1102ea0dac91: Status 404 returned error can't find the container with id 85ed4a1db6d73a9388405800af6fc469692deeae89e2232329fd1102ea0dac91
Sep 10 20:59:03.033129 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:03.033106    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1477d2e_55a6_415f_85d0_b325df3980f1.slice/crio-e554729f003bdde542323486a5d5d90ed9ad5cf2c235f308be1999c5ae2e9c18 WatchSource:0}: Error finding container e554729f003bdde542323486a5d5d90ed9ad5cf2c235f308be1999c5ae2e9c18: Status 404 returned error can't find the container with id e554729f003bdde542323486a5d5d90ed9ad5cf2c235f308be1999c5ae2e9c18
Sep 10 20:59:03.034025 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.034005    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6tbjl"
Sep 10 20:59:03.038917 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.038897    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r282d"
Sep 10 20:59:03.045013 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:03.044988    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbed4e47_f58b_451c_8205_283e0392aec5.slice/crio-f46b5759fd7dc2bd88c4c7261523af506d24d2ac33a30082144ff0ab15573353 WatchSource:0}: Error finding container f46b5759fd7dc2bd88c4c7261523af506d24d2ac33a30082144ff0ab15573353: Status 404 returned error can't find the container with id f46b5759fd7dc2bd88c4c7261523af506d24d2ac33a30082144ff0ab15573353
Sep 10 20:59:03.046317 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.046291    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c6cwt"
Sep 10 20:59:03.050629 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:03.050606    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdafa9fe_0815_45f4_b40b_d41b196f2735.slice/crio-82b3ef7d21abc35f79be126270c6e67f584d6bd489c864997c28915ed2fddae1 WatchSource:0}: Error finding container 82b3ef7d21abc35f79be126270c6e67f584d6bd489c864997c28915ed2fddae1: Status 404 returned error can't find the container with id 82b3ef7d21abc35f79be126270c6e67f584d6bd489c864997c28915ed2fddae1
Sep 10 20:59:03.054772 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.054746    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76"
Sep 10 20:59:03.057007 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:03.056973    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841ad25d_3c20_4383_a1c2_31b401b6a6d1.slice/crio-fae68b3800a08b834631556fd0fa9757fb525d8cf33541def5d39fe27af24569 WatchSource:0}: Error finding container fae68b3800a08b834631556fd0fa9757fb525d8cf33541def5d39fe27af24569: Status 404 returned error can't find the container with id fae68b3800a08b834631556fd0fa9757fb525d8cf33541def5d39fe27af24569
Sep 10 20:59:03.060563 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.060540    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xswnp"
Sep 10 20:59:03.066177 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:03.066144    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c065834_094d_4fef_8f9c_ad8e42c34084.slice/crio-b1281410e7e5dd597832bb76e8438d14bee564d918831c26b687a4a49387ce8d WatchSource:0}: Error finding container b1281410e7e5dd597832bb76e8438d14bee564d918831c26b687a4a49387ce8d: Status 404 returned error can't find the container with id b1281410e7e5dd597832bb76e8438d14bee564d918831c26b687a4a49387ce8d
Sep 10 20:59:03.067560 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.067540    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wprl7"
Sep 10 20:59:03.070851 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:03.070811    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8032963a_4ac1_4d7b_a4de_e9bac1e2c9a1.slice/crio-de066ba0f36a103c771f6bcc65a2df38d3a2e75e075a70095100e690e54e799e WatchSource:0}: Error finding container de066ba0f36a103c771f6bcc65a2df38d3a2e75e075a70095100e690e54e799e: Status 404 returned error can't find the container with id de066ba0f36a103c771f6bcc65a2df38d3a2e75e075a70095100e690e54e799e
Sep 10 20:59:03.074088 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.074044    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nrl66"
Sep 10 20:59:03.077432 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:03.077390    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa58f1a_77a8_4479_a7a8_a68b6a4673cc.slice/crio-12b116f823a675039395299de22b03601af5864ffc3ebe349c762b976f710589 WatchSource:0}: Error finding container 12b116f823a675039395299de22b03601af5864ffc3ebe349c762b976f710589: Status 404 returned error can't find the container with id 12b116f823a675039395299de22b03601af5864ffc3ebe349c762b976f710589
Sep 10 20:59:03.087145 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:03.087125    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod445e9438_de21_4264_86b9_3fad26c5ebb3.slice/crio-25eae5448d16b438069ef9a6b8b87f2546730f444e24cf9ea9301fcb9130e8fe WatchSource:0}: Error finding container 25eae5448d16b438069ef9a6b8b87f2546730f444e24cf9ea9301fcb9130e8fe: Status 404 returned error can't find the container with id 25eae5448d16b438069ef9a6b8b87f2546730f444e24cf9ea9301fcb9130e8fe
Sep 10 20:59:03.426572 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.426479    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-metrics-certs\") pod \"network-metrics-daemon-jskzm\" (UID: \"b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e\") " pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:03.426735 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.426585    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-original-pull-secret\") pod \"global-pull-secret-syncer-k8v5s\" (UID: \"eb76c9c7-abc6-4d93-b8e8-f68110f905a9\") " pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:03.426735 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:03.426641    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:59:03.426735 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:03.426689    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:59:03.426735 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:03.426720    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-metrics-certs podName:b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e nodeName:}" failed. No retries permitted until 2026-09-10 20:59:04.426700905 +0000 UTC m=+4.091725705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-metrics-certs") pod "network-metrics-daemon-jskzm" (UID: "b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:59:03.426925 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:03.426743    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-original-pull-secret podName:eb76c9c7-abc6-4d93-b8e8-f68110f905a9 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:04.426732159 +0000 UTC m=+4.091756939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-original-pull-secret") pod "global-pull-secret-syncer-k8v5s" (UID: "eb76c9c7-abc6-4d93-b8e8-f68110f905a9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:59:03.527610 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.527563    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfzk5\" (UniqueName: \"kubernetes.io/projected/c12d6b27-9952-4214-8b9d-2864f13966f9-kube-api-access-lfzk5\") pod \"network-check-target-6hc86\" (UID: \"c12d6b27-9952-4214-8b9d-2864f13966f9\") " pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:03.527796 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:03.527774    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:59:03.527876 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:03.527808    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:59:03.527876 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:03.527822    2584 projected.go:196] Error preparing data for projected volume kube-api-access-lfzk5 for pod openshift-network-diagnostics/network-check-target-6hc86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:59:03.527949 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:03.527891    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c12d6b27-9952-4214-8b9d-2864f13966f9-kube-api-access-lfzk5 podName:c12d6b27-9952-4214-8b9d-2864f13966f9 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:04.527872981 +0000 UTC m=+4.192897763 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfzk5" (UniqueName: "kubernetes.io/projected/c12d6b27-9952-4214-8b9d-2864f13966f9-kube-api-access-lfzk5") pod "network-check-target-6hc86" (UID: "c12d6b27-9952-4214-8b9d-2864f13966f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:59:03.761549 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.761450    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 20:54:03 +0000 UTC" deadline="2028-06-03 09:41:44.456765448 +0000 UTC"
Sep 10 20:59:03.761549 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.761498    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15156h42m40.695275358s"
Sep 10 20:59:03.834463 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.834409    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:03.834648 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:03.834567    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6hc86" podUID="c12d6b27-9952-4214-8b9d-2864f13966f9"
Sep 10 20:59:03.848901 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.848770    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wprl7" event={"ID":"6aa58f1a-77a8-4479-a7a8-a68b6a4673cc","Type":"ContainerStarted","Data":"12b116f823a675039395299de22b03601af5864ffc3ebe349c762b976f710589"}
Sep 10 20:59:03.852261 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.852229    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76" event={"ID":"5c065834-094d-4fef-8f9c-ad8e42c34084","Type":"ContainerStarted","Data":"b1281410e7e5dd597832bb76e8438d14bee564d918831c26b687a4a49387ce8d"}
Sep 10 20:59:03.855273 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.855103    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs" event={"ID":"c1477d2e-55a6-415f-85d0-b325df3980f1","Type":"ContainerStarted","Data":"e554729f003bdde542323486a5d5d90ed9ad5cf2c235f308be1999c5ae2e9c18"}
Sep 10 20:59:03.856804 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.856777    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vmvb8" event={"ID":"22f578b7-ad26-41b3-b9df-8b61f1770171","Type":"ContainerStarted","Data":"85ed4a1db6d73a9388405800af6fc469692deeae89e2232329fd1102ea0dac91"}
Sep 10 20:59:03.858511 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.858485    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nrl66" event={"ID":"445e9438-de21-4264-86b9-3fad26c5ebb3","Type":"ContainerStarted","Data":"25eae5448d16b438069ef9a6b8b87f2546730f444e24cf9ea9301fcb9130e8fe"}
Sep 10 20:59:03.860763 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.860690    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xswnp" event={"ID":"8032963a-4ac1-4d7b-a4de-e9bac1e2c9a1","Type":"ContainerStarted","Data":"de066ba0f36a103c771f6bcc65a2df38d3a2e75e075a70095100e690e54e799e"}
Sep 10 20:59:03.862861 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.862789    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6cwt" event={"ID":"841ad25d-3c20-4383-a1c2-31b401b6a6d1","Type":"ContainerStarted","Data":"fae68b3800a08b834631556fd0fa9757fb525d8cf33541def5d39fe27af24569"}
Sep 10 20:59:03.864467 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.864395    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r282d" event={"ID":"cdafa9fe-0815-45f4-b40b-d41b196f2735","Type":"ContainerStarted","Data":"82b3ef7d21abc35f79be126270c6e67f584d6bd489c864997c28915ed2fddae1"}
Sep 10 20:59:03.873169 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:03.872254    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6tbjl" event={"ID":"dbed4e47-f58b-451c-8205-283e0392aec5","Type":"ContainerStarted","Data":"f46b5759fd7dc2bd88c4c7261523af506d24d2ac33a30082144ff0ab15573353"}
Sep 10 20:59:04.435889 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:04.435843    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-original-pull-secret\") pod \"global-pull-secret-syncer-k8v5s\" (UID: \"eb76c9c7-abc6-4d93-b8e8-f68110f905a9\") " pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:04.436080 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:04.435940    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-metrics-certs\") pod \"network-metrics-daemon-jskzm\" (UID: \"b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e\") " pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:04.436145 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:04.436118    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:59:04.436209 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:04.436192    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-metrics-certs podName:b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e nodeName:}" failed. No retries permitted until 2026-09-10 20:59:06.436170146 +0000 UTC m=+6.101194927 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-metrics-certs") pod "network-metrics-daemon-jskzm" (UID: "b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:59:04.436701 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:04.436672    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:59:04.436834 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:04.436745    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-original-pull-secret podName:eb76c9c7-abc6-4d93-b8e8-f68110f905a9 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:06.436726509 +0000 UTC m=+6.101751307 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-original-pull-secret") pod "global-pull-secret-syncer-k8v5s" (UID: "eb76c9c7-abc6-4d93-b8e8-f68110f905a9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:59:04.537139 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:04.537053    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfzk5\" (UniqueName: \"kubernetes.io/projected/c12d6b27-9952-4214-8b9d-2864f13966f9-kube-api-access-lfzk5\") pod \"network-check-target-6hc86\" (UID: \"c12d6b27-9952-4214-8b9d-2864f13966f9\") " pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:04.537332 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:04.537249    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:59:04.537332 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:04.537271    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:59:04.537332 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:04.537284    2584 projected.go:196] Error preparing data for projected volume kube-api-access-lfzk5 for pod openshift-network-diagnostics/network-check-target-6hc86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:59:04.537528 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:04.537347    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c12d6b27-9952-4214-8b9d-2864f13966f9-kube-api-access-lfzk5 podName:c12d6b27-9952-4214-8b9d-2864f13966f9 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:06.537327391 +0000 UTC m=+6.202352186 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfzk5" (UniqueName: "kubernetes.io/projected/c12d6b27-9952-4214-8b9d-2864f13966f9-kube-api-access-lfzk5") pod "network-check-target-6hc86" (UID: "c12d6b27-9952-4214-8b9d-2864f13966f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:59:04.762589 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:04.762464    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 20:54:03 +0000 UTC" deadline="2028-02-16 03:37:51.319342233 +0000 UTC"
Sep 10 20:59:04.762589 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:04.762537    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12558h38m46.556810055s"
Sep 10 20:59:04.838074 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:04.837325    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:04.838074 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:04.837486    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jskzm" podUID="b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e"
Sep 10 20:59:04.838074 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:04.837595    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:04.838074 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:04.837687    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8v5s" podUID="eb76c9c7-abc6-4d93-b8e8-f68110f905a9"
Sep 10 20:59:05.834433 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:05.834192    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:05.835094 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:05.834555    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6hc86" podUID="c12d6b27-9952-4214-8b9d-2864f13966f9"
Sep 10 20:59:05.886491 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:05.886450    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal" event={"ID":"f0f127f626c5949a577288cc12509aa1","Type":"ContainerStarted","Data":"ad10e98c90667ccf262bcb6702257fa38b99a0d6ce6a682b47c693ecca886fa7"}
Sep 10 20:59:05.901931 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:05.901898    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-205.ec2.internal" event={"ID":"77a4c4cd902dc458728fadabb7f791eb","Type":"ContainerStarted","Data":"06938b1435c82ebc36c1f368734a5f3eb6dc6b187cb8e5aa93d0f1a8160457a3"}
Sep 10 20:59:06.462450 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:06.461684    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-original-pull-secret\") pod \"global-pull-secret-syncer-k8v5s\" (UID: \"eb76c9c7-abc6-4d93-b8e8-f68110f905a9\") " pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:06.462450 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:06.461752    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-metrics-certs\") pod \"network-metrics-daemon-jskzm\" (UID: \"b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e\") " pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:06.462450 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:06.461890    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:59:06.462450 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:06.461951    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-metrics-certs podName:b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e nodeName:}" failed. No retries permitted until 2026-09-10 20:59:10.461931393 +0000 UTC m=+10.126956189 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-metrics-certs") pod "network-metrics-daemon-jskzm" (UID: "b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:59:06.462450 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:06.462345    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:59:06.462450 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:06.462417    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-original-pull-secret podName:eb76c9c7-abc6-4d93-b8e8-f68110f905a9 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:10.462400282 +0000 UTC m=+10.127425078 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-original-pull-secret") pod "global-pull-secret-syncer-k8v5s" (UID: "eb76c9c7-abc6-4d93-b8e8-f68110f905a9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:59:06.563188 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:06.563152    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfzk5\" (UniqueName: \"kubernetes.io/projected/c12d6b27-9952-4214-8b9d-2864f13966f9-kube-api-access-lfzk5\") pod \"network-check-target-6hc86\" (UID: \"c12d6b27-9952-4214-8b9d-2864f13966f9\") " pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:06.563385 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:06.563327    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:59:06.563385 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:06.563347    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:59:06.563385 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:06.563361    2584 projected.go:196] Error preparing data for projected volume kube-api-access-lfzk5 for pod openshift-network-diagnostics/network-check-target-6hc86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:59:06.563618 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:06.563426    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c12d6b27-9952-4214-8b9d-2864f13966f9-kube-api-access-lfzk5 podName:c12d6b27-9952-4214-8b9d-2864f13966f9 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:10.563406847 +0000 UTC m=+10.228431645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfzk5" (UniqueName: "kubernetes.io/projected/c12d6b27-9952-4214-8b9d-2864f13966f9-kube-api-access-lfzk5") pod "network-check-target-6hc86" (UID: "c12d6b27-9952-4214-8b9d-2864f13966f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:59:06.835084 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:06.834995    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:06.835598 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:06.835152    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8v5s" podUID="eb76c9c7-abc6-4d93-b8e8-f68110f905a9"
Sep 10 20:59:06.835715 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:06.835673    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:06.835839 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:06.835814    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jskzm" podUID="b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e"
Sep 10 20:59:07.834679 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:07.834591    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:07.834858 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:07.834750    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6hc86" podUID="c12d6b27-9952-4214-8b9d-2864f13966f9"
Sep 10 20:59:07.909385 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:07.909340    2584 generic.go:356] "Generic (PLEG): container finished" podID="f0f127f626c5949a577288cc12509aa1" containerID="ad10e98c90667ccf262bcb6702257fa38b99a0d6ce6a682b47c693ecca886fa7" exitCode=0
Sep 10 20:59:07.909943 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:07.909414    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal" event={"ID":"f0f127f626c5949a577288cc12509aa1","Type":"ContainerDied","Data":"ad10e98c90667ccf262bcb6702257fa38b99a0d6ce6a682b47c693ecca886fa7"}
Sep 10 20:59:07.931199 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:07.931117    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-205.ec2.internal" podStartSLOduration=5.931095325 podStartE2EDuration="5.931095325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:05.924753492 +0000 UTC m=+5.589778296" watchObservedRunningTime="2026-09-10 20:59:07.931095325 +0000 UTC m=+7.596120129"
Sep 10 20:59:08.834346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:08.834300    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:08.834577 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:08.834469    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jskzm" podUID="b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e"
Sep 10 20:59:08.834577 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:08.834532    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:08.834807 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:08.834664    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8v5s" podUID="eb76c9c7-abc6-4d93-b8e8-f68110f905a9"
Sep 10 20:59:09.834255 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:09.834181    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:09.834695 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:09.834293    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6hc86" podUID="c12d6b27-9952-4214-8b9d-2864f13966f9"
Sep 10 20:59:10.499867 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:10.499159    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-original-pull-secret\") pod \"global-pull-secret-syncer-k8v5s\" (UID: \"eb76c9c7-abc6-4d93-b8e8-f68110f905a9\") " pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:10.499867 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:10.499229    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-metrics-certs\") pod \"network-metrics-daemon-jskzm\" (UID: \"b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e\") " pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:10.499867 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:10.499357    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:59:10.499867 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:10.499419    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-metrics-certs podName:b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e nodeName:}" failed. No retries permitted until 2026-09-10 20:59:18.499400105 +0000 UTC m=+18.164424889 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-metrics-certs") pod "network-metrics-daemon-jskzm" (UID: "b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:59:10.499867 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:10.499502    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:59:10.499867 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:10.499535    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-original-pull-secret podName:eb76c9c7-abc6-4d93-b8e8-f68110f905a9 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:18.499524447 +0000 UTC m=+18.164549243 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-original-pull-secret") pod "global-pull-secret-syncer-k8v5s" (UID: "eb76c9c7-abc6-4d93-b8e8-f68110f905a9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:59:10.600365 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:10.600326    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfzk5\" (UniqueName: \"kubernetes.io/projected/c12d6b27-9952-4214-8b9d-2864f13966f9-kube-api-access-lfzk5\") pod \"network-check-target-6hc86\" (UID: \"c12d6b27-9952-4214-8b9d-2864f13966f9\") " pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:10.600570 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:10.600512    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:59:10.600570 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:10.600538    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:59:10.600570 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:10.600551    2584 projected.go:196] Error preparing data for projected volume kube-api-access-lfzk5 for pod openshift-network-diagnostics/network-check-target-6hc86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:59:10.600732 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:10.600615    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c12d6b27-9952-4214-8b9d-2864f13966f9-kube-api-access-lfzk5 podName:c12d6b27-9952-4214-8b9d-2864f13966f9 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:18.6005944 +0000 UTC m=+18.265619189 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfzk5" (UniqueName: "kubernetes.io/projected/c12d6b27-9952-4214-8b9d-2864f13966f9-kube-api-access-lfzk5") pod "network-check-target-6hc86" (UID: "c12d6b27-9952-4214-8b9d-2864f13966f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:59:10.835595 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:10.835513    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:10.836020 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:10.835640    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jskzm" podUID="b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e"
Sep 10 20:59:10.836020 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:10.835661    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:10.836020 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:10.835743    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8v5s" podUID="eb76c9c7-abc6-4d93-b8e8-f68110f905a9"
Sep 10 20:59:11.834823 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:11.834667    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:11.834823 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:11.834799    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6hc86" podUID="c12d6b27-9952-4214-8b9d-2864f13966f9"
Sep 10 20:59:12.834503 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:12.834333    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:12.834503 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:12.834381    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:12.834503 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:12.834502    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8v5s" podUID="eb76c9c7-abc6-4d93-b8e8-f68110f905a9"
Sep 10 20:59:12.835006 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:12.834778    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jskzm" podUID="b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e"
Sep 10 20:59:13.834293 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:13.834262    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:13.834462 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:13.834363    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6hc86" podUID="c12d6b27-9952-4214-8b9d-2864f13966f9"
Sep 10 20:59:14.834132 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:14.834103    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:14.834690 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:14.834230    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jskzm" podUID="b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e"
Sep 10 20:59:14.834690 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:14.834270    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:14.834690 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:14.834380    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8v5s" podUID="eb76c9c7-abc6-4d93-b8e8-f68110f905a9"
Sep 10 20:59:15.834111 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:15.834077    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:15.834290 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:15.834202    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6hc86" podUID="c12d6b27-9952-4214-8b9d-2864f13966f9"
Sep 10 20:59:16.834746 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:16.834709    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:16.834746 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:16.834726    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:16.835262 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:16.834857    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jskzm" podUID="b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e"
Sep 10 20:59:16.835262 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:16.834979    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8v5s" podUID="eb76c9c7-abc6-4d93-b8e8-f68110f905a9"
Sep 10 20:59:17.834191 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:17.834156    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:17.834414 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:17.834276    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6hc86" podUID="c12d6b27-9952-4214-8b9d-2864f13966f9"
Sep 10 20:59:18.555232 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:18.555191    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-metrics-certs\") pod \"network-metrics-daemon-jskzm\" (UID: \"b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e\") " pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:18.555776 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:18.555282    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-original-pull-secret\") pod \"global-pull-secret-syncer-k8v5s\" (UID: \"eb76c9c7-abc6-4d93-b8e8-f68110f905a9\") " pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:18.555776 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:18.555371    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:59:18.555776 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:18.555401    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:59:18.555776 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:18.555458    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-metrics-certs podName:b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e nodeName:}" failed. No retries permitted until 2026-09-10 20:59:34.555439053 +0000 UTC m=+34.220463837 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-metrics-certs") pod "network-metrics-daemon-jskzm" (UID: "b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:59:18.555776 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:18.555479    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-original-pull-secret podName:eb76c9c7-abc6-4d93-b8e8-f68110f905a9 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:34.555470607 +0000 UTC m=+34.220495385 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-original-pull-secret") pod "global-pull-secret-syncer-k8v5s" (UID: "eb76c9c7-abc6-4d93-b8e8-f68110f905a9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:59:18.655993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:18.655956    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfzk5\" (UniqueName: \"kubernetes.io/projected/c12d6b27-9952-4214-8b9d-2864f13966f9-kube-api-access-lfzk5\") pod \"network-check-target-6hc86\" (UID: \"c12d6b27-9952-4214-8b9d-2864f13966f9\") " pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:18.656177 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:18.656103    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:59:18.656177 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:18.656119    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:59:18.656177 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:18.656128    2584 projected.go:196] Error preparing data for projected volume kube-api-access-lfzk5 for pod openshift-network-diagnostics/network-check-target-6hc86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:59:18.656352 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:18.656180    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c12d6b27-9952-4214-8b9d-2864f13966f9-kube-api-access-lfzk5 podName:c12d6b27-9952-4214-8b9d-2864f13966f9 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:34.656165378 +0000 UTC m=+34.321190163 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfzk5" (UniqueName: "kubernetes.io/projected/c12d6b27-9952-4214-8b9d-2864f13966f9-kube-api-access-lfzk5") pod "network-check-target-6hc86" (UID: "c12d6b27-9952-4214-8b9d-2864f13966f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:59:18.834739 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:18.834653    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:18.834896 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:18.834657    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:18.834896 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:18.834802    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jskzm" podUID="b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e"
Sep 10 20:59:18.834896 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:18.834846    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8v5s" podUID="eb76c9c7-abc6-4d93-b8e8-f68110f905a9"
Sep 10 20:59:19.833872 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:19.833842    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:19.834280 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:19.833954    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6hc86" podUID="c12d6b27-9952-4214-8b9d-2864f13966f9"
Sep 10 20:59:20.835100 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:20.835060    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:20.835599 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:20.835183    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8v5s" podUID="eb76c9c7-abc6-4d93-b8e8-f68110f905a9"
Sep 10 20:59:20.835599 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:20.835249    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:20.835599 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:20.835383    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jskzm" podUID="b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e"
Sep 10 20:59:21.834657 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:21.834477    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:21.834838 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:21.834724    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6hc86" podUID="c12d6b27-9952-4214-8b9d-2864f13966f9"
Sep 10 20:59:21.939187 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:21.938987    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76" event={"ID":"5c065834-094d-4fef-8f9c-ad8e42c34084","Type":"ContainerStarted","Data":"3975c41f4acf582248374b887d71c175587504fd85bf7e5bc9e81c34ba2bfe30"}
Sep 10 20:59:21.940637 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:21.940612    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs" event={"ID":"c1477d2e-55a6-415f-85d0-b325df3980f1","Type":"ContainerStarted","Data":"b423ffc52b344eab1d7dfddea27235f9297ce52f3f5398a466ea62b2bd1b7bab"}
Sep 10 20:59:21.942134 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:21.942111    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vmvb8" event={"ID":"22f578b7-ad26-41b3-b9df-8b61f1770171","Type":"ContainerStarted","Data":"d4babd1b43bd490957ee6330bf1d6869d675672acfc4304e92297f587a73e0e1"}
Sep 10 20:59:21.945849 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:21.945811    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal" event={"ID":"f0f127f626c5949a577288cc12509aa1","Type":"ContainerStarted","Data":"192d9a844e0f22d2ca951cff52ea4a923ce0f599ef1be34133464244ab42039e"}
Sep 10 20:59:21.948691 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:21.948665    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nrl66" event={"ID":"445e9438-de21-4264-86b9-3fad26c5ebb3","Type":"ContainerStarted","Data":"3017b9ccae6ad0bc7868a74489095946393249d412df2570fa13bb8606076632"}
Sep 10 20:59:21.950233 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:21.950208    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xswnp" event={"ID":"8032963a-4ac1-4d7b-a4de-e9bac1e2c9a1","Type":"ContainerStarted","Data":"3dd31cbd6effc0b7097aea94cd0d38d4013bde6edd1808eaf8c4796733f123a1"}
Sep 10 20:59:21.951665 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:21.951645    2584 generic.go:356] "Generic (PLEG): container finished" podID="841ad25d-3c20-4383-a1c2-31b401b6a6d1" containerID="56b480d1270825f425e8e088b03ecd2ed78d5f879c12273550e4082cf81cbc4b" exitCode=0
Sep 10 20:59:21.951737 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:21.951706    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6cwt" event={"ID":"841ad25d-3c20-4383-a1c2-31b401b6a6d1","Type":"ContainerDied","Data":"56b480d1270825f425e8e088b03ecd2ed78d5f879c12273550e4082cf81cbc4b"}
Sep 10 20:59:21.953216 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:21.953195    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r282d" event={"ID":"cdafa9fe-0815-45f4-b40b-d41b196f2735","Type":"ContainerStarted","Data":"6512bc0df890acd6173e323e66b6c2f20dbf22cc37139c0d90c5cb4c231d5307"}
Sep 10 20:59:21.954764 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:21.954738    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wprl7" event={"ID":"6aa58f1a-77a8-4479-a7a8-a68b6a4673cc","Type":"ContainerStarted","Data":"c7d524eeca7346fc5257d5162c6d917dcc286018e1931ba393f2df12a991273f"}
Sep 10 20:59:21.964470 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:21.964400    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vmvb8" podStartSLOduration=2.757182749 podStartE2EDuration="20.964382858s" totalImagesPullingTime="18.207200109s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:21.964331678 +0000 UTC m=+21.629356477" watchObservedRunningTime="2026-09-10 20:59:21.964382858 +0000 UTC m=+21.629407660"
Sep 10 20:59:22.023705 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:22.023657    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-r282d" podStartSLOduration=2.842515187 podStartE2EDuration="21.023640424s" totalImagesPullingTime="18.181125237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:21.991050994 +0000 UTC m=+21.656075793" watchObservedRunningTime="2026-09-10 20:59:22.023640424 +0000 UTC m=+21.688665228"
Sep 10 20:59:22.057006 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:22.056955    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal" podStartSLOduration=20.056937823 podStartE2EDuration="20.056937823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:22.054537394 +0000 UTC m=+21.719562195" watchObservedRunningTime="2026-09-10 20:59:22.056937823 +0000 UTC m=+21.721962624"
Sep 10 20:59:22.093078 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:22.093035    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xswnp" podStartSLOduration=2.933492644 podStartE2EDuration="21.093018791s" totalImagesPullingTime="18.159526147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:22.075638371 +0000 UTC m=+21.740663171" watchObservedRunningTime="2026-09-10 20:59:22.093018791 +0000 UTC m=+21.758043643"
Sep 10 20:59:22.093373 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:22.093349    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wprl7" podStartSLOduration=6.591755791 podStartE2EDuration="21.093342591s" totalImagesPullingTime="14.5015868s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:22.093298153 +0000 UTC m=+21.758322952" watchObservedRunningTime="2026-09-10 20:59:22.093342591 +0000 UTC m=+21.758367391"
Sep 10 20:59:22.115278 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:22.115239    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nrl66" podStartSLOduration=2.931959088 podStartE2EDuration="21.115223652s" totalImagesPullingTime="18.183264564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:22.115089872 +0000 UTC m=+21.780114672" watchObservedRunningTime="2026-09-10 20:59:22.115223652 +0000 UTC m=+21.780248453"
Sep 10 20:59:22.734571 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:22.734546    2584 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 20:59:22.777141 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:22.776978    2584 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T20:59:22.734567328Z","UUID":"ea4fa291-d6fd-466e-b0df-9912006593a3","Handler":null,"Name":"","Endpoint":""}
Sep 10 20:59:22.778607 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:22.778585    2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 20:59:22.778734 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:22.778618    2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 20:59:22.834470 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:22.834442    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:22.834634 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:22.834563    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8v5s" podUID="eb76c9c7-abc6-4d93-b8e8-f68110f905a9"
Sep 10 20:59:22.834634 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:22.834612    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:22.834753 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:22.834737    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jskzm" podUID="b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e"
Sep 10 20:59:22.959720 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:22.959578    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6tbjl" event={"ID":"dbed4e47-f58b-451c-8205-283e0392aec5","Type":"ContainerStarted","Data":"86b191e7e4d45164f08e009cec45f7eaa9256e75434bc12cd054d467b2bfee12"}
Sep 10 20:59:22.961764 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:22.961736    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76" event={"ID":"5c065834-094d-4fef-8f9c-ad8e42c34084","Type":"ContainerStarted","Data":"30daa7b7238e35db7e3a2391d48f0199973e7bb074cc16f6d55f2b8d0d12ce74"}
Sep 10 20:59:22.965476 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:22.965443    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs" event={"ID":"c1477d2e-55a6-415f-85d0-b325df3980f1","Type":"ContainerStarted","Data":"3e6b2626ceba7360ad69b0c1f3b24231c00ab074d7a7e43df440383ec0db2f85"}
Sep 10 20:59:22.965578 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:22.965516    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs" event={"ID":"c1477d2e-55a6-415f-85d0-b325df3980f1","Type":"ContainerStarted","Data":"4557dd1f74a4dcb781969649423652df98db961c9cb1f05cb45b95cc9354d762"}
Sep 10 20:59:22.965578 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:22.965556    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs" event={"ID":"c1477d2e-55a6-415f-85d0-b325df3980f1","Type":"ContainerStarted","Data":"c295b39f7ffdd7345ad6bfede7ae1c6e280e92326e22d1d5da0784bc5bc5a3b2"}
Sep 10 20:59:22.965690 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:22.965589    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs" event={"ID":"c1477d2e-55a6-415f-85d0-b325df3980f1","Type":"ContainerStarted","Data":"f33292a2d558fc9c7e228a06856a9aad094644814ea17ef0785cbb125ea5835a"}
Sep 10 20:59:22.965690 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:22.965620    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs" event={"ID":"c1477d2e-55a6-415f-85d0-b325df3980f1","Type":"ContainerStarted","Data":"711d13cc3007abd3fffc1d1467ef8566028ddefc1c38a10cb4ee130ebfc82666"}
Sep 10 20:59:22.978097 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:22.978045    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6tbjl" podStartSLOduration=7.44067821 podStartE2EDuration="21.978027783s" totalImagesPullingTime="14.537349573s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:22.977292882 +0000 UTC m=+22.642317684" watchObservedRunningTime="2026-09-10 20:59:22.978027783 +0000 UTC m=+22.643052583"
Sep 10 20:59:23.834343 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:23.834304    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:23.834557 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:23.834468    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6hc86" podUID="c12d6b27-9952-4214-8b9d-2864f13966f9"
Sep 10 20:59:24.834933 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:24.834894    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:24.835380 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:24.835052    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jskzm" podUID="b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e"
Sep 10 20:59:24.835380 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:24.835117    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:24.835380 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:24.835243    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8v5s" podUID="eb76c9c7-abc6-4d93-b8e8-f68110f905a9"
Sep 10 20:59:24.972185 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:24.972148    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76" event={"ID":"5c065834-094d-4fef-8f9c-ad8e42c34084","Type":"ContainerStarted","Data":"07ff5c9d3cf0eb860bb11afe16a568781f889110dd9b994c497069fd4f872013"}
Sep 10 20:59:24.975984 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:24.975948    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs" event={"ID":"c1477d2e-55a6-415f-85d0-b325df3980f1","Type":"ContainerStarted","Data":"91ab0978a121a3e8a5f650ac2b8d91d6171c21dfe3a554d99d0ac6ad204752ff"}
Sep 10 20:59:24.994947 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:24.994892    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vct76" podStartSLOduration=3.150580835 podStartE2EDuration="23.994874665s" totalImagesPullingTime="20.84429383s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:59:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:24.994254572 +0000 UTC m=+24.659279388" watchObservedRunningTime="2026-09-10 20:59:24.994874665 +0000 UTC m=+24.659899462"
Sep 10 20:59:25.136915 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:25.136882    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-r282d"
Sep 10 20:59:25.137514 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:25.137493    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-r282d"
Sep 10 20:59:25.834845 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:25.834808    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:25.835003 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:25.834925    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6hc86" podUID="c12d6b27-9952-4214-8b9d-2864f13966f9"
Sep 10 20:59:25.978245 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:25.978210    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-r282d"
Sep 10 20:59:25.978736 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:25.978714    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r282d"
Sep 10 20:59:26.834957 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:26.834716    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:26.835089 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:26.834760    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:26.835699 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:26.835082    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jskzm" podUID="b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e"
Sep 10 20:59:26.835699 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:26.835112    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8v5s" podUID="eb76c9c7-abc6-4d93-b8e8-f68110f905a9"
Sep 10 20:59:26.984233 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:26.984206    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs" event={"ID":"c1477d2e-55a6-415f-85d0-b325df3980f1","Type":"ContainerStarted","Data":"e181520201f4e226a7fb939934732700f9a1f8a5d4ad03c7f9bf64db218e01ce"}
Sep 10 20:59:26.984462 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:26.984443    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:26.986171 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:26.986134    2584 generic.go:356] "Generic (PLEG): container finished" podID="841ad25d-3c20-4383-a1c2-31b401b6a6d1" containerID="46cdd79029a00df66adfef18a3924bfd227c930f78ad8bf6178777227772ff5f" exitCode=0
Sep 10 20:59:26.986287 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:26.986199    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6cwt" event={"ID":"841ad25d-3c20-4383-a1c2-31b401b6a6d1","Type":"ContainerDied","Data":"46cdd79029a00df66adfef18a3924bfd227c930f78ad8bf6178777227772ff5f"}
Sep 10 20:59:27.000655 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:27.000523    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:27.016994 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:27.016957    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs" podStartSLOduration=7.293852791 podStartE2EDuration="26.016944484s" totalImagesPullingTime="18.723091693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:27.016651544 +0000 UTC m=+26.681676343" watchObservedRunningTime="2026-09-10 20:59:27.016944484 +0000 UTC m=+26.681969320"
Sep 10 20:59:27.834182 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:27.834160    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:27.834298 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:27.834275    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6hc86" podUID="c12d6b27-9952-4214-8b9d-2864f13966f9"
Sep 10 20:59:27.990730 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:27.990642    2584 generic.go:356] "Generic (PLEG): container finished" podID="841ad25d-3c20-4383-a1c2-31b401b6a6d1" containerID="249a3df683a8d9112ec4b321afd9f44241e0725cd3aa830a74a32ee6c32da9c1" exitCode=0
Sep 10 20:59:27.991095 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:27.990727    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6cwt" event={"ID":"841ad25d-3c20-4383-a1c2-31b401b6a6d1","Type":"ContainerDied","Data":"249a3df683a8d9112ec4b321afd9f44241e0725cd3aa830a74a32ee6c32da9c1"}
Sep 10 20:59:27.991452 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:27.991414    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:27.991570 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:27.991504    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:28.006093 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:28.006073    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 20:59:28.595407 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:28.595362    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jskzm"]
Sep 10 20:59:28.595610 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:28.595527    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:28.595663 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:28.595619    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jskzm" podUID="b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e"
Sep 10 20:59:28.595933 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:28.595912    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6hc86"]
Sep 10 20:59:28.595999 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:28.595985    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:28.596070 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:28.596054    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6hc86" podUID="c12d6b27-9952-4214-8b9d-2864f13966f9"
Sep 10 20:59:28.596824 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:28.596803    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k8v5s"]
Sep 10 20:59:28.596908 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:28.596896    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:28.596999 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:28.596977    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8v5s" podUID="eb76c9c7-abc6-4d93-b8e8-f68110f905a9"
Sep 10 20:59:28.997845 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:28.997812    2584 generic.go:356] "Generic (PLEG): container finished" podID="841ad25d-3c20-4383-a1c2-31b401b6a6d1" containerID="d9d65027227fbc0a533d9fbc673036e86cd7c5c9d7f2391c8290fecc7d396b04" exitCode=0
Sep 10 20:59:28.998116 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:28.997869    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6cwt" event={"ID":"841ad25d-3c20-4383-a1c2-31b401b6a6d1","Type":"ContainerDied","Data":"d9d65027227fbc0a533d9fbc673036e86cd7c5c9d7f2391c8290fecc7d396b04"}
Sep 10 20:59:29.834732 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:29.834465    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:29.834902 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:29.834507    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:29.834902 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:29.834844    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jskzm" podUID="b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e"
Sep 10 20:59:29.835012 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:29.834892    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8v5s" podUID="eb76c9c7-abc6-4d93-b8e8-f68110f905a9"
Sep 10 20:59:30.835156 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:30.835123    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:30.835573 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:30.835226    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6hc86" podUID="c12d6b27-9952-4214-8b9d-2864f13966f9"
Sep 10 20:59:31.834185 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:31.834148    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:31.834339 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:31.834150    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:31.834339 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:31.834284    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8v5s" podUID="eb76c9c7-abc6-4d93-b8e8-f68110f905a9"
Sep 10 20:59:31.834457 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:31.834376    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jskzm" podUID="b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e"
Sep 10 20:59:32.833964 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:32.833927    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:32.834548 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:32.834046    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6hc86" podUID="c12d6b27-9952-4214-8b9d-2864f13966f9"
Sep 10 20:59:33.834109 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:33.834081    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:33.834109 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:33.834117    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:33.834673 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:33.834201    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jskzm" podUID="b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e"
Sep 10 20:59:33.834673 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:33.834319    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8v5s" podUID="eb76c9c7-abc6-4d93-b8e8-f68110f905a9"
Sep 10 20:59:34.013220 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:34.013142    2584 generic.go:356] "Generic (PLEG): container finished" podID="841ad25d-3c20-4383-a1c2-31b401b6a6d1" containerID="0296454f136456b7f9403a215478577f085e81411690e3d1abff1ee9b8e960ef" exitCode=0
Sep 10 20:59:34.013220 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:34.013212    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6cwt" event={"ID":"841ad25d-3c20-4383-a1c2-31b401b6a6d1","Type":"ContainerDied","Data":"0296454f136456b7f9403a215478577f085e81411690e3d1abff1ee9b8e960ef"}
Sep 10 20:59:34.570632 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:34.570596    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-original-pull-secret\") pod \"global-pull-secret-syncer-k8v5s\" (UID: \"eb76c9c7-abc6-4d93-b8e8-f68110f905a9\") " pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:34.570800 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:34.570648    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-metrics-certs\") pod \"network-metrics-daemon-jskzm\" (UID: \"b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e\") " pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:34.570800 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:34.570743    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:59:34.570800 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:34.570796    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-metrics-certs podName:b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e nodeName:}" failed. No retries permitted until 2026-09-10 21:00:06.570782797 +0000 UTC m=+66.235807574 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-metrics-certs") pod "network-metrics-daemon-jskzm" (UID: "b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:59:34.570932 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:34.570743    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:59:34.570932 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:34.570862    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-original-pull-secret podName:eb76c9c7-abc6-4d93-b8e8-f68110f905a9 nodeName:}" failed. No retries permitted until 2026-09-10 21:00:06.570849424 +0000 UTC m=+66.235874217 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-original-pull-secret") pod "global-pull-secret-syncer-k8v5s" (UID: "eb76c9c7-abc6-4d93-b8e8-f68110f905a9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:59:34.671234 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:34.671198    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfzk5\" (UniqueName: \"kubernetes.io/projected/c12d6b27-9952-4214-8b9d-2864f13966f9-kube-api-access-lfzk5\") pod \"network-check-target-6hc86\" (UID: \"c12d6b27-9952-4214-8b9d-2864f13966f9\") " pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:34.671419 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:34.671323    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:59:34.671419 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:34.671337    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:59:34.671419 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:34.671346    2584 projected.go:196] Error preparing data for projected volume kube-api-access-lfzk5 for pod openshift-network-diagnostics/network-check-target-6hc86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:59:34.671419 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:34.671394    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c12d6b27-9952-4214-8b9d-2864f13966f9-kube-api-access-lfzk5 podName:c12d6b27-9952-4214-8b9d-2864f13966f9 nodeName:}" failed. No retries permitted until 2026-09-10 21:00:06.671380127 +0000 UTC m=+66.336404905 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfzk5" (UniqueName: "kubernetes.io/projected/c12d6b27-9952-4214-8b9d-2864f13966f9-kube-api-access-lfzk5") pod "network-check-target-6hc86" (UID: "c12d6b27-9952-4214-8b9d-2864f13966f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:59:34.834033 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:34.833959    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:34.834179 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:34.834068    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6hc86" podUID="c12d6b27-9952-4214-8b9d-2864f13966f9"
Sep 10 20:59:35.018968 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.018938    2584 generic.go:356] "Generic (PLEG): container finished" podID="841ad25d-3c20-4383-a1c2-31b401b6a6d1" containerID="fc70f31201b9c4f87380663248f5d56f64777bed8ce972848bfc6dc80da02d6c" exitCode=0
Sep 10 20:59:35.019118 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.018978    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6cwt" event={"ID":"841ad25d-3c20-4383-a1c2-31b401b6a6d1","Type":"ContainerDied","Data":"fc70f31201b9c4f87380663248f5d56f64777bed8ce972848bfc6dc80da02d6c"}
Sep 10 20:59:35.651577 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.651549    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeReady"
Sep 10 20:59:35.651745 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.651661    2584 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 20:59:35.701844 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.701815    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dzgxv"]
Sep 10 20:59:35.716838 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.716815    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ql42f"]
Sep 10 20:59:35.716947 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.716930    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dzgxv"
Sep 10 20:59:35.719303 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.719256    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 20:59:35.719440 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.719350    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 20:59:35.719440 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.719374    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7j9n7\""
Sep 10 20:59:35.743148 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.743119    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dzgxv"]
Sep 10 20:59:35.743148 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.743153    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ql42f"]
Sep 10 20:59:35.743324 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.743238    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ql42f"
Sep 10 20:59:35.746396 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.746375    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 20:59:35.746873 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.746856    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 20:59:35.746989 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.746972    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9vpqg\""
Sep 10 20:59:35.747051 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.746979    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 20:59:35.834399 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.834370    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 20:59:35.834899 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.834416    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 20:59:35.837671 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.837655    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5khf6\""
Sep 10 20:59:35.838134 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.838120    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 20:59:35.838188 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.838124    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 20:59:35.880228 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.880204    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dfe9291-e76b-4d25-91bf-4c3d78d62841-tmp-dir\") pod \"dns-default-dzgxv\" (UID: \"3dfe9291-e76b-4d25-91bf-4c3d78d62841\") " pod="openshift-dns/dns-default-dzgxv"
Sep 10 20:59:35.880314 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.880251    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3dfe9291-e76b-4d25-91bf-4c3d78d62841-config-volume\") pod \"dns-default-dzgxv\" (UID: \"3dfe9291-e76b-4d25-91bf-4c3d78d62841\") " pod="openshift-dns/dns-default-dzgxv"
Sep 10 20:59:35.880314 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.880282    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3dfe9291-e76b-4d25-91bf-4c3d78d62841-metrics-tls\") pod \"dns-default-dzgxv\" (UID: \"3dfe9291-e76b-4d25-91bf-4c3d78d62841\") " pod="openshift-dns/dns-default-dzgxv"
Sep 10 20:59:35.880314 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.880301    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8bn7\" (UniqueName: \"kubernetes.io/projected/e3464d4e-8d40-4389-9be1-49529d8636ae-kube-api-access-n8bn7\") pod \"ingress-canary-ql42f\" (UID: \"e3464d4e-8d40-4389-9be1-49529d8636ae\") " pod="openshift-ingress-canary/ingress-canary-ql42f"
Sep 10 20:59:35.880412 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.880319    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3464d4e-8d40-4389-9be1-49529d8636ae-cert\") pod \"ingress-canary-ql42f\" (UID: \"e3464d4e-8d40-4389-9be1-49529d8636ae\") " pod="openshift-ingress-canary/ingress-canary-ql42f"
Sep 10 20:59:35.880412 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.880338    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mmdm\" (UniqueName: \"kubernetes.io/projected/3dfe9291-e76b-4d25-91bf-4c3d78d62841-kube-api-access-9mmdm\") pod \"dns-default-dzgxv\" (UID: \"3dfe9291-e76b-4d25-91bf-4c3d78d62841\") " pod="openshift-dns/dns-default-dzgxv"
Sep 10 20:59:35.981093 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.981029    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3dfe9291-e76b-4d25-91bf-4c3d78d62841-metrics-tls\") pod \"dns-default-dzgxv\" (UID: \"3dfe9291-e76b-4d25-91bf-4c3d78d62841\") " pod="openshift-dns/dns-default-dzgxv"
Sep 10 20:59:35.981093 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.981061    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8bn7\" (UniqueName: \"kubernetes.io/projected/e3464d4e-8d40-4389-9be1-49529d8636ae-kube-api-access-n8bn7\") pod \"ingress-canary-ql42f\" (UID: \"e3464d4e-8d40-4389-9be1-49529d8636ae\") " pod="openshift-ingress-canary/ingress-canary-ql42f"
Sep 10 20:59:35.981093 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.981078    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3464d4e-8d40-4389-9be1-49529d8636ae-cert\") pod \"ingress-canary-ql42f\" (UID: \"e3464d4e-8d40-4389-9be1-49529d8636ae\") " pod="openshift-ingress-canary/ingress-canary-ql42f"
Sep 10 20:59:35.981299 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.981109    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mmdm\" (UniqueName: \"kubernetes.io/projected/3dfe9291-e76b-4d25-91bf-4c3d78d62841-kube-api-access-9mmdm\") pod \"dns-default-dzgxv\" (UID: \"3dfe9291-e76b-4d25-91bf-4c3d78d62841\") " pod="openshift-dns/dns-default-dzgxv"
Sep 10 20:59:35.981299 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:35.981193    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:59:35.981299 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.981244    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dfe9291-e76b-4d25-91bf-4c3d78d62841-tmp-dir\") pod \"dns-default-dzgxv\" (UID: \"3dfe9291-e76b-4d25-91bf-4c3d78d62841\") " pod="openshift-dns/dns-default-dzgxv"
Sep 10 20:59:35.981299 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:35.981256    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dfe9291-e76b-4d25-91bf-4c3d78d62841-metrics-tls podName:3dfe9291-e76b-4d25-91bf-4c3d78d62841 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:36.481237121 +0000 UTC m=+36.146261912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3dfe9291-e76b-4d25-91bf-4c3d78d62841-metrics-tls") pod "dns-default-dzgxv" (UID: "3dfe9291-e76b-4d25-91bf-4c3d78d62841") : secret "dns-default-metrics-tls" not found
Sep 10 20:59:35.981515 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.981326    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3dfe9291-e76b-4d25-91bf-4c3d78d62841-config-volume\") pod \"dns-default-dzgxv\" (UID: \"3dfe9291-e76b-4d25-91bf-4c3d78d62841\") " pod="openshift-dns/dns-default-dzgxv"
Sep 10 20:59:35.981572 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.981552    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dfe9291-e76b-4d25-91bf-4c3d78d62841-tmp-dir\") pod \"dns-default-dzgxv\" (UID: \"3dfe9291-e76b-4d25-91bf-4c3d78d62841\") " pod="openshift-dns/dns-default-dzgxv"
Sep 10 20:59:35.981837 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.981817    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3dfe9291-e76b-4d25-91bf-4c3d78d62841-config-volume\") pod \"dns-default-dzgxv\" (UID: \"3dfe9291-e76b-4d25-91bf-4c3d78d62841\") " pod="openshift-dns/dns-default-dzgxv"
Sep 10 20:59:35.985126 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.985002    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3464d4e-8d40-4389-9be1-49529d8636ae-cert\") pod \"ingress-canary-ql42f\" (UID: \"e3464d4e-8d40-4389-9be1-49529d8636ae\") " pod="openshift-ingress-canary/ingress-canary-ql42f"
Sep 10 20:59:35.990910 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.990890    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8bn7\" (UniqueName: \"kubernetes.io/projected/e3464d4e-8d40-4389-9be1-49529d8636ae-kube-api-access-n8bn7\") pod \"ingress-canary-ql42f\" (UID: \"e3464d4e-8d40-4389-9be1-49529d8636ae\") " pod="openshift-ingress-canary/ingress-canary-ql42f"
Sep 10 20:59:35.991844 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:35.991828    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mmdm\" (UniqueName: \"kubernetes.io/projected/3dfe9291-e76b-4d25-91bf-4c3d78d62841-kube-api-access-9mmdm\") pod \"dns-default-dzgxv\" (UID: \"3dfe9291-e76b-4d25-91bf-4c3d78d62841\") " pod="openshift-dns/dns-default-dzgxv"
Sep 10 20:59:36.024874 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:36.024849    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6cwt" event={"ID":"841ad25d-3c20-4383-a1c2-31b401b6a6d1","Type":"ContainerStarted","Data":"78e22dd8a9f7b7353a44f4c90f470d2398c1ba71c5af354811e282ece3e0adb2"}
Sep 10 20:59:36.050993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:36.050967    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ql42f"
Sep 10 20:59:36.052901 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:36.052860    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-c6cwt" podStartSLOduration=6.313771351 podStartE2EDuration="35.05284332s" totalImagesPullingTime="28.739071969s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:59:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:36.051137383 +0000 UTC m=+35.716162193" watchObservedRunningTime="2026-09-10 20:59:36.05284332 +0000 UTC m=+35.717868121"
Sep 10 20:59:36.368548 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:36.368325    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ql42f"]
Sep 10 20:59:36.371985 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:36.371957    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3464d4e_8d40_4389_9be1_49529d8636ae.slice/crio-75d758c248010c92d5427f69677f8d976a138aecd5e01d1463b2ba8424e3f3ad WatchSource:0}: Error finding container 75d758c248010c92d5427f69677f8d976a138aecd5e01d1463b2ba8424e3f3ad: Status 404 returned error can't find the container with id 75d758c248010c92d5427f69677f8d976a138aecd5e01d1463b2ba8424e3f3ad
Sep 10 20:59:36.485924 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:36.485896    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3dfe9291-e76b-4d25-91bf-4c3d78d62841-metrics-tls\") pod \"dns-default-dzgxv\" (UID: \"3dfe9291-e76b-4d25-91bf-4c3d78d62841\") " pod="openshift-dns/dns-default-dzgxv"
Sep 10 20:59:36.486074 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:36.486050    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:59:36.486117 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:36.486109    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dfe9291-e76b-4d25-91bf-4c3d78d62841-metrics-tls podName:3dfe9291-e76b-4d25-91bf-4c3d78d62841 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:37.486093733 +0000 UTC m=+37.151118539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3dfe9291-e76b-4d25-91bf-4c3d78d62841-metrics-tls") pod "dns-default-dzgxv" (UID: "3dfe9291-e76b-4d25-91bf-4c3d78d62841") : secret "dns-default-metrics-tls" not found
Sep 10 20:59:36.833965 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:36.833931    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 20:59:36.836744 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:36.836724    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 20:59:36.837357 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:36.837340    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 20:59:36.837419 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:36.837338    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wb7zl\""
Sep 10 20:59:37.027743 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:37.027710    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ql42f" event={"ID":"e3464d4e-8d40-4389-9be1-49529d8636ae","Type":"ContainerStarted","Data":"75d758c248010c92d5427f69677f8d976a138aecd5e01d1463b2ba8424e3f3ad"}
Sep 10 20:59:37.493488 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:37.493456    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3dfe9291-e76b-4d25-91bf-4c3d78d62841-metrics-tls\") pod \"dns-default-dzgxv\" (UID: \"3dfe9291-e76b-4d25-91bf-4c3d78d62841\") " pod="openshift-dns/dns-default-dzgxv"
Sep 10 20:59:37.493654 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:37.493579    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:59:37.493654 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:37.493647    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dfe9291-e76b-4d25-91bf-4c3d78d62841-metrics-tls podName:3dfe9291-e76b-4d25-91bf-4c3d78d62841 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:39.493630575 +0000 UTC m=+39.158655353 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3dfe9291-e76b-4d25-91bf-4c3d78d62841-metrics-tls") pod "dns-default-dzgxv" (UID: "3dfe9291-e76b-4d25-91bf-4c3d78d62841") : secret "dns-default-metrics-tls" not found
Sep 10 20:59:38.845234 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.845198    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-zkbqs"]
Sep 10 20:59:38.872476 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.872399    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-zkbqs"]
Sep 10 20:59:38.872476 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.872450    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-pjkzj"]
Sep 10 20:59:38.872652 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.872547    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zkbqs"
Sep 10 20:59:38.876882 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.876630    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 10 20:59:38.877043 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.876649    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 20:59:38.877110 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.876694    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xjdhx\""
Sep 10 20:59:38.877257 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.876748    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 20:59:38.877360 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.876807    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 10 20:59:38.890268 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.890247    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-pjkzj"]
Sep 10 20:59:38.890398 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.890333    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pjkzj"
Sep 10 20:59:38.892744 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.892724    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 10 20:59:38.892854 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.892760    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-nsbld\""
Sep 10 20:59:38.892854 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.892780    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 10 20:59:38.944403 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.944371    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnf52"]
Sep 10 20:59:38.969184 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.969156    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-hpt25"]
Sep 10 20:59:38.969346 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.969296    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnf52"
Sep 10 20:59:38.972785 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.971656    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 10 20:59:38.972910 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.972874    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 10 20:59:38.973015 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.972993    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-qcfk7\""
Sep 10 20:59:38.973090 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.973028    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 10 20:59:38.973090 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.973082    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:59:38.985447 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.985410    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-jkkfm"]
Sep 10 20:59:38.985546 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.985485    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-hpt25"
Sep 10 20:59:38.988476 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.988455    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 20:59:38.989076 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.989055    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 10 20:59:38.989355 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.989340    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-44x2w\""
Sep 10 20:59:38.989808 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.989722    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 10 20:59:38.990629 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:38.990596    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 20:59:39.005889 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.005856    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/93462573-692a-4246-afd9-08c9430c7f53-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-zkbqs\" (UID: \"93462573-692a-4246-afd9-08c9430c7f53\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zkbqs"
Sep 10 20:59:39.005976 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.005897    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w89mc\" (UniqueName: \"kubernetes.io/projected/93462573-692a-4246-afd9-08c9430c7f53-kube-api-access-w89mc\") pod \"cluster-monitoring-operator-6cd954b897-zkbqs\" (UID: \"93462573-692a-4246-afd9-08c9430c7f53\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zkbqs"
Sep 10 20:59:39.006033 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.005972    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/29f5e959-7b2f-42e9-98f2-a3547859bb56-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-pjkzj\" (UID: \"29f5e959-7b2f-42e9-98f2-a3547859bb56\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pjkzj"
Sep 10 20:59:39.006033 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.005998    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29f5e959-7b2f-42e9-98f2-a3547859bb56-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-pjkzj\" (UID: \"29f5e959-7b2f-42e9-98f2-a3547859bb56\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pjkzj"
Sep 10 20:59:39.006033 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.006017    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/93462573-692a-4246-afd9-08c9430c7f53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zkbqs\" (UID: \"93462573-692a-4246-afd9-08c9430c7f53\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zkbqs"
Sep 10 20:59:39.006476 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.006459    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnf52"]
Sep 10 20:59:39.006547 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.006496    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-hpt25"]
Sep 10 20:59:39.006547 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.006515    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-jkkfm"]
Sep 10 20:59:39.006639 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.006598    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-jkkfm"
Sep 10 20:59:39.006729 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.006711    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 10 20:59:39.008778 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.008759    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 10 20:59:39.008871 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.008792    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-5fgfx\""
Sep 10 20:59:39.009197 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.009175    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 10 20:59:39.009197 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.009192    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:59:39.009473 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.009454    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 10 20:59:39.018793 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.018771    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 10 20:59:39.045168 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.045137    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qsxcs"]
Sep 10 20:59:39.064568 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.064516    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qsxcs"]
Sep 10 20:59:39.064698 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.064650    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qsxcs"
Sep 10 20:59:39.066964 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.066943    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 10 20:59:39.067075 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.066943    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:59:39.067075 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.066945    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 10 20:59:39.067075 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.067057    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9lchx\""
Sep 10 20:59:39.106773 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.106744    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/93462573-692a-4246-afd9-08c9430c7f53-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-zkbqs\" (UID: \"93462573-692a-4246-afd9-08c9430c7f53\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zkbqs"
Sep 10 20:59:39.106949 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.106783    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fd81843e-6283-43e3-a571-d5e47be26e38-config\") pod \"console-operator-696884fb9c-jkkfm\" (UID: \"fd81843e-6283-43e3-a571-d5e47be26e38\") " pod="openshift-console-operator/console-operator-696884fb9c-jkkfm"
Sep 10 20:59:39.106949 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.106817    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w89mc\" (UniqueName: \"kubernetes.io/projected/93462573-692a-4246-afd9-08c9430c7f53-kube-api-access-w89mc\") pod \"cluster-monitoring-operator-6cd954b897-zkbqs\" (UID: \"93462573-692a-4246-afd9-08c9430c7f53\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zkbqs"
Sep 10 20:59:39.106949 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.106852    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/367d0cac-af3c-4dce-acc3-ce2e5f85fce1-tmp\") pod \"insights-operator-5844b856db-hpt25\" (UID: \"367d0cac-af3c-4dce-acc3-ce2e5f85fce1\") " pod="openshift-insights/insights-operator-5844b856db-hpt25"
Sep 10 20:59:39.106949 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.106877    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zr4b\" (UniqueName: \"kubernetes.io/projected/367d0cac-af3c-4dce-acc3-ce2e5f85fce1-kube-api-access-7zr4b\") pod \"insights-operator-5844b856db-hpt25\" (UID: \"367d0cac-af3c-4dce-acc3-ce2e5f85fce1\") " pod="openshift-insights/insights-operator-5844b856db-hpt25"
Sep 10 20:59:39.106949 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.106905    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd81843e-6283-43e3-a571-d5e47be26e38-trusted-ca\") pod \"console-operator-696884fb9c-jkkfm\" (UID: \"fd81843e-6283-43e3-a571-d5e47be26e38\") " pod="openshift-console-operator/console-operator-696884fb9c-jkkfm"
Sep 10 20:59:39.106949 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.106930    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbqfw\" (UniqueName: \"kubernetes.io/projected/fd81843e-6283-43e3-a571-d5e47be26e38-kube-api-access-gbqfw\") pod \"console-operator-696884fb9c-jkkfm\" (UID: \"fd81843e-6283-43e3-a571-d5e47be26e38\") " pod="openshift-console-operator/console-operator-696884fb9c-jkkfm"
Sep 10 20:59:39.107290 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.107004    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/29f5e959-7b2f-42e9-98f2-a3547859bb56-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-pjkzj\" (UID: \"29f5e959-7b2f-42e9-98f2-a3547859bb56\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pjkzj"
Sep 10 20:59:39.107290 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.107051    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29f5e959-7b2f-42e9-98f2-a3547859bb56-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-pjkzj\" (UID: \"29f5e959-7b2f-42e9-98f2-a3547859bb56\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pjkzj"
Sep 10 20:59:39.107290 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.107077    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/93462573-692a-4246-afd9-08c9430c7f53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zkbqs\" (UID: \"93462573-692a-4246-afd9-08c9430c7f53\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zkbqs"
Sep 10 20:59:39.107290 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.107100    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dbc735a8-26d3-4267-afe5-b334f6e03fd8-serving-cert\") pod \"service-ca-operator-cf7c776b7-xnf52\" (UID: \"dbc735a8-26d3-4267-afe5-b334f6e03fd8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnf52"
Sep 10 20:59:39.107290 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.107131    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffzz8\" (UniqueName: \"kubernetes.io/projected/dbc735a8-26d3-4267-afe5-b334f6e03fd8-kube-api-access-ffzz8\") pod \"service-ca-operator-cf7c776b7-xnf52\" (UID: \"dbc735a8-26d3-4267-afe5-b334f6e03fd8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnf52"
Sep 10 20:59:39.107290 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.107155    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/367d0cac-af3c-4dce-acc3-ce2e5f85fce1-trusted-ca-bundle\") pod \"insights-operator-5844b856db-hpt25\" (UID: \"367d0cac-af3c-4dce-acc3-ce2e5f85fce1\") " pod="openshift-insights/insights-operator-5844b856db-hpt25"
Sep 10 20:59:39.107290 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.107188    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fd81843e-6283-43e3-a571-d5e47be26e38-serving-cert\") pod \"console-operator-696884fb9c-jkkfm\" (UID: \"fd81843e-6283-43e3-a571-d5e47be26e38\") " pod="openshift-console-operator/console-operator-696884fb9c-jkkfm"
Sep 10 20:59:39.107290 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:39.107195    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 20:59:39.107290 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.107221    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dbc735a8-26d3-4267-afe5-b334f6e03fd8-config\") pod \"service-ca-operator-cf7c776b7-xnf52\" (UID: \"dbc735a8-26d3-4267-afe5-b334f6e03fd8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnf52"
Sep 10 20:59:39.107290 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:39.107196    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 20:59:39.107290 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:39.107297    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93462573-692a-4246-afd9-08c9430c7f53-cluster-monitoring-operator-tls podName:93462573-692a-4246-afd9-08c9430c7f53 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:39.607275754 +0000 UTC m=+39.272300534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/93462573-692a-4246-afd9-08c9430c7f53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-zkbqs" (UID: "93462573-692a-4246-afd9-08c9430c7f53") : secret "cluster-monitoring-operator-tls" not found
Sep 10 20:59:39.107849 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.107264    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/367d0cac-af3c-4dce-acc3-ce2e5f85fce1-snapshots\") pod \"insights-operator-5844b856db-hpt25\" (UID: \"367d0cac-af3c-4dce-acc3-ce2e5f85fce1\") " pod="openshift-insights/insights-operator-5844b856db-hpt25"
Sep 10 20:59:39.107849 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:39.107322    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29f5e959-7b2f-42e9-98f2-a3547859bb56-networking-console-plugin-cert podName:29f5e959-7b2f-42e9-98f2-a3547859bb56 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:39.607312962 +0000 UTC m=+39.272337742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/29f5e959-7b2f-42e9-98f2-a3547859bb56-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-pjkzj" (UID: "29f5e959-7b2f-42e9-98f2-a3547859bb56") : secret "networking-console-plugin-cert" not found
Sep 10 20:59:39.107849 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.107348    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/367d0cac-af3c-4dce-acc3-ce2e5f85fce1-service-ca-bundle\") pod \"insights-operator-5844b856db-hpt25\" (UID: \"367d0cac-af3c-4dce-acc3-ce2e5f85fce1\") " pod="openshift-insights/insights-operator-5844b856db-hpt25"
Sep 10 20:59:39.107849 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.107380    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/367d0cac-af3c-4dce-acc3-ce2e5f85fce1-serving-cert\") pod \"insights-operator-5844b856db-hpt25\" (UID: \"367d0cac-af3c-4dce-acc3-ce2e5f85fce1\") " pod="openshift-insights/insights-operator-5844b856db-hpt25"
Sep 10 20:59:39.107849 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.107557    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/93462573-692a-4246-afd9-08c9430c7f53-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-zkbqs\" (UID: \"93462573-692a-4246-afd9-08c9430c7f53\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zkbqs"
Sep 10 20:59:39.107849 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.107682    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/29f5e959-7b2f-42e9-98f2-a3547859bb56-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-pjkzj\" (UID: \"29f5e959-7b2f-42e9-98f2-a3547859bb56\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pjkzj"
Sep 10 20:59:39.122349 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.122321    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w89mc\" (UniqueName: \"kubernetes.io/projected/93462573-692a-4246-afd9-08c9430c7f53-kube-api-access-w89mc\") pod \"cluster-monitoring-operator-6cd954b897-zkbqs\" (UID: \"93462573-692a-4246-afd9-08c9430c7f53\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zkbqs"
Sep 10 20:59:39.207879 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.207786    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/367d0cac-af3c-4dce-acc3-ce2e5f85fce1-tmp\") pod \"insights-operator-5844b856db-hpt25\" (UID: \"367d0cac-af3c-4dce-acc3-ce2e5f85fce1\") " pod="openshift-insights/insights-operator-5844b856db-hpt25"
Sep 10 20:59:39.207879 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.207849    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zr4b\" (UniqueName: \"kubernetes.io/projected/367d0cac-af3c-4dce-acc3-ce2e5f85fce1-kube-api-access-7zr4b\") pod \"insights-operator-5844b856db-hpt25\" (UID: \"367d0cac-af3c-4dce-acc3-ce2e5f85fce1\") " pod="openshift-insights/insights-operator-5844b856db-hpt25"
Sep 10 20:59:39.208074 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.207886    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd81843e-6283-43e3-a571-d5e47be26e38-trusted-ca\") pod \"console-operator-696884fb9c-jkkfm\" (UID: \"fd81843e-6283-43e3-a571-d5e47be26e38\") " pod="openshift-console-operator/console-operator-696884fb9c-jkkfm"
Sep 10 20:59:39.208074 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.207928    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbqfw\" (UniqueName: \"kubernetes.io/projected/fd81843e-6283-43e3-a571-d5e47be26e38-kube-api-access-gbqfw\") pod \"console-operator-696884fb9c-jkkfm\" (UID: \"fd81843e-6283-43e3-a571-d5e47be26e38\") " pod="openshift-console-operator/console-operator-696884fb9c-jkkfm"
Sep 10 20:59:39.208074 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.207981    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dbc735a8-26d3-4267-afe5-b334f6e03fd8-serving-cert\") pod \"service-ca-operator-cf7c776b7-xnf52\" (UID: \"dbc735a8-26d3-4267-afe5-b334f6e03fd8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnf52"
Sep 10 20:59:39.208074 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.208007    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffzz8\" (UniqueName: \"kubernetes.io/projected/dbc735a8-26d3-4267-afe5-b334f6e03fd8-kube-api-access-ffzz8\") pod \"service-ca-operator-cf7c776b7-xnf52\" (UID: \"dbc735a8-26d3-4267-afe5-b334f6e03fd8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnf52"
Sep 10 20:59:39.208074 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.208032    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/367d0cac-af3c-4dce-acc3-ce2e5f85fce1-trusted-ca-bundle\") pod \"insights-operator-5844b856db-hpt25\" (UID: \"367d0cac-af3c-4dce-acc3-ce2e5f85fce1\") " pod="openshift-insights/insights-operator-5844b856db-hpt25"
Sep 10 20:59:39.208324 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.208087    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7ng5\" (UniqueName: \"kubernetes.io/projected/b3f4bee0-04e0-4178-9001-49792931508b-kube-api-access-n7ng5\") pod \"cluster-samples-operator-86499c89c4-qsxcs\" (UID: \"b3f4bee0-04e0-4178-9001-49792931508b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qsxcs"
Sep 10 20:59:39.208324 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.208137    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3f4bee0-04e0-4178-9001-49792931508b-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qsxcs\" (UID: \"b3f4bee0-04e0-4178-9001-49792931508b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qsxcs"
Sep 10 20:59:39.208324 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.208137    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/367d0cac-af3c-4dce-acc3-ce2e5f85fce1-tmp\") pod \"insights-operator-5844b856db-hpt25\" (UID: \"367d0cac-af3c-4dce-acc3-ce2e5f85fce1\") " pod="openshift-insights/insights-operator-5844b856db-hpt25"
Sep 10 20:59:39.208324 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.208250    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fd81843e-6283-43e3-a571-d5e47be26e38-serving-cert\") pod \"console-operator-696884fb9c-jkkfm\" (UID: \"fd81843e-6283-43e3-a571-d5e47be26e38\") " pod="openshift-console-operator/console-operator-696884fb9c-jkkfm"
Sep 10 20:59:39.208324 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.208303    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dbc735a8-26d3-4267-afe5-b334f6e03fd8-config\") pod \"service-ca-operator-cf7c776b7-xnf52\" (UID: \"dbc735a8-26d3-4267-afe5-b334f6e03fd8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnf52"
Sep 10 20:59:39.208591 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.208336    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/367d0cac-af3c-4dce-acc3-ce2e5f85fce1-snapshots\") pod \"insights-operator-5844b856db-hpt25\" (UID: \"367d0cac-af3c-4dce-acc3-ce2e5f85fce1\") " pod="openshift-insights/insights-operator-5844b856db-hpt25"
Sep 10 20:59:39.208591 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.208360    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/367d0cac-af3c-4dce-acc3-ce2e5f85fce1-service-ca-bundle\") pod \"insights-operator-5844b856db-hpt25\" (UID: \"367d0cac-af3c-4dce-acc3-ce2e5f85fce1\") " pod="openshift-insights/insights-operator-5844b856db-hpt25"
Sep 10 20:59:39.208591 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.208415    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/367d0cac-af3c-4dce-acc3-ce2e5f85fce1-serving-cert\") pod \"insights-operator-5844b856db-hpt25\" (UID: \"367d0cac-af3c-4dce-acc3-ce2e5f85fce1\") " pod="openshift-insights/insights-operator-5844b856db-hpt25"
Sep 10 20:59:39.208591 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.208531    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fd81843e-6283-43e3-a571-d5e47be26e38-config\") pod \"console-operator-696884fb9c-jkkfm\" (UID: \"fd81843e-6283-43e3-a571-d5e47be26e38\") " pod="openshift-console-operator/console-operator-696884fb9c-jkkfm"
Sep 10 20:59:39.209062 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.208976    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/367d0cac-af3c-4dce-acc3-ce2e5f85fce1-snapshots\") pod \"insights-operator-5844b856db-hpt25\" (UID: \"367d0cac-af3c-4dce-acc3-ce2e5f85fce1\") " pod="openshift-insights/insights-operator-5844b856db-hpt25"
Sep 10 20:59:39.209062 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.209011    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dbc735a8-26d3-4267-afe5-b334f6e03fd8-config\") pod \"service-ca-operator-cf7c776b7-xnf52\" (UID: \"dbc735a8-26d3-4267-afe5-b334f6e03fd8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnf52"
Sep 10 20:59:39.209257 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.209015    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd81843e-6283-43e3-a571-d5e47be26e38-trusted-ca\") pod \"console-operator-696884fb9c-jkkfm\" (UID: \"fd81843e-6283-43e3-a571-d5e47be26e38\") " pod="openshift-console-operator/console-operator-696884fb9c-jkkfm"
Sep 10 20:59:39.209257 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.209212    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fd81843e-6283-43e3-a571-d5e47be26e38-config\") pod \"console-operator-696884fb9c-jkkfm\" (UID: \"fd81843e-6283-43e3-a571-d5e47be26e38\") " pod="openshift-console-operator/console-operator-696884fb9c-jkkfm"
Sep 10 20:59:39.209371 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.209287    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/367d0cac-af3c-4dce-acc3-ce2e5f85fce1-service-ca-bundle\") pod \"insights-operator-5844b856db-hpt25\" (UID: \"367d0cac-af3c-4dce-acc3-ce2e5f85fce1\") " pod="openshift-insights/insights-operator-5844b856db-hpt25"
Sep 10 20:59:39.209718 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.209692    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/367d0cac-af3c-4dce-acc3-ce2e5f85fce1-trusted-ca-bundle\") pod \"insights-operator-5844b856db-hpt25\" (UID: \"367d0cac-af3c-4dce-acc3-ce2e5f85fce1\") " pod="openshift-insights/insights-operator-5844b856db-hpt25"
Sep 10 20:59:39.210668 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.210638    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dbc735a8-26d3-4267-afe5-b334f6e03fd8-serving-cert\") pod \"service-ca-operator-cf7c776b7-xnf52\" (UID: \"dbc735a8-26d3-4267-afe5-b334f6e03fd8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnf52"
Sep 10 20:59:39.210814 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.210800    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fd81843e-6283-43e3-a571-d5e47be26e38-serving-cert\") pod \"console-operator-696884fb9c-jkkfm\" (UID: \"fd81843e-6283-43e3-a571-d5e47be26e38\") " pod="openshift-console-operator/console-operator-696884fb9c-jkkfm"
Sep 10 20:59:39.211177 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.211157    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/367d0cac-af3c-4dce-acc3-ce2e5f85fce1-serving-cert\") pod \"insights-operator-5844b856db-hpt25\" (UID: \"367d0cac-af3c-4dce-acc3-ce2e5f85fce1\") " pod="openshift-insights/insights-operator-5844b856db-hpt25"
Sep 10 20:59:39.215733 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.215687    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffzz8\" (UniqueName: \"kubernetes.io/projected/dbc735a8-26d3-4267-afe5-b334f6e03fd8-kube-api-access-ffzz8\") pod \"service-ca-operator-cf7c776b7-xnf52\" (UID: \"dbc735a8-26d3-4267-afe5-b334f6e03fd8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnf52"
Sep 10 20:59:39.215987 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.215972    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zr4b\" (UniqueName: \"kubernetes.io/projected/367d0cac-af3c-4dce-acc3-ce2e5f85fce1-kube-api-access-7zr4b\") pod \"insights-operator-5844b856db-hpt25\" (UID: \"367d0cac-af3c-4dce-acc3-ce2e5f85fce1\") " pod="openshift-insights/insights-operator-5844b856db-hpt25"
Sep 10 20:59:39.216236 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.216215    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbqfw\" (UniqueName: \"kubernetes.io/projected/fd81843e-6283-43e3-a571-d5e47be26e38-kube-api-access-gbqfw\") pod \"console-operator-696884fb9c-jkkfm\" (UID: \"fd81843e-6283-43e3-a571-d5e47be26e38\") " pod="openshift-console-operator/console-operator-696884fb9c-jkkfm"
Sep 10 20:59:39.281151 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.281126    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnf52"
Sep 10 20:59:39.295325 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.295305    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-hpt25"
Sep 10 20:59:39.309693 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.309673    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7ng5\" (UniqueName: \"kubernetes.io/projected/b3f4bee0-04e0-4178-9001-49792931508b-kube-api-access-n7ng5\") pod \"cluster-samples-operator-86499c89c4-qsxcs\" (UID: \"b3f4bee0-04e0-4178-9001-49792931508b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qsxcs"
Sep 10 20:59:39.309783 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.309702    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3f4bee0-04e0-4178-9001-49792931508b-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qsxcs\" (UID: \"b3f4bee0-04e0-4178-9001-49792931508b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qsxcs"
Sep 10 20:59:39.309822 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:39.309797    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 20:59:39.309855 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:39.309847    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3f4bee0-04e0-4178-9001-49792931508b-samples-operator-tls podName:b3f4bee0-04e0-4178-9001-49792931508b nodeName:}" failed. No retries permitted until 2026-09-10 20:59:39.80983305 +0000 UTC m=+39.474857829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b3f4bee0-04e0-4178-9001-49792931508b-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-qsxcs" (UID: "b3f4bee0-04e0-4178-9001-49792931508b") : secret "samples-operator-tls" not found
Sep 10 20:59:39.317485 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.317464    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-jkkfm"
Sep 10 20:59:39.318584 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.318566    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7ng5\" (UniqueName: \"kubernetes.io/projected/b3f4bee0-04e0-4178-9001-49792931508b-kube-api-access-n7ng5\") pod \"cluster-samples-operator-86499c89c4-qsxcs\" (UID: \"b3f4bee0-04e0-4178-9001-49792931508b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qsxcs"
Sep 10 20:59:39.512264 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.512172    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3dfe9291-e76b-4d25-91bf-4c3d78d62841-metrics-tls\") pod \"dns-default-dzgxv\" (UID: \"3dfe9291-e76b-4d25-91bf-4c3d78d62841\") " pod="openshift-dns/dns-default-dzgxv"
Sep 10 20:59:39.512390 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:39.512357    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:59:39.512458 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:39.512449    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dfe9291-e76b-4d25-91bf-4c3d78d62841-metrics-tls podName:3dfe9291-e76b-4d25-91bf-4c3d78d62841 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:43.512414408 +0000 UTC m=+43.177439191 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3dfe9291-e76b-4d25-91bf-4c3d78d62841-metrics-tls") pod "dns-default-dzgxv" (UID: "3dfe9291-e76b-4d25-91bf-4c3d78d62841") : secret "dns-default-metrics-tls" not found
Sep 10 20:59:39.565150 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.565080    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnf52"]
Sep 10 20:59:39.571440 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:39.571401    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc735a8_26d3_4267_afe5_b334f6e03fd8.slice/crio-237188cf0cbdc20828e81909034b6abb8cbbb50420c18aedff4db6744dc51e3a WatchSource:0}: Error finding container 237188cf0cbdc20828e81909034b6abb8cbbb50420c18aedff4db6744dc51e3a: Status 404 returned error can't find the container with id 237188cf0cbdc20828e81909034b6abb8cbbb50420c18aedff4db6744dc51e3a
Sep 10 20:59:39.613345 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.613317    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29f5e959-7b2f-42e9-98f2-a3547859bb56-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-pjkzj\" (UID: \"29f5e959-7b2f-42e9-98f2-a3547859bb56\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pjkzj"
Sep 10 20:59:39.613483 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.613349    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/93462573-692a-4246-afd9-08c9430c7f53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zkbqs\" (UID: \"93462573-692a-4246-afd9-08c9430c7f53\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zkbqs"
Sep 10 20:59:39.613534 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:39.613524    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 20:59:39.613601 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:39.613589    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93462573-692a-4246-afd9-08c9430c7f53-cluster-monitoring-operator-tls podName:93462573-692a-4246-afd9-08c9430c7f53 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:40.613573845 +0000 UTC m=+40.278598637 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/93462573-692a-4246-afd9-08c9430c7f53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-zkbqs" (UID: "93462573-692a-4246-afd9-08c9430c7f53") : secret "cluster-monitoring-operator-tls" not found
Sep 10 20:59:39.613669 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:39.613526    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 20:59:39.613755 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:39.613741    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29f5e959-7b2f-42e9-98f2-a3547859bb56-networking-console-plugin-cert podName:29f5e959-7b2f-42e9-98f2-a3547859bb56 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:40.613693226 +0000 UTC m=+40.278718019 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/29f5e959-7b2f-42e9-98f2-a3547859bb56-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-pjkzj" (UID: "29f5e959-7b2f-42e9-98f2-a3547859bb56") : secret "networking-console-plugin-cert" not found
Sep 10 20:59:39.615298 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.615271    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-hpt25"]
Sep 10 20:59:39.619162 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:39.619144    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod367d0cac_af3c_4dce_acc3_ce2e5f85fce1.slice/crio-73f3b33a4d9add0e6af048af5e2a8f792f217cddf521519e5c072cb2b79baacf WatchSource:0}: Error finding container 73f3b33a4d9add0e6af048af5e2a8f792f217cddf521519e5c072cb2b79baacf: Status 404 returned error can't find the container with id 73f3b33a4d9add0e6af048af5e2a8f792f217cddf521519e5c072cb2b79baacf
Sep 10 20:59:39.630784 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.630765    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-jkkfm"]
Sep 10 20:59:39.641807 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:39.641784    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd81843e_6283_43e3_a571_d5e47be26e38.slice/crio-af6b7e843239d73750dcbdd4329049421603c172769c9eaad15343af5b28c4d1 WatchSource:0}: Error finding container af6b7e843239d73750dcbdd4329049421603c172769c9eaad15343af5b28c4d1: Status 404 returned error can't find the container with id af6b7e843239d73750dcbdd4329049421603c172769c9eaad15343af5b28c4d1
Sep 10 20:59:39.814527 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:39.814438    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3f4bee0-04e0-4178-9001-49792931508b-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qsxcs\" (UID: \"b3f4bee0-04e0-4178-9001-49792931508b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qsxcs"
Sep 10 20:59:39.814684 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:39.814596    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 20:59:39.814684 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:39.814665    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3f4bee0-04e0-4178-9001-49792931508b-samples-operator-tls podName:b3f4bee0-04e0-4178-9001-49792931508b nodeName:}" failed. No retries permitted until 2026-09-10 20:59:40.814649473 +0000 UTC m=+40.479674251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b3f4bee0-04e0-4178-9001-49792931508b-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-qsxcs" (UID: "b3f4bee0-04e0-4178-9001-49792931508b") : secret "samples-operator-tls" not found
Sep 10 20:59:40.039557 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:40.039520    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-jkkfm" event={"ID":"fd81843e-6283-43e3-a571-d5e47be26e38","Type":"ContainerStarted","Data":"af6b7e843239d73750dcbdd4329049421603c172769c9eaad15343af5b28c4d1"}
Sep 10 20:59:40.040809 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:40.040783    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-hpt25" event={"ID":"367d0cac-af3c-4dce-acc3-ce2e5f85fce1","Type":"ContainerStarted","Data":"73f3b33a4d9add0e6af048af5e2a8f792f217cddf521519e5c072cb2b79baacf"}
Sep 10 20:59:40.042070 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:40.042048    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnf52" event={"ID":"dbc735a8-26d3-4267-afe5-b334f6e03fd8","Type":"ContainerStarted","Data":"237188cf0cbdc20828e81909034b6abb8cbbb50420c18aedff4db6744dc51e3a"}
Sep 10 20:59:40.043857 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:40.043832    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ql42f" event={"ID":"e3464d4e-8d40-4389-9be1-49529d8636ae","Type":"ContainerStarted","Data":"7d349f0fa84e0b462a5f25db5bccab1d4cace282fe3c50f7010167f5e0d5a0b4"}
Sep 10 20:59:40.059804 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:40.059761    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ql42f" podStartSLOduration=1.917570472 podStartE2EDuration="5.059746473s" totalImagesPullingTime="3.142176001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:40.059170212 +0000 UTC m=+39.724195014" watchObservedRunningTime="2026-09-10 20:59:40.059746473 +0000 UTC m=+39.724771302"
Sep 10 20:59:40.621691 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:40.621651    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29f5e959-7b2f-42e9-98f2-a3547859bb56-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-pjkzj\" (UID: \"29f5e959-7b2f-42e9-98f2-a3547859bb56\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pjkzj"
Sep 10 20:59:40.621881 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:40.621707    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/93462573-692a-4246-afd9-08c9430c7f53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zkbqs\" (UID: \"93462573-692a-4246-afd9-08c9430c7f53\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zkbqs"
Sep 10 20:59:40.621881 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:40.621823    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 20:59:40.622044 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:40.621892    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93462573-692a-4246-afd9-08c9430c7f53-cluster-monitoring-operator-tls podName:93462573-692a-4246-afd9-08c9430c7f53 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:42.62187207 +0000 UTC m=+42.286896851 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/93462573-692a-4246-afd9-08c9430c7f53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-zkbqs" (UID: "93462573-692a-4246-afd9-08c9430c7f53") : secret "cluster-monitoring-operator-tls" not found
Sep 10 20:59:40.622044 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:40.621822    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 20:59:40.622044 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:40.621968    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29f5e959-7b2f-42e9-98f2-a3547859bb56-networking-console-plugin-cert podName:29f5e959-7b2f-42e9-98f2-a3547859bb56 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:42.621949294 +0000 UTC m=+42.286974076 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/29f5e959-7b2f-42e9-98f2-a3547859bb56-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-pjkzj" (UID: "29f5e959-7b2f-42e9-98f2-a3547859bb56") : secret "networking-console-plugin-cert" not found
Sep 10 20:59:40.822765 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:40.822722    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3f4bee0-04e0-4178-9001-49792931508b-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qsxcs\" (UID: \"b3f4bee0-04e0-4178-9001-49792931508b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qsxcs"
Sep 10 20:59:40.822930 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:40.822875    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 20:59:40.823009 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:40.822955    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3f4bee0-04e0-4178-9001-49792931508b-samples-operator-tls podName:b3f4bee0-04e0-4178-9001-49792931508b nodeName:}" failed. No retries permitted until 2026-09-10 20:59:42.822934527 +0000 UTC m=+42.487959312 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b3f4bee0-04e0-4178-9001-49792931508b-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-qsxcs" (UID: "b3f4bee0-04e0-4178-9001-49792931508b") : secret "samples-operator-tls" not found
Sep 10 20:59:42.636940 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:42.636885    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29f5e959-7b2f-42e9-98f2-a3547859bb56-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-pjkzj\" (UID: \"29f5e959-7b2f-42e9-98f2-a3547859bb56\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pjkzj"
Sep 10 20:59:42.637413 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:42.636960    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/93462573-692a-4246-afd9-08c9430c7f53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zkbqs\" (UID: \"93462573-692a-4246-afd9-08c9430c7f53\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zkbqs"
Sep 10 20:59:42.637413 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:42.637000    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 20:59:42.637413 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:42.637090    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29f5e959-7b2f-42e9-98f2-a3547859bb56-networking-console-plugin-cert podName:29f5e959-7b2f-42e9-98f2-a3547859bb56 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:46.637066609 +0000 UTC m=+46.302091404 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/29f5e959-7b2f-42e9-98f2-a3547859bb56-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-pjkzj" (UID: "29f5e959-7b2f-42e9-98f2-a3547859bb56") : secret "networking-console-plugin-cert" not found
Sep 10 20:59:42.637413 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:42.637116    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 20:59:42.637413 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:42.637186    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93462573-692a-4246-afd9-08c9430c7f53-cluster-monitoring-operator-tls podName:93462573-692a-4246-afd9-08c9430c7f53 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:46.637168376 +0000 UTC m=+46.302193168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/93462573-692a-4246-afd9-08c9430c7f53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-zkbqs" (UID: "93462573-692a-4246-afd9-08c9430c7f53") : secret "cluster-monitoring-operator-tls" not found
Sep 10 20:59:42.838999 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:42.838964    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3f4bee0-04e0-4178-9001-49792931508b-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qsxcs\" (UID: \"b3f4bee0-04e0-4178-9001-49792931508b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qsxcs"
Sep 10 20:59:42.839177 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:42.839102    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 20:59:42.839245 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:42.839185    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3f4bee0-04e0-4178-9001-49792931508b-samples-operator-tls podName:b3f4bee0-04e0-4178-9001-49792931508b nodeName:}" failed. No retries permitted until 2026-09-10 20:59:46.839162786 +0000 UTC m=+46.504187581 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b3f4bee0-04e0-4178-9001-49792931508b-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-qsxcs" (UID: "b3f4bee0-04e0-4178-9001-49792931508b") : secret "samples-operator-tls" not found
Sep 10 20:59:43.544506 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:43.544478    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3dfe9291-e76b-4d25-91bf-4c3d78d62841-metrics-tls\") pod \"dns-default-dzgxv\" (UID: \"3dfe9291-e76b-4d25-91bf-4c3d78d62841\") " pod="openshift-dns/dns-default-dzgxv"
Sep 10 20:59:43.544687 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:43.544657    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:59:43.544760 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:43.544736    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dfe9291-e76b-4d25-91bf-4c3d78d62841-metrics-tls podName:3dfe9291-e76b-4d25-91bf-4c3d78d62841 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:51.544715827 +0000 UTC m=+51.209740605 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3dfe9291-e76b-4d25-91bf-4c3d78d62841-metrics-tls") pod "dns-default-dzgxv" (UID: "3dfe9291-e76b-4d25-91bf-4c3d78d62841") : secret "dns-default-metrics-tls" not found
Sep 10 20:59:43.656531 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:43.656505    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-ptzj9"]
Sep 10 20:59:43.674284 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:43.674266    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ptzj9"
Sep 10 20:59:43.677549 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:43.677525    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-ptzj9"]
Sep 10 20:59:43.679149 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:43.679128    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 10 20:59:43.679266 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:43.679153    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6q7jp\""
Sep 10 20:59:43.679266 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:43.679157    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 10 20:59:43.846993 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:43.846954    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxsb8\" (UniqueName: \"kubernetes.io/projected/68115039-7bd0-49f4-a4d7-eca804f130f1-kube-api-access-zxsb8\") pod \"migrator-8b4d697c5-ptzj9\" (UID: \"68115039-7bd0-49f4-a4d7-eca804f130f1\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ptzj9"
Sep 10 20:59:43.947543 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:43.947510    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxsb8\" (UniqueName: \"kubernetes.io/projected/68115039-7bd0-49f4-a4d7-eca804f130f1-kube-api-access-zxsb8\") pod \"migrator-8b4d697c5-ptzj9\" (UID: \"68115039-7bd0-49f4-a4d7-eca804f130f1\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ptzj9"
Sep 10 20:59:43.957088 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:43.957056    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxsb8\" (UniqueName: \"kubernetes.io/projected/68115039-7bd0-49f4-a4d7-eca804f130f1-kube-api-access-zxsb8\") pod \"migrator-8b4d697c5-ptzj9\" (UID: \"68115039-7bd0-49f4-a4d7-eca804f130f1\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ptzj9"
Sep 10 20:59:44.012202 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:44.012124    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ptzj9"
Sep 10 20:59:44.056071 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:44.056047    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-jkkfm_fd81843e-6283-43e3-a571-d5e47be26e38/console-operator/0.log"
Sep 10 20:59:44.056197 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:44.056093    2584 generic.go:356] "Generic (PLEG): container finished" podID="fd81843e-6283-43e3-a571-d5e47be26e38" containerID="efaa2ce162b0711e3541e92a73f29b39160e3f8fc540adcfb0930ca49e04b2a7" exitCode=255
Sep 10 20:59:44.056257 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:44.056208    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-jkkfm" event={"ID":"fd81843e-6283-43e3-a571-d5e47be26e38","Type":"ContainerDied","Data":"efaa2ce162b0711e3541e92a73f29b39160e3f8fc540adcfb0930ca49e04b2a7"}
Sep 10 20:59:44.056441 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:44.056405    2584 scope.go:117] "RemoveContainer" containerID="efaa2ce162b0711e3541e92a73f29b39160e3f8fc540adcfb0930ca49e04b2a7"
Sep 10 20:59:44.058244 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:44.058211    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-hpt25" event={"ID":"367d0cac-af3c-4dce-acc3-ce2e5f85fce1","Type":"ContainerStarted","Data":"a27c1cdc77ac5504b5ac8b8d9324767c7eb0616d0d4bfaf65e35eae519c09297"}
Sep 10 20:59:44.059930 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:44.059907    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnf52" event={"ID":"dbc735a8-26d3-4267-afe5-b334f6e03fd8","Type":"ContainerStarted","Data":"dbd28f54759c558cb994afd9e07c68ebb05f607e56bbdfb9b1f2985f98803015"}
Sep 10 20:59:44.093447 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:44.092991    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnf52" podStartSLOduration=2.254100687 podStartE2EDuration="6.092964131s" totalImagesPullingTime="3.838863444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:44.092366656 +0000 UTC m=+43.757391455" watchObservedRunningTime="2026-09-10 20:59:44.092964131 +0000 UTC m=+43.757988931"
Sep 10 20:59:44.321752 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:44.321634    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-hpt25" podStartSLOduration=2.530451036 podStartE2EDuration="6.321617624s" totalImagesPullingTime="3.791166588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:44.11667134 +0000 UTC m=+43.781696140" watchObservedRunningTime="2026-09-10 20:59:44.321617624 +0000 UTC m=+43.986642424"
Sep 10 20:59:44.322797 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:44.322773    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-ptzj9"]
Sep 10 20:59:44.326719 ip-10-0-135-205 kubenswrapper[2584]: W0910 20:59:44.326694    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68115039_7bd0_49f4_a4d7_eca804f130f1.slice/crio-d2145733c7a283813a7ea3de9b0c4cefcec30de1f2dbf59c016d70c54492d4a6 WatchSource:0}: Error finding container d2145733c7a283813a7ea3de9b0c4cefcec30de1f2dbf59c016d70c54492d4a6: Status 404 returned error can't find the container with id d2145733c7a283813a7ea3de9b0c4cefcec30de1f2dbf59c016d70c54492d4a6
Sep 10 20:59:45.065484 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.065453    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-jkkfm_fd81843e-6283-43e3-a571-d5e47be26e38/console-operator/1.log"
Sep 10 20:59:45.065962 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.065943    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-jkkfm_fd81843e-6283-43e3-a571-d5e47be26e38/console-operator/0.log"
Sep 10 20:59:45.066059 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.065987    2584 generic.go:356] "Generic (PLEG): container finished" podID="fd81843e-6283-43e3-a571-d5e47be26e38" containerID="05e40778cba564758f9fec2483662ac6df8d3cd93d111eb0f570b560fa5c8820" exitCode=255
Sep 10 20:59:45.066130 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.066091    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-jkkfm" event={"ID":"fd81843e-6283-43e3-a571-d5e47be26e38","Type":"ContainerDied","Data":"05e40778cba564758f9fec2483662ac6df8d3cd93d111eb0f570b560fa5c8820"}
Sep 10 20:59:45.066191 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.066147    2584 scope.go:117] "RemoveContainer" containerID="efaa2ce162b0711e3541e92a73f29b39160e3f8fc540adcfb0930ca49e04b2a7"
Sep 10 20:59:45.066441 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.066407    2584 scope.go:117] "RemoveContainer" containerID="05e40778cba564758f9fec2483662ac6df8d3cd93d111eb0f570b560fa5c8820"
Sep 10 20:59:45.066646 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:45.066621    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-jkkfm_openshift-console-operator(fd81843e-6283-43e3-a571-d5e47be26e38)\"" pod="openshift-console-operator/console-operator-696884fb9c-jkkfm" podUID="fd81843e-6283-43e3-a571-d5e47be26e38"
Sep 10 20:59:45.067770 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.067745    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ptzj9" event={"ID":"68115039-7bd0-49f4-a4d7-eca804f130f1","Type":"ContainerStarted","Data":"d2145733c7a283813a7ea3de9b0c4cefcec30de1f2dbf59c016d70c54492d4a6"}
Sep 10 20:59:45.716396 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.716358    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f5r9p"]
Sep 10 20:59:45.737119 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.737090    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f5r9p"]
Sep 10 20:59:45.737276 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.737236    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 20:59:45.740135 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.740112    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 20:59:45.740278 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.740175    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 20:59:45.740381 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.740279    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-89wjp\""
Sep 10 20:59:45.765960 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.765934    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-749m9\" (UniqueName: \"kubernetes.io/projected/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-kube-api-access-749m9\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 20:59:45.766066 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.765982    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 20:59:45.766066 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.766020    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-crio-socket\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 20:59:45.766155 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.766065    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 20:59:45.766155 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.766148    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-data-volume\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 20:59:45.866497 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.866465    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-crio-socket\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 20:59:45.866647 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.866527    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 20:59:45.866647 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.866608    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-data-volume\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 20:59:45.866752 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.866661    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-749m9\" (UniqueName: \"kubernetes.io/projected/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-kube-api-access-749m9\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 20:59:45.866752 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.866683    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-crio-socket\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 20:59:45.866752 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:45.866698    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:59:45.866752 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.866714    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 20:59:45.866941 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:45.866767    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-insights-runtime-extractor-tls podName:6b961536-58d3-41eb-aa81-d1edb5dd1d0e nodeName:}" failed. No retries permitted until 2026-09-10 20:59:46.36675162 +0000 UTC m=+46.031776399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f5r9p" (UID: "6b961536-58d3-41eb-aa81-d1edb5dd1d0e") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:59:45.866941 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:45.866787    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:59:45.866941 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:45.866839    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-kube-rbac-proxy-cm podName:6b961536-58d3-41eb-aa81-d1edb5dd1d0e nodeName:}" failed. No retries permitted until 2026-09-10 20:59:46.366822658 +0000 UTC m=+46.031847456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f5r9p" (UID: "6b961536-58d3-41eb-aa81-d1edb5dd1d0e") : configmap "kube-rbac-proxy" not found
Sep 10 20:59:45.867075 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.866999    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-data-volume\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 20:59:45.876634 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:45.876608    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-749m9\" (UniqueName: \"kubernetes.io/projected/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-kube-api-access-749m9\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 20:59:46.072332 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:46.072266    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-jkkfm_fd81843e-6283-43e3-a571-d5e47be26e38/console-operator/1.log"
Sep 10 20:59:46.072706 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:46.072640    2584 scope.go:117] "RemoveContainer" containerID="05e40778cba564758f9fec2483662ac6df8d3cd93d111eb0f570b560fa5c8820"
Sep 10 20:59:46.072799 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:46.072780    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-jkkfm_openshift-console-operator(fd81843e-6283-43e3-a571-d5e47be26e38)\"" pod="openshift-console-operator/console-operator-696884fb9c-jkkfm" podUID="fd81843e-6283-43e3-a571-d5e47be26e38"
Sep 10 20:59:46.371471 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:46.371440    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 20:59:46.371621 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:46.371503    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 20:59:46.371621 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:46.371576    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:59:46.371696 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:46.371638    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:59:46.371696 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:46.371648    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-kube-rbac-proxy-cm podName:6b961536-58d3-41eb-aa81-d1edb5dd1d0e nodeName:}" failed. No retries permitted until 2026-09-10 20:59:47.37163064 +0000 UTC m=+47.036655419 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f5r9p" (UID: "6b961536-58d3-41eb-aa81-d1edb5dd1d0e") : configmap "kube-rbac-proxy" not found
Sep 10 20:59:46.371696 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:46.371681    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-insights-runtime-extractor-tls podName:6b961536-58d3-41eb-aa81-d1edb5dd1d0e nodeName:}" failed. No retries permitted until 2026-09-10 20:59:47.371668759 +0000 UTC m=+47.036693537 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f5r9p" (UID: "6b961536-58d3-41eb-aa81-d1edb5dd1d0e") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:59:46.673414 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:46.673381    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29f5e959-7b2f-42e9-98f2-a3547859bb56-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-pjkzj\" (UID: \"29f5e959-7b2f-42e9-98f2-a3547859bb56\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pjkzj"
Sep 10 20:59:46.673608 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:46.673559    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 20:59:46.673680 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:46.673625    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 20:59:46.673680 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:46.673641    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29f5e959-7b2f-42e9-98f2-a3547859bb56-networking-console-plugin-cert podName:29f5e959-7b2f-42e9-98f2-a3547859bb56 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:54.673619486 +0000 UTC m=+54.338644285 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/29f5e959-7b2f-42e9-98f2-a3547859bb56-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-pjkzj" (UID: "29f5e959-7b2f-42e9-98f2-a3547859bb56") : secret "networking-console-plugin-cert" not found
Sep 10 20:59:46.673680 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:46.673672    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93462573-692a-4246-afd9-08c9430c7f53-cluster-monitoring-operator-tls podName:93462573-692a-4246-afd9-08c9430c7f53 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:54.673660859 +0000 UTC m=+54.338685637 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/93462573-692a-4246-afd9-08c9430c7f53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-zkbqs" (UID: "93462573-692a-4246-afd9-08c9430c7f53") : secret "cluster-monitoring-operator-tls" not found
Sep 10 20:59:46.673680 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:46.673562    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/93462573-692a-4246-afd9-08c9430c7f53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zkbqs\" (UID: \"93462573-692a-4246-afd9-08c9430c7f53\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zkbqs"
Sep 10 20:59:46.875082 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:46.875003    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3f4bee0-04e0-4178-9001-49792931508b-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qsxcs\" (UID: \"b3f4bee0-04e0-4178-9001-49792931508b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qsxcs"
Sep 10 20:59:46.875219 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:46.875121    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 20:59:46.875219 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:46.875197    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3f4bee0-04e0-4178-9001-49792931508b-samples-operator-tls podName:b3f4bee0-04e0-4178-9001-49792931508b nodeName:}" failed. No retries permitted until 2026-09-10 20:59:54.875177781 +0000 UTC m=+54.540202560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b3f4bee0-04e0-4178-9001-49792931508b-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-qsxcs" (UID: "b3f4bee0-04e0-4178-9001-49792931508b") : secret "samples-operator-tls" not found
Sep 10 20:59:47.080508 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:47.080473    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ptzj9" event={"ID":"68115039-7bd0-49f4-a4d7-eca804f130f1","Type":"ContainerStarted","Data":"f765526e83193ef574b418286e85f4c42f2d741dcd1ac3f901989ecf8ecd0071"}
Sep 10 20:59:47.080848 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:47.080547    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ptzj9" event={"ID":"68115039-7bd0-49f4-a4d7-eca804f130f1","Type":"ContainerStarted","Data":"def57479923915bdaa4eb8bbe30a189d7ae42293acab87005baabbb4863498c5"}
Sep 10 20:59:47.099201 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:47.099158    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ptzj9" podStartSLOduration=2.336682617 podStartE2EDuration="4.099142801s" totalImagesPullingTime="1.762460184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:47.098150234 +0000 UTC m=+46.763175033" watchObservedRunningTime="2026-09-10 20:59:47.099142801 +0000 UTC m=+46.764167592"
Sep 10 20:59:47.379175 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:47.379142    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 20:59:47.379318 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:47.379255    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 20:59:47.379318 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:47.379285    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:59:47.379385 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:47.379344    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:59:47.379385 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:47.379351    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-insights-runtime-extractor-tls podName:6b961536-58d3-41eb-aa81-d1edb5dd1d0e nodeName:}" failed. No retries permitted until 2026-09-10 20:59:49.379335232 +0000 UTC m=+49.044360011 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f5r9p" (UID: "6b961536-58d3-41eb-aa81-d1edb5dd1d0e") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:59:47.379488 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:47.379403    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-kube-rbac-proxy-cm podName:6b961536-58d3-41eb-aa81-d1edb5dd1d0e nodeName:}" failed. No retries permitted until 2026-09-10 20:59:49.379391585 +0000 UTC m=+49.044416363 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f5r9p" (UID: "6b961536-58d3-41eb-aa81-d1edb5dd1d0e") : configmap "kube-rbac-proxy" not found
Sep 10 20:59:48.473873 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:48.473843    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wprl7_6aa58f1a-77a8-4479-a7a8-a68b6a4673cc/dns-node-resolver/0.log"
Sep 10 20:59:49.317918 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:49.317878    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-jkkfm"
Sep 10 20:59:49.318094 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:49.317945    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-jkkfm"
Sep 10 20:59:49.318293 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:49.318281    2584 scope.go:117] "RemoveContainer" containerID="05e40778cba564758f9fec2483662ac6df8d3cd93d111eb0f570b560fa5c8820"
Sep 10 20:59:49.318485 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:49.318461    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-jkkfm_openshift-console-operator(fd81843e-6283-43e3-a571-d5e47be26e38)\"" pod="openshift-console-operator/console-operator-696884fb9c-jkkfm" podUID="fd81843e-6283-43e3-a571-d5e47be26e38"
Sep 10 20:59:49.396871 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:49.396836    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 20:59:49.397002 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:49.396923    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 20:59:49.397002 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:49.396947    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:59:49.397101 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:49.397021    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-kube-rbac-proxy-cm podName:6b961536-58d3-41eb-aa81-d1edb5dd1d0e nodeName:}" failed. No retries permitted until 2026-09-10 20:59:53.397001369 +0000 UTC m=+53.062026165 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f5r9p" (UID: "6b961536-58d3-41eb-aa81-d1edb5dd1d0e") : configmap "kube-rbac-proxy" not found
Sep 10 20:59:49.397160 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:49.397104    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:59:49.397209 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:49.397178    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-insights-runtime-extractor-tls podName:6b961536-58d3-41eb-aa81-d1edb5dd1d0e nodeName:}" failed. No retries permitted until 2026-09-10 20:59:53.397164757 +0000 UTC m=+53.062189548 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f5r9p" (UID: "6b961536-58d3-41eb-aa81-d1edb5dd1d0e") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:59:49.672253 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:49.672223    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xswnp_8032963a-4ac1-4d7b-a4de-e9bac1e2c9a1/node-ca/0.log"
Sep 10 20:59:50.473525 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:50.473491    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ql42f_e3464d4e-8d40-4389-9be1-49529d8636ae/serve-healthcheck-canary/0.log"
Sep 10 20:59:50.673251 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:50.673216    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-ptzj9_68115039-7bd0-49f4-a4d7-eca804f130f1/migrator/0.log"
Sep 10 20:59:50.874893 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:50.874858    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-ptzj9_68115039-7bd0-49f4-a4d7-eca804f130f1/graceful-termination/0.log"
Sep 10 20:59:51.615323 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:51.615272    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3dfe9291-e76b-4d25-91bf-4c3d78d62841-metrics-tls\") pod \"dns-default-dzgxv\" (UID: \"3dfe9291-e76b-4d25-91bf-4c3d78d62841\") " pod="openshift-dns/dns-default-dzgxv"
Sep 10 20:59:51.615561 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:51.615471    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:59:51.615641 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:51.615569    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dfe9291-e76b-4d25-91bf-4c3d78d62841-metrics-tls podName:3dfe9291-e76b-4d25-91bf-4c3d78d62841 nodeName:}" failed. No retries permitted until 2026-09-10 21:00:07.615544177 +0000 UTC m=+67.280568954 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3dfe9291-e76b-4d25-91bf-4c3d78d62841-metrics-tls") pod "dns-default-dzgxv" (UID: "3dfe9291-e76b-4d25-91bf-4c3d78d62841") : secret "dns-default-metrics-tls" not found
Sep 10 20:59:53.431592 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:53.431557    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 20:59:53.431969 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:53.431615    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 20:59:53.431969 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:53.431697    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:59:53.431969 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:53.431740    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:59:53.431969 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:53.431766    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-kube-rbac-proxy-cm podName:6b961536-58d3-41eb-aa81-d1edb5dd1d0e nodeName:}" failed. No retries permitted until 2026-09-10 21:00:01.431748269 +0000 UTC m=+61.096773052 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f5r9p" (UID: "6b961536-58d3-41eb-aa81-d1edb5dd1d0e") : configmap "kube-rbac-proxy" not found
Sep 10 20:59:53.431969 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:53.431788    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-insights-runtime-extractor-tls podName:6b961536-58d3-41eb-aa81-d1edb5dd1d0e nodeName:}" failed. No retries permitted until 2026-09-10 21:00:01.431776077 +0000 UTC m=+61.096800855 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f5r9p" (UID: "6b961536-58d3-41eb-aa81-d1edb5dd1d0e") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:59:54.742747 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:54.742706    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29f5e959-7b2f-42e9-98f2-a3547859bb56-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-pjkzj\" (UID: \"29f5e959-7b2f-42e9-98f2-a3547859bb56\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pjkzj"
Sep 10 20:59:54.743213 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:54.742805    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/93462573-692a-4246-afd9-08c9430c7f53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zkbqs\" (UID: \"93462573-692a-4246-afd9-08c9430c7f53\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zkbqs"
Sep 10 20:59:54.743213 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:54.742811    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 20:59:54.743213 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:54.742874    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29f5e959-7b2f-42e9-98f2-a3547859bb56-networking-console-plugin-cert podName:29f5e959-7b2f-42e9-98f2-a3547859bb56 nodeName:}" failed. No retries permitted until 2026-09-10 21:00:10.742859628 +0000 UTC m=+70.407884411 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/29f5e959-7b2f-42e9-98f2-a3547859bb56-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-pjkzj" (UID: "29f5e959-7b2f-42e9-98f2-a3547859bb56") : secret "networking-console-plugin-cert" not found
Sep 10 20:59:54.743213 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:54.742907    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 20:59:54.743213 ip-10-0-135-205 kubenswrapper[2584]: E0910 20:59:54.742946    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93462573-692a-4246-afd9-08c9430c7f53-cluster-monitoring-operator-tls podName:93462573-692a-4246-afd9-08c9430c7f53 nodeName:}" failed. No retries permitted until 2026-09-10 21:00:10.742935578 +0000 UTC m=+70.407960356 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/93462573-692a-4246-afd9-08c9430c7f53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-zkbqs" (UID: "93462573-692a-4246-afd9-08c9430c7f53") : secret "cluster-monitoring-operator-tls" not found
Sep 10 20:59:54.944277 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:54.944236    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3f4bee0-04e0-4178-9001-49792931508b-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qsxcs\" (UID: \"b3f4bee0-04e0-4178-9001-49792931508b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qsxcs"
Sep 10 20:59:54.946567 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:54.946547    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3f4bee0-04e0-4178-9001-49792931508b-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qsxcs\" (UID: \"b3f4bee0-04e0-4178-9001-49792931508b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qsxcs"
Sep 10 20:59:54.975986 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:54.975956    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qsxcs"
Sep 10 20:59:55.243868 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:55.243834    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qsxcs"]
Sep 10 20:59:56.108288 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:56.108251    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qsxcs" event={"ID":"b3f4bee0-04e0-4178-9001-49792931508b","Type":"ContainerStarted","Data":"a1006d673dd2cedd9cc6f519f94376d034f60652864c1c8f5eb68c60154e0b7a"}
Sep 10 20:59:58.115923 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:58.115875    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qsxcs" event={"ID":"b3f4bee0-04e0-4178-9001-49792931508b","Type":"ContainerStarted","Data":"dfe69be6d7ecf3a3603adcc28831746b6ad69a8a11336b3b4ee1310652cb2dac"}
Sep 10 20:59:58.116285 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:58.115944    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qsxcs" event={"ID":"b3f4bee0-04e0-4178-9001-49792931508b","Type":"ContainerStarted","Data":"ee3fb52813441f2cb46d4c0fd8fe36e69284da248f7aad0baa68d33aebbd14f9"}
Sep 10 20:59:58.136970 ip-10-0-135-205 kubenswrapper[2584]: I0910 20:59:58.136922    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qsxcs" podStartSLOduration=17.16613232 podStartE2EDuration="19.136908246s" totalImagesPullingTime="1.970775926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:58.136648558 +0000 UTC m=+57.801673359" watchObservedRunningTime="2026-09-10 20:59:58.136908246 +0000 UTC m=+57.801933028"
Sep 10 21:00:00.010620 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:00.010594    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbwqs"
Sep 10 21:00:01.498085 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:01.498031    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 21:00:01.498509 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:01.498205    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 21:00:01.498509 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:00:01.498327    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 21:00:01.498509 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:00:01.498388    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-insights-runtime-extractor-tls podName:6b961536-58d3-41eb-aa81-d1edb5dd1d0e nodeName:}" failed. No retries permitted until 2026-09-10 21:00:17.498373728 +0000 UTC m=+77.163398505 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f5r9p" (UID: "6b961536-58d3-41eb-aa81-d1edb5dd1d0e") : secret "insights-runtime-extractor-tls" not found
Sep 10 21:00:01.498621 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:01.498577    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 21:00:04.834819 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:04.834780    2584 scope.go:117] "RemoveContainer" containerID="05e40778cba564758f9fec2483662ac6df8d3cd93d111eb0f570b560fa5c8820"
Sep 10 21:00:05.139479 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:05.139448    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-jkkfm_fd81843e-6283-43e3-a571-d5e47be26e38/console-operator/1.log"
Sep 10 21:00:05.139639 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:05.139530    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-jkkfm" event={"ID":"fd81843e-6283-43e3-a571-d5e47be26e38","Type":"ContainerStarted","Data":"9b58316b4cbe9914691c254f85fd5a6e93420d73ef5c7a6de74ad5921195266c"}
Sep 10 21:00:05.139811 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:05.139783    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-jkkfm"
Sep 10 21:00:05.157355 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:05.157309    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-jkkfm" podStartSLOduration=23.383911111 podStartE2EDuration="27.157295559s" totalImagesPullingTime="3.773384448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:05.156503382 +0000 UTC m=+64.821528183" watchObservedRunningTime="2026-09-10 21:00:05.157295559 +0000 UTC m=+64.822320358"
Sep 10 21:00:05.262966 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:05.262937    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-jkkfm"
Sep 10 21:00:06.639279 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:06.639240    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-metrics-certs\") pod \"network-metrics-daemon-jskzm\" (UID: \"b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e\") " pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 21:00:06.639744 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:06.639311    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-original-pull-secret\") pod \"global-pull-secret-syncer-k8v5s\" (UID: \"eb76c9c7-abc6-4d93-b8e8-f68110f905a9\") " pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 21:00:06.641725 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:06.641707    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 21:00:06.641773 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:06.641740    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 21:00:06.651833 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:06.651810    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e-metrics-certs\") pod \"network-metrics-daemon-jskzm\" (UID: \"b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e\") " pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 21:00:06.651940 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:06.651813    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb76c9c7-abc6-4d93-b8e8-f68110f905a9-original-pull-secret\") pod \"global-pull-secret-syncer-k8v5s\" (UID: \"eb76c9c7-abc6-4d93-b8e8-f68110f905a9\") " pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 21:00:06.740719 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:06.740675    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfzk5\" (UniqueName: \"kubernetes.io/projected/c12d6b27-9952-4214-8b9d-2864f13966f9-kube-api-access-lfzk5\") pod \"network-check-target-6hc86\" (UID: \"c12d6b27-9952-4214-8b9d-2864f13966f9\") " pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 21:00:06.743504 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:06.743480    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 21:00:06.745375 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:06.745360    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5khf6\""
Sep 10 21:00:06.748027 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:06.748011    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8v5s"
Sep 10 21:00:06.753584 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:06.753566    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jskzm"
Sep 10 21:00:06.753678 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:06.753610    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 21:00:06.764176 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:06.764153    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfzk5\" (UniqueName: \"kubernetes.io/projected/c12d6b27-9952-4214-8b9d-2864f13966f9-kube-api-access-lfzk5\") pod \"network-check-target-6hc86\" (UID: \"c12d6b27-9952-4214-8b9d-2864f13966f9\") " pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 21:00:06.845997 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:06.845976    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wb7zl\""
Sep 10 21:00:06.853747 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:06.853732    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 21:00:07.030128 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:07.030106    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k8v5s"]
Sep 10 21:00:07.032204 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:00:07.032183    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb76c9c7_abc6_4d93_b8e8_f68110f905a9.slice/crio-475d2f432780432bd874f5b7260ba94d3e02b14bb8fe4635f71557a9ece3b940 WatchSource:0}: Error finding container 475d2f432780432bd874f5b7260ba94d3e02b14bb8fe4635f71557a9ece3b940: Status 404 returned error can't find the container with id 475d2f432780432bd874f5b7260ba94d3e02b14bb8fe4635f71557a9ece3b940
Sep 10 21:00:07.050465 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:07.050440    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jskzm"]
Sep 10 21:00:07.052978 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:00:07.052953    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb981d7c0_ffc5_4b52_a4f5_dc2ee420c32e.slice/crio-d6672ed96a9d79a192d6579f8293f43a81a18f95b6ba52f7b01f63cfae4d5a33 WatchSource:0}: Error finding container d6672ed96a9d79a192d6579f8293f43a81a18f95b6ba52f7b01f63cfae4d5a33: Status 404 returned error can't find the container with id d6672ed96a9d79a192d6579f8293f43a81a18f95b6ba52f7b01f63cfae4d5a33
Sep 10 21:00:07.120814 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:07.120787    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6hc86"]
Sep 10 21:00:07.124095 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:00:07.124063    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12d6b27_9952_4214_8b9d_2864f13966f9.slice/crio-5b70c96964d17c76e086685887adb2c643c3e26cbefbe9aee40d8720e7ccb5cc WatchSource:0}: Error finding container 5b70c96964d17c76e086685887adb2c643c3e26cbefbe9aee40d8720e7ccb5cc: Status 404 returned error can't find the container with id 5b70c96964d17c76e086685887adb2c643c3e26cbefbe9aee40d8720e7ccb5cc
Sep 10 21:00:07.147010 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:07.146950    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jskzm" event={"ID":"b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e","Type":"ContainerStarted","Data":"d6672ed96a9d79a192d6579f8293f43a81a18f95b6ba52f7b01f63cfae4d5a33"}
Sep 10 21:00:07.148196 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:07.148170    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-k8v5s" event={"ID":"eb76c9c7-abc6-4d93-b8e8-f68110f905a9","Type":"ContainerStarted","Data":"475d2f432780432bd874f5b7260ba94d3e02b14bb8fe4635f71557a9ece3b940"}
Sep 10 21:00:07.149477 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:07.149455    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6hc86" event={"ID":"c12d6b27-9952-4214-8b9d-2864f13966f9","Type":"ContainerStarted","Data":"5b70c96964d17c76e086685887adb2c643c3e26cbefbe9aee40d8720e7ccb5cc"}
Sep 10 21:00:07.647828 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:07.647789    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3dfe9291-e76b-4d25-91bf-4c3d78d62841-metrics-tls\") pod \"dns-default-dzgxv\" (UID: \"3dfe9291-e76b-4d25-91bf-4c3d78d62841\") " pod="openshift-dns/dns-default-dzgxv"
Sep 10 21:00:07.650693 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:07.650664    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3dfe9291-e76b-4d25-91bf-4c3d78d62841-metrics-tls\") pod \"dns-default-dzgxv\" (UID: \"3dfe9291-e76b-4d25-91bf-4c3d78d62841\") " pod="openshift-dns/dns-default-dzgxv"
Sep 10 21:00:07.828724 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:07.828687    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7j9n7\""
Sep 10 21:00:07.836802 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:07.836774    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dzgxv"
Sep 10 21:00:08.041843 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.041693    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5ddb5757f4-cbdk5"]
Sep 10 21:00:08.050104 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.050075    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.052901 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.052868    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-k4c27\""
Sep 10 21:00:08.053645 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.053619    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 21:00:08.054018 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.053622    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 21:00:08.057653 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.055958    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 21:00:08.062451 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.060942    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5ddb5757f4-cbdk5"]
Sep 10 21:00:08.064855 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.064829    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 21:00:08.138907 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.138808    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dzgxv"]
Sep 10 21:00:08.142165 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:00:08.142137    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dfe9291_e76b_4d25_91bf_4c3d78d62841.slice/crio-1b1702943e4c960cd3d007971a96bd507848815e1daad6588b7adf5630232a8a WatchSource:0}: Error finding container 1b1702943e4c960cd3d007971a96bd507848815e1daad6588b7adf5630232a8a: Status 404 returned error can't find the container with id 1b1702943e4c960cd3d007971a96bd507848815e1daad6588b7adf5630232a8a
Sep 10 21:00:08.152883 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.152440    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9d0687c-a6af-4745-9cbb-b745d52ef591-bound-sa-token\") pod \"image-registry-5ddb5757f4-cbdk5\" (UID: \"c9d0687c-a6af-4745-9cbb-b745d52ef591\") " pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.152883 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.152550    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9d0687c-a6af-4745-9cbb-b745d52ef591-registry-certificates\") pod \"image-registry-5ddb5757f4-cbdk5\" (UID: \"c9d0687c-a6af-4745-9cbb-b745d52ef591\") " pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.152883 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.152591    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9d0687c-a6af-4745-9cbb-b745d52ef591-image-registry-private-configuration\") pod \"image-registry-5ddb5757f4-cbdk5\" (UID: \"c9d0687c-a6af-4745-9cbb-b745d52ef591\") " pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.152883 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.152658    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9d0687c-a6af-4745-9cbb-b745d52ef591-registry-tls\") pod \"image-registry-5ddb5757f4-cbdk5\" (UID: \"c9d0687c-a6af-4745-9cbb-b745d52ef591\") " pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.152883 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.152678    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9d0687c-a6af-4745-9cbb-b745d52ef591-ca-trust-extracted\") pod \"image-registry-5ddb5757f4-cbdk5\" (UID: \"c9d0687c-a6af-4745-9cbb-b745d52ef591\") " pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.152883 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.152694    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhvnq\" (UniqueName: \"kubernetes.io/projected/c9d0687c-a6af-4745-9cbb-b745d52ef591-kube-api-access-lhvnq\") pod \"image-registry-5ddb5757f4-cbdk5\" (UID: \"c9d0687c-a6af-4745-9cbb-b745d52ef591\") " pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.152883 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.152712    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9d0687c-a6af-4745-9cbb-b745d52ef591-trusted-ca\") pod \"image-registry-5ddb5757f4-cbdk5\" (UID: \"c9d0687c-a6af-4745-9cbb-b745d52ef591\") " pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.152883 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.152733    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9d0687c-a6af-4745-9cbb-b745d52ef591-installation-pull-secrets\") pod \"image-registry-5ddb5757f4-cbdk5\" (UID: \"c9d0687c-a6af-4745-9cbb-b745d52ef591\") " pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.155378 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.155348    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dzgxv" event={"ID":"3dfe9291-e76b-4d25-91bf-4c3d78d62841","Type":"ContainerStarted","Data":"1b1702943e4c960cd3d007971a96bd507848815e1daad6588b7adf5630232a8a"}
Sep 10 21:00:08.158948 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.158919    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jskzm" event={"ID":"b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e","Type":"ContainerStarted","Data":"92a3f0f55797b30037b2ae16a69141920384c2804bc179f70e6d9389daae7210"}
Sep 10 21:00:08.253087 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.253054    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9d0687c-a6af-4745-9cbb-b745d52ef591-registry-tls\") pod \"image-registry-5ddb5757f4-cbdk5\" (UID: \"c9d0687c-a6af-4745-9cbb-b745d52ef591\") " pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.253087 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.253090    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9d0687c-a6af-4745-9cbb-b745d52ef591-ca-trust-extracted\") pod \"image-registry-5ddb5757f4-cbdk5\" (UID: \"c9d0687c-a6af-4745-9cbb-b745d52ef591\") " pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.253395 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.253114    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhvnq\" (UniqueName: \"kubernetes.io/projected/c9d0687c-a6af-4745-9cbb-b745d52ef591-kube-api-access-lhvnq\") pod \"image-registry-5ddb5757f4-cbdk5\" (UID: \"c9d0687c-a6af-4745-9cbb-b745d52ef591\") " pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.253395 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.253253    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9d0687c-a6af-4745-9cbb-b745d52ef591-trusted-ca\") pod \"image-registry-5ddb5757f4-cbdk5\" (UID: \"c9d0687c-a6af-4745-9cbb-b745d52ef591\") " pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.253395 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.253292    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9d0687c-a6af-4745-9cbb-b745d52ef591-installation-pull-secrets\") pod \"image-registry-5ddb5757f4-cbdk5\" (UID: \"c9d0687c-a6af-4745-9cbb-b745d52ef591\") " pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.253395 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.253321    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9d0687c-a6af-4745-9cbb-b745d52ef591-bound-sa-token\") pod \"image-registry-5ddb5757f4-cbdk5\" (UID: \"c9d0687c-a6af-4745-9cbb-b745d52ef591\") " pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.253395 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.253375    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9d0687c-a6af-4745-9cbb-b745d52ef591-registry-certificates\") pod \"image-registry-5ddb5757f4-cbdk5\" (UID: \"c9d0687c-a6af-4745-9cbb-b745d52ef591\") " pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.253693 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.253564    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9d0687c-a6af-4745-9cbb-b745d52ef591-ca-trust-extracted\") pod \"image-registry-5ddb5757f4-cbdk5\" (UID: \"c9d0687c-a6af-4745-9cbb-b745d52ef591\") " pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.253896 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.253853    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9d0687c-a6af-4745-9cbb-b745d52ef591-image-registry-private-configuration\") pod \"image-registry-5ddb5757f4-cbdk5\" (UID: \"c9d0687c-a6af-4745-9cbb-b745d52ef591\") " pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.254256 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.254230    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9d0687c-a6af-4745-9cbb-b745d52ef591-registry-certificates\") pod \"image-registry-5ddb5757f4-cbdk5\" (UID: \"c9d0687c-a6af-4745-9cbb-b745d52ef591\") " pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.255117 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.255071    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9d0687c-a6af-4745-9cbb-b745d52ef591-trusted-ca\") pod \"image-registry-5ddb5757f4-cbdk5\" (UID: \"c9d0687c-a6af-4745-9cbb-b745d52ef591\") " pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.256137 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.256117    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9d0687c-a6af-4745-9cbb-b745d52ef591-registry-tls\") pod \"image-registry-5ddb5757f4-cbdk5\" (UID: \"c9d0687c-a6af-4745-9cbb-b745d52ef591\") " pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.256274 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.256180    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9d0687c-a6af-4745-9cbb-b745d52ef591-installation-pull-secrets\") pod \"image-registry-5ddb5757f4-cbdk5\" (UID: \"c9d0687c-a6af-4745-9cbb-b745d52ef591\") " pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.256635 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.256607    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9d0687c-a6af-4745-9cbb-b745d52ef591-image-registry-private-configuration\") pod \"image-registry-5ddb5757f4-cbdk5\" (UID: \"c9d0687c-a6af-4745-9cbb-b745d52ef591\") " pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.264331 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.264299    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhvnq\" (UniqueName: \"kubernetes.io/projected/c9d0687c-a6af-4745-9cbb-b745d52ef591-kube-api-access-lhvnq\") pod \"image-registry-5ddb5757f4-cbdk5\" (UID: \"c9d0687c-a6af-4745-9cbb-b745d52ef591\") " pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.264513 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.264491    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9d0687c-a6af-4745-9cbb-b745d52ef591-bound-sa-token\") pod \"image-registry-5ddb5757f4-cbdk5\" (UID: \"c9d0687c-a6af-4745-9cbb-b745d52ef591\") " pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.368635 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.368594    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:08.653337 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:08.653279    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5ddb5757f4-cbdk5"]
Sep 10 21:00:08.657683 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:00:08.657640    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d0687c_a6af_4745_9cbb_b745d52ef591.slice/crio-b3a5064115cb8eb15e0a09edf4460cc9d2001bc2f0cf54bd7c319421730ffeef WatchSource:0}: Error finding container b3a5064115cb8eb15e0a09edf4460cc9d2001bc2f0cf54bd7c319421730ffeef: Status 404 returned error can't find the container with id b3a5064115cb8eb15e0a09edf4460cc9d2001bc2f0cf54bd7c319421730ffeef
Sep 10 21:00:09.168318 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:09.168170    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5" event={"ID":"c9d0687c-a6af-4745-9cbb-b745d52ef591","Type":"ContainerStarted","Data":"8dfb7ae078281a6ee029b75a4aebb460e994160a831890d6944c5b1925bb5c0b"}
Sep 10 21:00:09.168318 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:09.168247    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5" event={"ID":"c9d0687c-a6af-4745-9cbb-b745d52ef591","Type":"ContainerStarted","Data":"b3a5064115cb8eb15e0a09edf4460cc9d2001bc2f0cf54bd7c319421730ffeef"}
Sep 10 21:00:09.168318 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:09.168282    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:09.170973 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:09.170909    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jskzm" event={"ID":"b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e","Type":"ContainerStarted","Data":"f44a52745c60cfd3dcfba6cc50b5e41df8aa28b4ef689738d4d06f24bce5db8e"}
Sep 10 21:00:09.195249 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:09.195144    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5" podStartSLOduration=1.19512251 podStartE2EDuration="1.19512251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:09.193738132 +0000 UTC m=+68.858762929" watchObservedRunningTime="2026-09-10 21:00:09.19512251 +0000 UTC m=+68.860147317"
Sep 10 21:00:09.214057 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:09.213408    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jskzm" podStartSLOduration=67.372064507 podStartE2EDuration="1m8.213388459s" totalImagesPullingTime="841.323952ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:09.212306775 +0000 UTC m=+68.877331576" watchObservedRunningTime="2026-09-10 21:00:09.213388459 +0000 UTC m=+68.878413258"
Sep 10 21:00:10.786148 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:10.786109    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/93462573-692a-4246-afd9-08c9430c7f53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zkbqs\" (UID: \"93462573-692a-4246-afd9-08c9430c7f53\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zkbqs"
Sep 10 21:00:10.786655 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:10.786199    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29f5e959-7b2f-42e9-98f2-a3547859bb56-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-pjkzj\" (UID: \"29f5e959-7b2f-42e9-98f2-a3547859bb56\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pjkzj"
Sep 10 21:00:10.789120 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:10.789098    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29f5e959-7b2f-42e9-98f2-a3547859bb56-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-pjkzj\" (UID: \"29f5e959-7b2f-42e9-98f2-a3547859bb56\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pjkzj"
Sep 10 21:00:10.789246 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:10.789187    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/93462573-692a-4246-afd9-08c9430c7f53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zkbqs\" (UID: \"93462573-692a-4246-afd9-08c9430c7f53\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zkbqs"
Sep 10 21:00:10.987368 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:10.987149    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xjdhx\""
Sep 10 21:00:10.995203 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:10.995178    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zkbqs"
Sep 10 21:00:11.002858 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:11.002836    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-nsbld\""
Sep 10 21:00:11.010707 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:11.010688    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pjkzj"
Sep 10 21:00:12.683074 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:12.682120    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-pjkzj"]
Sep 10 21:00:12.684919 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:12.684701    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-zkbqs"]
Sep 10 21:00:12.687381 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:00:12.687361    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29f5e959_7b2f_42e9_98f2_a3547859bb56.slice/crio-4cdac97bafb31ce0adc249d687307a53c317b72679fe80c8674d04f1661901e5 WatchSource:0}: Error finding container 4cdac97bafb31ce0adc249d687307a53c317b72679fe80c8674d04f1661901e5: Status 404 returned error can't find the container with id 4cdac97bafb31ce0adc249d687307a53c317b72679fe80c8674d04f1661901e5
Sep 10 21:00:12.689049 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:00:12.689025    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93462573_692a_4246_afd9_08c9430c7f53.slice/crio-72bf63b268dec6a7f9aae1222e6cb356b236fdc6f0a5906d59cab5db7cb71f2b WatchSource:0}: Error finding container 72bf63b268dec6a7f9aae1222e6cb356b236fdc6f0a5906d59cab5db7cb71f2b: Status 404 returned error can't find the container with id 72bf63b268dec6a7f9aae1222e6cb356b236fdc6f0a5906d59cab5db7cb71f2b
Sep 10 21:00:13.185990 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:13.185951    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-k8v5s" event={"ID":"eb76c9c7-abc6-4d93-b8e8-f68110f905a9","Type":"ContainerStarted","Data":"2ac92b0298105f7297438c8c554f9731fc293a27bdb0b56f67a87461ae1b1434"}
Sep 10 21:00:13.187528 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:13.187502    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6hc86" event={"ID":"c12d6b27-9952-4214-8b9d-2864f13966f9","Type":"ContainerStarted","Data":"b247b9630442603fa3794f252e1bb57c149f495ab477615fca0979bbd97561ba"}
Sep 10 21:00:13.187657 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:13.187552    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 21:00:13.188770 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:13.188748    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zkbqs" event={"ID":"93462573-692a-4246-afd9-08c9430c7f53","Type":"ContainerStarted","Data":"72bf63b268dec6a7f9aae1222e6cb356b236fdc6f0a5906d59cab5db7cb71f2b"}
Sep 10 21:00:13.189920 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:13.189895    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pjkzj" event={"ID":"29f5e959-7b2f-42e9-98f2-a3547859bb56","Type":"ContainerStarted","Data":"4cdac97bafb31ce0adc249d687307a53c317b72679fe80c8674d04f1661901e5"}
Sep 10 21:00:13.191659 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:13.191636    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dzgxv" event={"ID":"3dfe9291-e76b-4d25-91bf-4c3d78d62841","Type":"ContainerStarted","Data":"12e53e08d9d097fa5d2882639b7d86ff018ea44a03d76fca958bb2d480337546"}
Sep 10 21:00:13.191753 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:13.191679    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dzgxv" event={"ID":"3dfe9291-e76b-4d25-91bf-4c3d78d62841","Type":"ContainerStarted","Data":"8d2182600f92e62621472614e0d8a1417f87001459424b4f709d45377ad7c771"}
Sep 10 21:00:13.191806 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:13.191761    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dzgxv"
Sep 10 21:00:13.226342 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:13.226282    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dzgxv" podStartSLOduration=33.976103475 podStartE2EDuration="38.226263372s" totalImagesPullingTime="4.250159897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:13.225103597 +0000 UTC m=+72.890128396" watchObservedRunningTime="2026-09-10 21:00:13.226263372 +0000 UTC m=+72.891288173"
Sep 10 21:00:13.226475 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:13.226386    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-k8v5s" podStartSLOduration=66.858939947 podStartE2EDuration="1m12.226381038s" totalImagesPullingTime="5.367441091s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:13.20478657 +0000 UTC m=+72.869811371" watchObservedRunningTime="2026-09-10 21:00:13.226381038 +0000 UTC m=+72.891405829"
Sep 10 21:00:13.242741 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:13.242695    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6hc86" podStartSLOduration=66.974089376 podStartE2EDuration="1m12.242681179s" totalImagesPullingTime="5.268591803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:13.241790209 +0000 UTC m=+72.906815015" watchObservedRunningTime="2026-09-10 21:00:13.242681179 +0000 UTC m=+72.907705983"
Sep 10 21:00:15.200528 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:15.200492    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zkbqs" event={"ID":"93462573-692a-4246-afd9-08c9430c7f53","Type":"ContainerStarted","Data":"d962a045994b67be5d183d4590964542d7d2b280a810a1763203f70d4561bfaf"}
Sep 10 21:00:15.202193 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:15.202167    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pjkzj" event={"ID":"29f5e959-7b2f-42e9-98f2-a3547859bb56","Type":"ContainerStarted","Data":"928c03ec36df40a20df18ad758d000f51ef57516c92c790c62f3016bdde06412"}
Sep 10 21:00:15.251911 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:15.251861    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zkbqs" podStartSLOduration=35.134045501 podStartE2EDuration="37.251846323s" totalImagesPullingTime="2.117800822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:15.226314307 +0000 UTC m=+74.891339106" watchObservedRunningTime="2026-09-10 21:00:15.251846323 +0000 UTC m=+74.916871113"
Sep 10 21:00:15.252339 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:15.252307    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pjkzj" podStartSLOduration=35.133105986 podStartE2EDuration="37.252298638s" totalImagesPullingTime="2.119192652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:15.251171471 +0000 UTC m=+74.916196271" watchObservedRunningTime="2026-09-10 21:00:15.252298638 +0000 UTC m=+74.917323503"
Sep 10 21:00:17.537921 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:17.537885    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 21:00:17.540114 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:17.540090    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b961536-58d3-41eb-aa81-d1edb5dd1d0e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5r9p\" (UID: \"6b961536-58d3-41eb-aa81-d1edb5dd1d0e\") " pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 21:00:17.551160 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:17.551137    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-89wjp\""
Sep 10 21:00:17.558141 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:17.558125    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f5r9p"
Sep 10 21:00:17.852065 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:17.851998    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f5r9p"]
Sep 10 21:00:17.855240 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:00:17.855211    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b961536_58d3_41eb_aa81_d1edb5dd1d0e.slice/crio-1ecffd5b95f09d1b3276f4ab28f068d28f60af5fff0e646d344a9a476e111aee WatchSource:0}: Error finding container 1ecffd5b95f09d1b3276f4ab28f068d28f60af5fff0e646d344a9a476e111aee: Status 404 returned error can't find the container with id 1ecffd5b95f09d1b3276f4ab28f068d28f60af5fff0e646d344a9a476e111aee
Sep 10 21:00:18.213393 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:18.213359    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f5r9p" event={"ID":"6b961536-58d3-41eb-aa81-d1edb5dd1d0e","Type":"ContainerStarted","Data":"46e757cbe86d23ebd74db948fa5ac43ce11a4b1ce5e53102ae73af3ac1c5095b"}
Sep 10 21:00:18.213574 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:18.213438    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f5r9p" event={"ID":"6b961536-58d3-41eb-aa81-d1edb5dd1d0e","Type":"ContainerStarted","Data":"1ecffd5b95f09d1b3276f4ab28f068d28f60af5fff0e646d344a9a476e111aee"}
Sep 10 21:00:19.218714 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:19.218677    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f5r9p" event={"ID":"6b961536-58d3-41eb-aa81-d1edb5dd1d0e","Type":"ContainerStarted","Data":"31c72b26009208d89943d1d67cfbaea80142e011de627aa36a96f533144890d3"}
Sep 10 21:00:21.228948 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:21.228910    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f5r9p" event={"ID":"6b961536-58d3-41eb-aa81-d1edb5dd1d0e","Type":"ContainerStarted","Data":"d68b8a2867ef8bd22067441e3b9099fc5a46c4f2d8a297c6b21b50cf7e49a2c9"}
Sep 10 21:00:21.252902 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:21.252848    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f5r9p" podStartSLOduration=33.8797266 podStartE2EDuration="36.252830771s" totalImagesPullingTime="2.373104171s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:59:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:21.252618242 +0000 UTC m=+80.917643042" watchObservedRunningTime="2026-09-10 21:00:21.252830771 +0000 UTC m=+80.917855570"
Sep 10 21:00:23.198079 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:23.197967    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dzgxv"
Sep 10 21:00:30.178306 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:30.178274    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5ddb5757f4-cbdk5"
Sep 10 21:00:31.884721 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:31.884683    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2qskx"]
Sep 10 21:00:31.891184 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:31.891159    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:31.893503 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:31.893482    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rpll2\""
Sep 10 21:00:31.893843 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:31.893828    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 21:00:31.894419 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:31.894399    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 21:00:31.894534 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:31.894409    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 21:00:31.894534 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:31.894397    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 21:00:32.046727 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.046690    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x27w9\" (UniqueName: \"kubernetes.io/projected/71550cf6-f533-4e66-8aa2-e9e208d5f700-kube-api-access-x27w9\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.046727 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.046729    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71550cf6-f533-4e66-8aa2-e9e208d5f700-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.046923 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.046752    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/71550cf6-f533-4e66-8aa2-e9e208d5f700-node-exporter-wtmp\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.046923 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.046776    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71550cf6-f533-4e66-8aa2-e9e208d5f700-metrics-client-ca\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.046923 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.046820    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/71550cf6-f533-4e66-8aa2-e9e208d5f700-node-exporter-tls\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.046923 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.046844    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71550cf6-f533-4e66-8aa2-e9e208d5f700-sys\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.046923 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.046887    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/71550cf6-f533-4e66-8aa2-e9e208d5f700-root\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.046923 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.046911    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/71550cf6-f533-4e66-8aa2-e9e208d5f700-node-exporter-accelerators-collector-config\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.047125 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.046961    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/71550cf6-f533-4e66-8aa2-e9e208d5f700-node-exporter-textfile\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.148215 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.148136    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x27w9\" (UniqueName: \"kubernetes.io/projected/71550cf6-f533-4e66-8aa2-e9e208d5f700-kube-api-access-x27w9\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.148215 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.148177    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71550cf6-f533-4e66-8aa2-e9e208d5f700-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.148215 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.148202    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/71550cf6-f533-4e66-8aa2-e9e208d5f700-node-exporter-wtmp\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.148526 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.148227    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71550cf6-f533-4e66-8aa2-e9e208d5f700-metrics-client-ca\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.148526 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.148265    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/71550cf6-f533-4e66-8aa2-e9e208d5f700-node-exporter-tls\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.148526 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.148310    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71550cf6-f533-4e66-8aa2-e9e208d5f700-sys\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.148526 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.148339    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/71550cf6-f533-4e66-8aa2-e9e208d5f700-root\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.148526 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.148367    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/71550cf6-f533-4e66-8aa2-e9e208d5f700-node-exporter-wtmp\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.148526 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.148397    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/71550cf6-f533-4e66-8aa2-e9e208d5f700-root\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.148526 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:00:32.148401    2584 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 21:00:32.148526 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.148409    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71550cf6-f533-4e66-8aa2-e9e208d5f700-sys\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.148526 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:00:32.148483    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71550cf6-f533-4e66-8aa2-e9e208d5f700-node-exporter-tls podName:71550cf6-f533-4e66-8aa2-e9e208d5f700 nodeName:}" failed. No retries permitted until 2026-09-10 21:00:32.6484597 +0000 UTC m=+92.313484504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/71550cf6-f533-4e66-8aa2-e9e208d5f700-node-exporter-tls") pod "node-exporter-2qskx" (UID: "71550cf6-f533-4e66-8aa2-e9e208d5f700") : secret "node-exporter-tls" not found
Sep 10 21:00:32.148526 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.148491    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/71550cf6-f533-4e66-8aa2-e9e208d5f700-node-exporter-accelerators-collector-config\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.149021 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.148553    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/71550cf6-f533-4e66-8aa2-e9e208d5f700-node-exporter-textfile\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.149021 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.148855    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/71550cf6-f533-4e66-8aa2-e9e208d5f700-node-exporter-textfile\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.149021 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.148971    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71550cf6-f533-4e66-8aa2-e9e208d5f700-metrics-client-ca\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.149129 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.149008    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/71550cf6-f533-4e66-8aa2-e9e208d5f700-node-exporter-accelerators-collector-config\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.150578 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.150559    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71550cf6-f533-4e66-8aa2-e9e208d5f700-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.161525 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.161501    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x27w9\" (UniqueName: \"kubernetes.io/projected/71550cf6-f533-4e66-8aa2-e9e208d5f700-kube-api-access-x27w9\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.653491 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:32.653460    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/71550cf6-f533-4e66-8aa2-e9e208d5f700-node-exporter-tls\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:32.653651 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:00:32.653567    2584 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 21:00:32.653651 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:00:32.653626    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71550cf6-f533-4e66-8aa2-e9e208d5f700-node-exporter-tls podName:71550cf6-f533-4e66-8aa2-e9e208d5f700 nodeName:}" failed. No retries permitted until 2026-09-10 21:00:33.653610745 +0000 UTC m=+93.318635524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/71550cf6-f533-4e66-8aa2-e9e208d5f700-node-exporter-tls") pod "node-exporter-2qskx" (UID: "71550cf6-f533-4e66-8aa2-e9e208d5f700") : secret "node-exporter-tls" not found
Sep 10 21:00:33.661384 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:33.661348    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/71550cf6-f533-4e66-8aa2-e9e208d5f700-node-exporter-tls\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:33.663745 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:33.663719    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/71550cf6-f533-4e66-8aa2-e9e208d5f700-node-exporter-tls\") pod \"node-exporter-2qskx\" (UID: \"71550cf6-f533-4e66-8aa2-e9e208d5f700\") " pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:33.699998 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:33.699962    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2qskx"
Sep 10 21:00:33.711370 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:00:33.711341    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71550cf6_f533_4e66_8aa2_e9e208d5f700.slice/crio-4d338953e640e0dc04e12c92afec1376b932eda6aad9f63689fe12a199da8ba0 WatchSource:0}: Error finding container 4d338953e640e0dc04e12c92afec1376b932eda6aad9f63689fe12a199da8ba0: Status 404 returned error can't find the container with id 4d338953e640e0dc04e12c92afec1376b932eda6aad9f63689fe12a199da8ba0
Sep 10 21:00:34.277343 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:34.277304    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2qskx" event={"ID":"71550cf6-f533-4e66-8aa2-e9e208d5f700","Type":"ContainerStarted","Data":"4d338953e640e0dc04e12c92afec1376b932eda6aad9f63689fe12a199da8ba0"}
Sep 10 21:00:35.282144 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:35.282109    2584 generic.go:356] "Generic (PLEG): container finished" podID="71550cf6-f533-4e66-8aa2-e9e208d5f700" containerID="579b3fddbf0f1c8c357f1eade996cb7223dc2e3c9bf78a6ad2375af35391a5cd" exitCode=0
Sep 10 21:00:35.282529 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:35.282148    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2qskx" event={"ID":"71550cf6-f533-4e66-8aa2-e9e208d5f700","Type":"ContainerDied","Data":"579b3fddbf0f1c8c357f1eade996cb7223dc2e3c9bf78a6ad2375af35391a5cd"}
Sep 10 21:00:36.288320 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.288273    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2qskx" event={"ID":"71550cf6-f533-4e66-8aa2-e9e208d5f700","Type":"ContainerStarted","Data":"0fc13ed452e2cf1239a707b8c19b85324c035d5738b61614e3c2d10faa298543"}
Sep 10 21:00:36.288814 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.288333    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2qskx" event={"ID":"71550cf6-f533-4e66-8aa2-e9e208d5f700","Type":"ContainerStarted","Data":"b999eb36b097fcbf770765fe05d39f36bc7b391df297e293da692486ea3e2657"}
Sep 10 21:00:36.323667 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.323616    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2qskx" podStartSLOduration=4.210563824 podStartE2EDuration="5.323602098s" totalImagesPullingTime="1.050788052s" totalInitContainerRuntime="62.250222ms" isStatefulPod=true podCreationTimestamp="2026-09-10 21:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:36.321535061 +0000 UTC m=+95.986559862" watchObservedRunningTime="2026-09-10 21:00:36.323602098 +0000 UTC m=+95.988626897"
Sep 10 21:00:36.339680 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.339650    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-d6cc4887f-2xhtg"]
Sep 10 21:00:36.342742 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.342723    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:00:36.346816 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.346796    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xcv29\""
Sep 10 21:00:36.347078 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.347064    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 10 21:00:36.347202 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.347184    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 10 21:00:36.347257 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.347206    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8ens7tactovti\""
Sep 10 21:00:36.347306 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.347256    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 10 21:00:36.347306 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.347282    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 21:00:36.368397 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.368374    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d6cc4887f-2xhtg"]
Sep 10 21:00:36.385604 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.385582    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/42269fc7-bacc-4820-9835-a8a814cb2134-audit-log\") pod \"metrics-server-d6cc4887f-2xhtg\" (UID: \"42269fc7-bacc-4820-9835-a8a814cb2134\") " pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:00:36.385701 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.385614    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/42269fc7-bacc-4820-9835-a8a814cb2134-secret-metrics-server-client-certs\") pod \"metrics-server-d6cc4887f-2xhtg\" (UID: \"42269fc7-bacc-4820-9835-a8a814cb2134\") " pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:00:36.385701 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.385638    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/42269fc7-bacc-4820-9835-a8a814cb2134-metrics-server-audit-profiles\") pod \"metrics-server-d6cc4887f-2xhtg\" (UID: \"42269fc7-bacc-4820-9835-a8a814cb2134\") " pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:00:36.385803 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.385703    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh4cb\" (UniqueName: \"kubernetes.io/projected/42269fc7-bacc-4820-9835-a8a814cb2134-kube-api-access-vh4cb\") pod \"metrics-server-d6cc4887f-2xhtg\" (UID: \"42269fc7-bacc-4820-9835-a8a814cb2134\") " pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:00:36.385803 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.385758    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/42269fc7-bacc-4820-9835-a8a814cb2134-secret-metrics-server-tls\") pod \"metrics-server-d6cc4887f-2xhtg\" (UID: \"42269fc7-bacc-4820-9835-a8a814cb2134\") " pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:00:36.385900 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.385828    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42269fc7-bacc-4820-9835-a8a814cb2134-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d6cc4887f-2xhtg\" (UID: \"42269fc7-bacc-4820-9835-a8a814cb2134\") " pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:00:36.385900 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.385854    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/42269fc7-bacc-4820-9835-a8a814cb2134-client-ca-bundle\") pod \"metrics-server-d6cc4887f-2xhtg\" (UID: \"42269fc7-bacc-4820-9835-a8a814cb2134\") " pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:00:36.486593 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.486562    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42269fc7-bacc-4820-9835-a8a814cb2134-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d6cc4887f-2xhtg\" (UID: \"42269fc7-bacc-4820-9835-a8a814cb2134\") " pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:00:36.486593 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.486596    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/42269fc7-bacc-4820-9835-a8a814cb2134-client-ca-bundle\") pod \"metrics-server-d6cc4887f-2xhtg\" (UID: \"42269fc7-bacc-4820-9835-a8a814cb2134\") " pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:00:36.486763 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.486629    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/42269fc7-bacc-4820-9835-a8a814cb2134-audit-log\") pod \"metrics-server-d6cc4887f-2xhtg\" (UID: \"42269fc7-bacc-4820-9835-a8a814cb2134\") " pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:00:36.486763 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.486653    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/42269fc7-bacc-4820-9835-a8a814cb2134-secret-metrics-server-client-certs\") pod \"metrics-server-d6cc4887f-2xhtg\" (UID: \"42269fc7-bacc-4820-9835-a8a814cb2134\") " pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:00:36.486763 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.486683    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/42269fc7-bacc-4820-9835-a8a814cb2134-metrics-server-audit-profiles\") pod \"metrics-server-d6cc4887f-2xhtg\" (UID: \"42269fc7-bacc-4820-9835-a8a814cb2134\") " pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:00:36.486763 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.486706    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh4cb\" (UniqueName: \"kubernetes.io/projected/42269fc7-bacc-4820-9835-a8a814cb2134-kube-api-access-vh4cb\") pod \"metrics-server-d6cc4887f-2xhtg\" (UID: \"42269fc7-bacc-4820-9835-a8a814cb2134\") " pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:00:36.486763 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.486729    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/42269fc7-bacc-4820-9835-a8a814cb2134-secret-metrics-server-tls\") pod \"metrics-server-d6cc4887f-2xhtg\" (UID: \"42269fc7-bacc-4820-9835-a8a814cb2134\") " pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:00:36.487042 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.487018    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/42269fc7-bacc-4820-9835-a8a814cb2134-audit-log\") pod \"metrics-server-d6cc4887f-2xhtg\" (UID: \"42269fc7-bacc-4820-9835-a8a814cb2134\") " pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:00:36.487355 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.487322    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42269fc7-bacc-4820-9835-a8a814cb2134-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d6cc4887f-2xhtg\" (UID: \"42269fc7-bacc-4820-9835-a8a814cb2134\") " pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:00:36.487735 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.487716    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/42269fc7-bacc-4820-9835-a8a814cb2134-metrics-server-audit-profiles\") pod \"metrics-server-d6cc4887f-2xhtg\" (UID: \"42269fc7-bacc-4820-9835-a8a814cb2134\") " pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:00:36.489448 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.489404    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/42269fc7-bacc-4820-9835-a8a814cb2134-secret-metrics-server-client-certs\") pod \"metrics-server-d6cc4887f-2xhtg\" (UID: \"42269fc7-bacc-4820-9835-a8a814cb2134\") " pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:00:36.489679 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.489662    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/42269fc7-bacc-4820-9835-a8a814cb2134-secret-metrics-server-tls\") pod \"metrics-server-d6cc4887f-2xhtg\" (UID: \"42269fc7-bacc-4820-9835-a8a814cb2134\") " pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:00:36.489808 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.489790    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/42269fc7-bacc-4820-9835-a8a814cb2134-client-ca-bundle\") pod \"metrics-server-d6cc4887f-2xhtg\" (UID: \"42269fc7-bacc-4820-9835-a8a814cb2134\") " pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:00:36.511341 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.511312    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh4cb\" (UniqueName: \"kubernetes.io/projected/42269fc7-bacc-4820-9835-a8a814cb2134-kube-api-access-vh4cb\") pod \"metrics-server-d6cc4887f-2xhtg\" (UID: \"42269fc7-bacc-4820-9835-a8a814cb2134\") " pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:00:36.652260 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.652229    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:00:36.952329 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:36.952298    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d6cc4887f-2xhtg"]
Sep 10 21:00:36.954754 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:00:36.954720    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42269fc7_bacc_4820_9835_a8a814cb2134.slice/crio-54ccd82b0238b73a696a0f0636e1bb01c7e947074314ce27f9156db4b7ce495a WatchSource:0}: Error finding container 54ccd82b0238b73a696a0f0636e1bb01c7e947074314ce27f9156db4b7ce495a: Status 404 returned error can't find the container with id 54ccd82b0238b73a696a0f0636e1bb01c7e947074314ce27f9156db4b7ce495a
Sep 10 21:00:37.292606 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:37.292521    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg" event={"ID":"42269fc7-bacc-4820-9835-a8a814cb2134","Type":"ContainerStarted","Data":"54ccd82b0238b73a696a0f0636e1bb01c7e947074314ce27f9156db4b7ce495a"}
Sep 10 21:00:38.339758 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.339722    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 21:00:38.343301 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.343283    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.347179 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.347150    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 21:00:38.348669 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.348647    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 21:00:38.350123 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.349853    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 21:00:38.350247 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.350228    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 21:00:38.350347 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.350326    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 21:00:38.350500 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.350476    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 21:00:38.350570 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.350487    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 21:00:38.350570 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.350517    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 21:00:38.350570 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.350532    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 21:00:38.350715 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.350653    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 21:00:38.350771 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.350757    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fec6qsthb40hv\""
Sep 10 21:00:38.351089 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.350830    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-84wdq\""
Sep 10 21:00:38.351089 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.350838    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 21:00:38.351381 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.351364    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 21:00:38.357231 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.357215    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 21:00:38.365370 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.365346    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 21:00:38.387136 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.387114    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 21:00:38.403395 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.403298    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.403395 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.403354    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-config\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.403539 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.403463    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a48b2cb1-5c53-4002-9cc9-f54d995541dd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.403539 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.403497    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a48b2cb1-5c53-4002-9cc9-f54d995541dd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.403539 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.403520    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a48b2cb1-5c53-4002-9cc9-f54d995541dd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.403679 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.403548    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a48b2cb1-5c53-4002-9cc9-f54d995541dd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.403679 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.403582    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a48b2cb1-5c53-4002-9cc9-f54d995541dd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.403679 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.403609    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a48b2cb1-5c53-4002-9cc9-f54d995541dd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.403679 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.403624    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.403679 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.403640    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a48b2cb1-5c53-4002-9cc9-f54d995541dd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.403679 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.403663    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.403899 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.403694    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.403899 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.403712    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qqfm\" (UniqueName: \"kubernetes.io/projected/a48b2cb1-5c53-4002-9cc9-f54d995541dd-kube-api-access-7qqfm\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.403899 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.403732    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.403899 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.403775    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a48b2cb1-5c53-4002-9cc9-f54d995541dd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.403899 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.403848    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.404109 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.403901    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.404109 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.403941    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-web-config\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.404109 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.403985    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a48b2cb1-5c53-4002-9cc9-f54d995541dd-config-out\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.404109 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.404005    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a48b2cb1-5c53-4002-9cc9-f54d995541dd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.504623 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.504588    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.504623 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.504626    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-config\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.504869 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.504651    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a48b2cb1-5c53-4002-9cc9-f54d995541dd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.504869 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.504671    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a48b2cb1-5c53-4002-9cc9-f54d995541dd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.504869 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.504694    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a48b2cb1-5c53-4002-9cc9-f54d995541dd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.505016 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.504944    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a48b2cb1-5c53-4002-9cc9-f54d995541dd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.505016 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.505005    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a48b2cb1-5c53-4002-9cc9-f54d995541dd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.505113 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.505046    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a48b2cb1-5c53-4002-9cc9-f54d995541dd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.505113 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.505076    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.505113 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.505104    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a48b2cb1-5c53-4002-9cc9-f54d995541dd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.505262 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.505128    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.505262 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.505170    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.505262 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.505202    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qqfm\" (UniqueName: \"kubernetes.io/projected/a48b2cb1-5c53-4002-9cc9-f54d995541dd-kube-api-access-7qqfm\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.505262 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.505231    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.505262 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.505251    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a48b2cb1-5c53-4002-9cc9-f54d995541dd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.505540 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.505286    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.505540 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.505316    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.505540 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.505337    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-web-config\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.505540 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.505362    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a48b2cb1-5c53-4002-9cc9-f54d995541dd-config-out\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.505540 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.505385    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a48b2cb1-5c53-4002-9cc9-f54d995541dd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.505540 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.505388    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a48b2cb1-5c53-4002-9cc9-f54d995541dd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.505827 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.505664    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a48b2cb1-5c53-4002-9cc9-f54d995541dd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.505827 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.505676    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a48b2cb1-5c53-4002-9cc9-f54d995541dd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.505943 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.505918    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a48b2cb1-5c53-4002-9cc9-f54d995541dd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.507006 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.506546    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a48b2cb1-5c53-4002-9cc9-f54d995541dd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.507649 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.507216    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a48b2cb1-5c53-4002-9cc9-f54d995541dd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.508595 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.508161    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.508595 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.508178    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.509567 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.509523    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-config\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.509859 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.509829    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a48b2cb1-5c53-4002-9cc9-f54d995541dd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.510125 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.510098    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.510229 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.510170    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a48b2cb1-5c53-4002-9cc9-f54d995541dd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.510410 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.510390    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.510624 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.510607    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a48b2cb1-5c53-4002-9cc9-f54d995541dd-config-out\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.511039 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.511018    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-web-config\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.511123 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.511050    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a48b2cb1-5c53-4002-9cc9-f54d995541dd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.511937 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.511897    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.512008 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.511958    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.512559 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.512540    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a48b2cb1-5c53-4002-9cc9-f54d995541dd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.518085 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.518060    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qqfm\" (UniqueName: \"kubernetes.io/projected/a48b2cb1-5c53-4002-9cc9-f54d995541dd-kube-api-access-7qqfm\") pod \"prometheus-k8s-0\" (UID: \"a48b2cb1-5c53-4002-9cc9-f54d995541dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.654433 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.654346    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:38.953703 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:38.953536    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 21:00:38.955860 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:00:38.955835    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48b2cb1_5c53_4002_9cc9_f54d995541dd.slice/crio-a3ecf5c2e4c48e043d66706f54b4424b40732c582cab9d705133f09a88c7d5fa WatchSource:0}: Error finding container a3ecf5c2e4c48e043d66706f54b4424b40732c582cab9d705133f09a88c7d5fa: Status 404 returned error can't find the container with id a3ecf5c2e4c48e043d66706f54b4424b40732c582cab9d705133f09a88c7d5fa
Sep 10 21:00:39.301470 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:39.301353    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg" event={"ID":"42269fc7-bacc-4820-9835-a8a814cb2134","Type":"ContainerStarted","Data":"727ed1381c91ea9c30ad6f37281a12bf6ab4a2822a343eacb9e7f67925c27325"}
Sep 10 21:00:39.302623 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:39.302591    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a48b2cb1-5c53-4002-9cc9-f54d995541dd","Type":"ContainerStarted","Data":"a3ecf5c2e4c48e043d66706f54b4424b40732c582cab9d705133f09a88c7d5fa"}
Sep 10 21:00:39.333956 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:39.333910    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg" podStartSLOduration=1.937227772 podStartE2EDuration="3.333896509s" totalImagesPullingTime="1.396668737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:00:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:39.333700112 +0000 UTC m=+98.998724912" watchObservedRunningTime="2026-09-10 21:00:39.333896509 +0000 UTC m=+98.998921309"
Sep 10 21:00:41.311679 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:41.311643    2584 generic.go:356] "Generic (PLEG): container finished" podID="a48b2cb1-5c53-4002-9cc9-f54d995541dd" containerID="bf1713fa0d13e6c6c64a5c58cb3f7ba5a9d7c7df6ab7d295f8f9515e02305643" exitCode=0
Sep 10 21:00:41.312041 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:41.311683    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a48b2cb1-5c53-4002-9cc9-f54d995541dd","Type":"ContainerDied","Data":"bf1713fa0d13e6c6c64a5c58cb3f7ba5a9d7c7df6ab7d295f8f9515e02305643"}
Sep 10 21:00:44.198033 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:44.198000    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6hc86"
Sep 10 21:00:45.331154 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:45.331122    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a48b2cb1-5c53-4002-9cc9-f54d995541dd","Type":"ContainerStarted","Data":"5b02fc6c3cee7f75f5d3fd9c4dd883c1dda56efe703b1ab239f76e46fbf61430"}
Sep 10 21:00:45.331571 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:45.331198    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a48b2cb1-5c53-4002-9cc9-f54d995541dd","Type":"ContainerStarted","Data":"7ae22ea2c2fedc6ce1b60022fb89c01f9c7982fb7b22470f1836100b0a6e668d"}
Sep 10 21:00:47.344209 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:47.344180    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a48b2cb1-5c53-4002-9cc9-f54d995541dd","Type":"ContainerStarted","Data":"11e86b5ad63d4dd8817bc8ecd818456326d1956f5cbb0bd8319fb39e29833e03"}
Sep 10 21:00:47.344548 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:47.344259    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a48b2cb1-5c53-4002-9cc9-f54d995541dd","Type":"ContainerStarted","Data":"7fe19e74fc2edf140c6fc696f62a39f82503052e7297b272dfbc1eb573cb140c"}
Sep 10 21:00:48.351466 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:48.351412    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a48b2cb1-5c53-4002-9cc9-f54d995541dd","Type":"ContainerStarted","Data":"692080a0a2f071c6377d46df32d4966be77582a0aeb3b8a74a316e6e4d2ad9b4"}
Sep 10 21:00:48.351931 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:48.351537    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a48b2cb1-5c53-4002-9cc9-f54d995541dd","Type":"ContainerStarted","Data":"e497c8b4ab7ed5f259e35f2a1c7fc359284bfceb7845da5587fbe5f7ab672d8d"}
Sep 10 21:00:48.401162 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:48.401114    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.109457284 podStartE2EDuration="10.401094646s" totalImagesPullingTime="7.276903733s" totalInitContainerRuntime="14.733629ms" isStatefulPod=false podCreationTimestamp="2026-09-10 21:00:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:48.400329049 +0000 UTC m=+108.065353842" watchObservedRunningTime="2026-09-10 21:00:48.401094646 +0000 UTC m=+108.066119445"
Sep 10 21:00:48.654879 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:48.654848    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:54.374865 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:54.374831    2584 generic.go:356] "Generic (PLEG): container finished" podID="dbc735a8-26d3-4267-afe5-b334f6e03fd8" containerID="dbd28f54759c558cb994afd9e07c68ebb05f607e56bbdfb9b1f2985f98803015" exitCode=0
Sep 10 21:00:54.375230 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:54.374887    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnf52" event={"ID":"dbc735a8-26d3-4267-afe5-b334f6e03fd8","Type":"ContainerDied","Data":"dbd28f54759c558cb994afd9e07c68ebb05f607e56bbdfb9b1f2985f98803015"}
Sep 10 21:00:54.375230 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:54.375198    2584 scope.go:117] "RemoveContainer" containerID="dbd28f54759c558cb994afd9e07c68ebb05f607e56bbdfb9b1f2985f98803015"
Sep 10 21:00:55.380489 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:55.380454    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnf52" event={"ID":"dbc735a8-26d3-4267-afe5-b334f6e03fd8","Type":"ContainerStarted","Data":"bbc8996df5a7656d6223633ca654b887dcc523bbc5547ad8da958b7a5f03f6a9"}
Sep 10 21:00:56.652727 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:56.652691    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:00:56.653167 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:00:56.652760    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:01:14.452169 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:01:14.452135    2584 generic.go:356] "Generic (PLEG): container finished" podID="367d0cac-af3c-4dce-acc3-ce2e5f85fce1" containerID="a27c1cdc77ac5504b5ac8b8d9324767c7eb0616d0d4bfaf65e35eae519c09297" exitCode=0
Sep 10 21:01:14.452665 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:01:14.452184    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-hpt25" event={"ID":"367d0cac-af3c-4dce-acc3-ce2e5f85fce1","Type":"ContainerDied","Data":"a27c1cdc77ac5504b5ac8b8d9324767c7eb0616d0d4bfaf65e35eae519c09297"}
Sep 10 21:01:14.452665 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:01:14.452529    2584 scope.go:117] "RemoveContainer" containerID="a27c1cdc77ac5504b5ac8b8d9324767c7eb0616d0d4bfaf65e35eae519c09297"
Sep 10 21:01:15.458059 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:01:15.458027    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-hpt25" event={"ID":"367d0cac-af3c-4dce-acc3-ce2e5f85fce1","Type":"ContainerStarted","Data":"34a2e38946774b900867c0e30d8d5239da37f719688baa1209f749c559818f3c"}
Sep 10 21:01:16.658490 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:01:16.658460    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:01:16.662208 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:01:16.662184    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-d6cc4887f-2xhtg"
Sep 10 21:01:38.655323 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:01:38.655288    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:01:38.674842 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:01:38.674818    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:01:39.561760 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:01:39.561728    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:04:00.713890 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:00.713862    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-jkkfm_fd81843e-6283-43e3-a571-d5e47be26e38/console-operator/1.log"
Sep 10 21:04:00.715560 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:00.715540    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-jkkfm_fd81843e-6283-43e3-a571-d5e47be26e38/console-operator/1.log"
Sep 10 21:04:00.730038 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:00.730015    2584 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 21:04:18.757122 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:18.757085    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb"]
Sep 10 21:04:18.759198 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:18.759181    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb"
Sep 10 21:04:18.761932 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:18.761909    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 21:04:18.762054 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:18.761919    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 21:04:18.762054 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:18.762022    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 21:04:18.762170 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:18.762098    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cv4zb\""
Sep 10 21:04:18.762170 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:18.762131    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 21:04:18.773328 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:18.773308    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb"]
Sep 10 21:04:18.830826 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:18.830797    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cace0196-0854-40eb-b897-917323c7b8ec-tmp\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-m5jrb\" (UID: \"cace0196-0854-40eb-b897-917323c7b8ec\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb"
Sep 10 21:04:18.830977 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:18.830837    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cace0196-0854-40eb-b897-917323c7b8ec-manifests\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-m5jrb\" (UID: \"cace0196-0854-40eb-b897-917323c7b8ec\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb"
Sep 10 21:04:18.830977 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:18.830868    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz4qn\" (UniqueName: \"kubernetes.io/projected/cace0196-0854-40eb-b897-917323c7b8ec-kube-api-access-zz4qn\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-m5jrb\" (UID: \"cace0196-0854-40eb-b897-917323c7b8ec\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb"
Sep 10 21:04:18.831055 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:18.830986    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cace0196-0854-40eb-b897-917323c7b8ec-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-m5jrb\" (UID: \"cace0196-0854-40eb-b897-917323c7b8ec\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb"
Sep 10 21:04:18.831055 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:18.831018    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cace0196-0854-40eb-b897-917323c7b8ec-webhook-cert\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-m5jrb\" (UID: \"cace0196-0854-40eb-b897-917323c7b8ec\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb"
Sep 10 21:04:18.931609 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:18.931571    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cace0196-0854-40eb-b897-917323c7b8ec-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-m5jrb\" (UID: \"cace0196-0854-40eb-b897-917323c7b8ec\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb"
Sep 10 21:04:18.931609 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:18.931612    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cace0196-0854-40eb-b897-917323c7b8ec-webhook-cert\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-m5jrb\" (UID: \"cace0196-0854-40eb-b897-917323c7b8ec\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb"
Sep 10 21:04:18.931853 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:18.931663    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cace0196-0854-40eb-b897-917323c7b8ec-tmp\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-m5jrb\" (UID: \"cace0196-0854-40eb-b897-917323c7b8ec\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb"
Sep 10 21:04:18.931913 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:18.931852    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cace0196-0854-40eb-b897-917323c7b8ec-manifests\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-m5jrb\" (UID: \"cace0196-0854-40eb-b897-917323c7b8ec\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb"
Sep 10 21:04:18.931966 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:18.931941    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz4qn\" (UniqueName: \"kubernetes.io/projected/cace0196-0854-40eb-b897-917323c7b8ec-kube-api-access-zz4qn\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-m5jrb\" (UID: \"cace0196-0854-40eb-b897-917323c7b8ec\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb"
Sep 10 21:04:18.932246 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:18.932210    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cace0196-0854-40eb-b897-917323c7b8ec-manifests\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-m5jrb\" (UID: \"cace0196-0854-40eb-b897-917323c7b8ec\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb"
Sep 10 21:04:18.933969 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:18.933947    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cace0196-0854-40eb-b897-917323c7b8ec-tmp\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-m5jrb\" (UID: \"cace0196-0854-40eb-b897-917323c7b8ec\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb"
Sep 10 21:04:18.934091 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:18.934070    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cace0196-0854-40eb-b897-917323c7b8ec-webhook-cert\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-m5jrb\" (UID: \"cace0196-0854-40eb-b897-917323c7b8ec\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb"
Sep 10 21:04:18.934160 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:18.934143    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cace0196-0854-40eb-b897-917323c7b8ec-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-m5jrb\" (UID: \"cace0196-0854-40eb-b897-917323c7b8ec\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb"
Sep 10 21:04:18.943693 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:18.943671    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz4qn\" (UniqueName: \"kubernetes.io/projected/cace0196-0854-40eb-b897-917323c7b8ec-kube-api-access-zz4qn\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-m5jrb\" (UID: \"cace0196-0854-40eb-b897-917323c7b8ec\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb"
Sep 10 21:04:19.069098 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:19.069012    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb"
Sep 10 21:04:19.346527 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:19.346447    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb"]
Sep 10 21:04:19.351254 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:04:19.351224    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcace0196_0854_40eb_b897_917323c7b8ec.slice/crio-00d89d45b736fc8abcd9c99f77f1503a6734e53696df3905af71a5464b1a0c06 WatchSource:0}: Error finding container 00d89d45b736fc8abcd9c99f77f1503a6734e53696df3905af71a5464b1a0c06: Status 404 returned error can't find the container with id 00d89d45b736fc8abcd9c99f77f1503a6734e53696df3905af71a5464b1a0c06
Sep 10 21:04:19.353482 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:19.353441    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:04:20.128626 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:20.128586    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb" event={"ID":"cace0196-0854-40eb-b897-917323c7b8ec","Type":"ContainerStarted","Data":"00d89d45b736fc8abcd9c99f77f1503a6734e53696df3905af71a5464b1a0c06"}
Sep 10 21:04:23.143630 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:23.143592    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb" event={"ID":"cace0196-0854-40eb-b897-917323c7b8ec","Type":"ContainerStarted","Data":"a93c6e9be47b2df5a2be5752a2443bc8218cbaa542904b32d8243f01a6ffbc75"}
Sep 10 21:04:24.148710 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:24.148673    2584 generic.go:356] "Generic (PLEG): container finished" podID="cace0196-0854-40eb-b897-917323c7b8ec" containerID="a93c6e9be47b2df5a2be5752a2443bc8218cbaa542904b32d8243f01a6ffbc75" exitCode=0
Sep 10 21:04:24.149098 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:24.148747    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb" event={"ID":"cace0196-0854-40eb-b897-917323c7b8ec","Type":"ContainerDied","Data":"a93c6e9be47b2df5a2be5752a2443bc8218cbaa542904b32d8243f01a6ffbc75"}
Sep 10 21:04:25.155690 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:25.155652    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb" event={"ID":"cace0196-0854-40eb-b897-917323c7b8ec","Type":"ContainerStarted","Data":"8f3149273deb14fb91587e9afa572c8122b8bdfb7d91026b66c229959d9b71c4"}
Sep 10 21:04:25.156081 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:25.155723    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb"
Sep 10 21:04:25.178361 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:25.178311    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb" podStartSLOduration=2.761632783 podStartE2EDuration="7.178296527s" totalImagesPullingTime="3.320804866s" totalInitContainerRuntime="1.095858878s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:04:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:04:25.177073367 +0000 UTC m=+324.842098166" watchObservedRunningTime="2026-09-10 21:04:25.178296527 +0000 UTC m=+324.843321326"
Sep 10 21:04:36.162356 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:36.162326    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-m5jrb"
Sep 10 21:04:43.961990 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:43.958833    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54c6c477f9-4bk2f"]
Sep 10 21:04:43.969080 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:43.969048    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54c6c477f9-4bk2f"
Sep 10 21:04:43.974672 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:43.974649    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 21:04:43.974916 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:43.974887    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ssw74\""
Sep 10 21:04:43.975028 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:43.974930    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 21:04:43.975028 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:43.974993    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 21:04:43.975129 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:43.975068    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 21:04:43.981410 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:43.981388    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54c6c477f9-4bk2f"]
Sep 10 21:04:44.040686 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:44.040658    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt8mt\" (UniqueName: \"kubernetes.io/projected/491006ce-4acf-49b3-9a19-210ed7756177-kube-api-access-jt8mt\") pod \"kube-auth-proxy-54c6c477f9-4bk2f\" (UID: \"491006ce-4acf-49b3-9a19-210ed7756177\") " pod="openshift-ingress/kube-auth-proxy-54c6c477f9-4bk2f"
Sep 10 21:04:44.040804 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:44.040704    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/491006ce-4acf-49b3-9a19-210ed7756177-tmp\") pod \"kube-auth-proxy-54c6c477f9-4bk2f\" (UID: \"491006ce-4acf-49b3-9a19-210ed7756177\") " pod="openshift-ingress/kube-auth-proxy-54c6c477f9-4bk2f"
Sep 10 21:04:44.040804 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:44.040726    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/491006ce-4acf-49b3-9a19-210ed7756177-tls-certs\") pod \"kube-auth-proxy-54c6c477f9-4bk2f\" (UID: \"491006ce-4acf-49b3-9a19-210ed7756177\") " pod="openshift-ingress/kube-auth-proxy-54c6c477f9-4bk2f"
Sep 10 21:04:44.142104 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:44.142072    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt8mt\" (UniqueName: \"kubernetes.io/projected/491006ce-4acf-49b3-9a19-210ed7756177-kube-api-access-jt8mt\") pod \"kube-auth-proxy-54c6c477f9-4bk2f\" (UID: \"491006ce-4acf-49b3-9a19-210ed7756177\") " pod="openshift-ingress/kube-auth-proxy-54c6c477f9-4bk2f"
Sep 10 21:04:44.142213 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:44.142115    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/491006ce-4acf-49b3-9a19-210ed7756177-tmp\") pod \"kube-auth-proxy-54c6c477f9-4bk2f\" (UID: \"491006ce-4acf-49b3-9a19-210ed7756177\") " pod="openshift-ingress/kube-auth-proxy-54c6c477f9-4bk2f"
Sep 10 21:04:44.142213 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:44.142135    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/491006ce-4acf-49b3-9a19-210ed7756177-tls-certs\") pod \"kube-auth-proxy-54c6c477f9-4bk2f\" (UID: \"491006ce-4acf-49b3-9a19-210ed7756177\") " pod="openshift-ingress/kube-auth-proxy-54c6c477f9-4bk2f"
Sep 10 21:04:44.144474 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:44.144441    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/491006ce-4acf-49b3-9a19-210ed7756177-tmp\") pod \"kube-auth-proxy-54c6c477f9-4bk2f\" (UID: \"491006ce-4acf-49b3-9a19-210ed7756177\") " pod="openshift-ingress/kube-auth-proxy-54c6c477f9-4bk2f"
Sep 10 21:04:44.144640 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:44.144624    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/491006ce-4acf-49b3-9a19-210ed7756177-tls-certs\") pod \"kube-auth-proxy-54c6c477f9-4bk2f\" (UID: \"491006ce-4acf-49b3-9a19-210ed7756177\") " pod="openshift-ingress/kube-auth-proxy-54c6c477f9-4bk2f"
Sep 10 21:04:44.151961 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:44.151933    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt8mt\" (UniqueName: \"kubernetes.io/projected/491006ce-4acf-49b3-9a19-210ed7756177-kube-api-access-jt8mt\") pod \"kube-auth-proxy-54c6c477f9-4bk2f\" (UID: \"491006ce-4acf-49b3-9a19-210ed7756177\") " pod="openshift-ingress/kube-auth-proxy-54c6c477f9-4bk2f"
Sep 10 21:04:44.278862 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:44.278778    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54c6c477f9-4bk2f"
Sep 10 21:04:44.547615 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:44.547545    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54c6c477f9-4bk2f"]
Sep 10 21:04:44.550828 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:04:44.550795    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod491006ce_4acf_49b3_9a19_210ed7756177.slice/crio-20108191886f01b7953a201fbde4d6d1a630030289c93f38f261149dcd033d76 WatchSource:0}: Error finding container 20108191886f01b7953a201fbde4d6d1a630030289c93f38f261149dcd033d76: Status 404 returned error can't find the container with id 20108191886f01b7953a201fbde4d6d1a630030289c93f38f261149dcd033d76
Sep 10 21:04:45.240970 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:45.240920    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54c6c477f9-4bk2f" event={"ID":"491006ce-4acf-49b3-9a19-210ed7756177","Type":"ContainerStarted","Data":"20108191886f01b7953a201fbde4d6d1a630030289c93f38f261149dcd033d76"}
Sep 10 21:04:47.252658 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:47.252623    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54c6c477f9-4bk2f" event={"ID":"491006ce-4acf-49b3-9a19-210ed7756177","Type":"ContainerStarted","Data":"729d33849ea43ded61e450dc388b0bb6061e14489a385c0d5553d6bac4f54fcb"}
Sep 10 21:04:47.271534 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:47.271487    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54c6c477f9-4bk2f" podStartSLOduration=1.65493932 podStartE2EDuration="4.271469427s" totalImagesPullingTime="2.616530107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:04:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:04:47.270078024 +0000 UTC m=+346.935102825" watchObservedRunningTime="2026-09-10 21:04:47.271469427 +0000 UTC m=+346.936494228"
Sep 10 21:04:49.357843 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:49.357809    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-x66cz"]
Sep 10 21:04:49.361649 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:49.361625    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-x66cz"
Sep 10 21:04:49.363668 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:49.363642    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 10 21:04:49.363812 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:49.363791    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-tggrr\""
Sep 10 21:04:49.370482 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:49.370457    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-x66cz"]
Sep 10 21:04:49.497451 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:49.493727    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ffde7537-c081-431e-a26a-f10f66c20b60-cert\") pod \"odh-model-controller-7998c67dc4-x66cz\" (UID: \"ffde7537-c081-431e-a26a-f10f66c20b60\") " pod="opendatahub/odh-model-controller-7998c67dc4-x66cz"
Sep 10 21:04:49.497451 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:49.493856    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj6n9\" (UniqueName: \"kubernetes.io/projected/ffde7537-c081-431e-a26a-f10f66c20b60-kube-api-access-xj6n9\") pod \"odh-model-controller-7998c67dc4-x66cz\" (UID: \"ffde7537-c081-431e-a26a-f10f66c20b60\") " pod="opendatahub/odh-model-controller-7998c67dc4-x66cz"
Sep 10 21:04:49.595229 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:49.595192    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj6n9\" (UniqueName: \"kubernetes.io/projected/ffde7537-c081-431e-a26a-f10f66c20b60-kube-api-access-xj6n9\") pod \"odh-model-controller-7998c67dc4-x66cz\" (UID: \"ffde7537-c081-431e-a26a-f10f66c20b60\") " pod="opendatahub/odh-model-controller-7998c67dc4-x66cz"
Sep 10 21:04:49.595376 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:49.595248    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ffde7537-c081-431e-a26a-f10f66c20b60-cert\") pod \"odh-model-controller-7998c67dc4-x66cz\" (UID: \"ffde7537-c081-431e-a26a-f10f66c20b60\") " pod="opendatahub/odh-model-controller-7998c67dc4-x66cz"
Sep 10 21:04:49.597519 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:49.597500    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ffde7537-c081-431e-a26a-f10f66c20b60-cert\") pod \"odh-model-controller-7998c67dc4-x66cz\" (UID: \"ffde7537-c081-431e-a26a-f10f66c20b60\") " pod="opendatahub/odh-model-controller-7998c67dc4-x66cz"
Sep 10 21:04:49.603251 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:49.603217    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj6n9\" (UniqueName: \"kubernetes.io/projected/ffde7537-c081-431e-a26a-f10f66c20b60-kube-api-access-xj6n9\") pod \"odh-model-controller-7998c67dc4-x66cz\" (UID: \"ffde7537-c081-431e-a26a-f10f66c20b60\") " pod="opendatahub/odh-model-controller-7998c67dc4-x66cz"
Sep 10 21:04:49.674741 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:49.674706    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-x66cz"
Sep 10 21:04:49.949364 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:49.949334    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-x66cz"]
Sep 10 21:04:49.951270 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:04:49.951234    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffde7537_c081_431e_a26a_f10f66c20b60.slice/crio-d08dedbe1b967999205dac9921500b4684137f3060607cce6cac477cab40d706 WatchSource:0}: Error finding container d08dedbe1b967999205dac9921500b4684137f3060607cce6cac477cab40d706: Status 404 returned error can't find the container with id d08dedbe1b967999205dac9921500b4684137f3060607cce6cac477cab40d706
Sep 10 21:04:50.199209 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.199126    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"]
Sep 10 21:04:50.203493 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.203470    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.205659 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.205632    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-dhfrz\""
Sep 10 21:04:50.205798 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.205658    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 21:04:50.214745 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.214680    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"]
Sep 10 21:04:50.265365 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.265324    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-x66cz" event={"ID":"ffde7537-c081-431e-a26a-f10f66c20b60","Type":"ContainerStarted","Data":"d08dedbe1b967999205dac9921500b4684137f3060607cce6cac477cab40d706"}
Sep 10 21:04:50.301399 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.301375    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19306155-ec71-4c18-9e12-a6afe565b642-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.301544 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.301408    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19306155-ec71-4c18-9e12-a6afe565b642-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.301544 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.301454    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19306155-ec71-4c18-9e12-a6afe565b642-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.301626 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.301568    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19306155-ec71-4c18-9e12-a6afe565b642-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.301660 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.301639    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19306155-ec71-4c18-9e12-a6afe565b642-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.301693 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.301661    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19306155-ec71-4c18-9e12-a6afe565b642-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.301693 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.301677    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19306155-ec71-4c18-9e12-a6afe565b642-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.301758 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.301731    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19306155-ec71-4c18-9e12-a6afe565b642-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.301758 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.301749    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hk2j\" (UniqueName: \"kubernetes.io/projected/19306155-ec71-4c18-9e12-a6afe565b642-kube-api-access-7hk2j\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.402696 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.402658    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19306155-ec71-4c18-9e12-a6afe565b642-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.403179 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.402708    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19306155-ec71-4c18-9e12-a6afe565b642-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.403179 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.402743    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19306155-ec71-4c18-9e12-a6afe565b642-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.403179 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.402810    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19306155-ec71-4c18-9e12-a6afe565b642-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.403179 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.402878    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19306155-ec71-4c18-9e12-a6afe565b642-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.403179 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.402924    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19306155-ec71-4c18-9e12-a6afe565b642-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.403179 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.402950    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19306155-ec71-4c18-9e12-a6afe565b642-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.403179 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.402999    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19306155-ec71-4c18-9e12-a6afe565b642-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.403551 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.403249    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19306155-ec71-4c18-9e12-a6afe565b642-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.403551 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.403310    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hk2j\" (UniqueName: \"kubernetes.io/projected/19306155-ec71-4c18-9e12-a6afe565b642-kube-api-access-7hk2j\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.403721 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.403696    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19306155-ec71-4c18-9e12-a6afe565b642-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.403790 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.403698    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19306155-ec71-4c18-9e12-a6afe565b642-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.403790 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.403769    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19306155-ec71-4c18-9e12-a6afe565b642-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.403899 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.403844    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19306155-ec71-4c18-9e12-a6afe565b642-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.405104 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.405083    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19306155-ec71-4c18-9e12-a6afe565b642-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.405802 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.405780    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19306155-ec71-4c18-9e12-a6afe565b642-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.410508 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.410480    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19306155-ec71-4c18-9e12-a6afe565b642-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.410771 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.410751    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hk2j\" (UniqueName: \"kubernetes.io/projected/19306155-ec71-4c18-9e12-a6afe565b642-kube-api-access-7hk2j\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c994s2\" (UID: \"19306155-ec71-4c18-9e12-a6afe565b642\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.517845 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.517753    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:50.792244 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:50.792126    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"]
Sep 10 21:04:50.794485 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:04:50.794461    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19306155_ec71_4c18_9e12_a6afe565b642.slice/crio-bab4500e494841b4f7748774cb85d900161cb4a6c14e63fce4c8bbb26763a611 WatchSource:0}: Error finding container bab4500e494841b4f7748774cb85d900161cb4a6c14e63fce4c8bbb26763a611: Status 404 returned error can't find the container with id bab4500e494841b4f7748774cb85d900161cb4a6c14e63fce4c8bbb26763a611
Sep 10 21:04:51.273024 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:51.272974    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2" event={"ID":"19306155-ec71-4c18-9e12-a6afe565b642","Type":"ContainerStarted","Data":"bab4500e494841b4f7748774cb85d900161cb4a6c14e63fce4c8bbb26763a611"}
Sep 10 21:04:53.289505 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:53.288974    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-x66cz" event={"ID":"ffde7537-c081-431e-a26a-f10f66c20b60","Type":"ContainerStarted","Data":"2e41be067f1e4040c5472e8e5a50b7f790299af1c665f4fef5351c37aeb16654"}
Sep 10 21:04:53.289947 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:53.289578    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-x66cz"
Sep 10 21:04:53.310972 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:53.310740    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-x66cz" podStartSLOduration=1.2482121529999999 podStartE2EDuration="4.310716607s" totalImagesPullingTime="3.062504454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:04:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:04:53.306320892 +0000 UTC m=+352.971345693" watchObservedRunningTime="2026-09-10 21:04:53.310716607 +0000 UTC m=+352.975741407"
Sep 10 21:04:55.200813 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:55.200776    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:04:55.201193 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:55.200873    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:04:55.201193 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:55.200903    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:04:55.301502 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:55.301464    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2" event={"ID":"19306155-ec71-4c18-9e12-a6afe565b642","Type":"ContainerStarted","Data":"f4cb0fb9280c56e2daa767103dcafb416fd680bcf0da7902d04ead2b72d4fc81"}
Sep 10 21:04:55.323047 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:55.322988    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2" podStartSLOduration=0.918974081 podStartE2EDuration="5.322970772s" totalImagesPullingTime="4.403996691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:04:55.321351171 +0000 UTC m=+354.986375969" watchObservedRunningTime="2026-09-10 21:04:55.322970772 +0000 UTC m=+354.987995572"
Sep 10 21:04:55.518889 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:55.518797    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:56.523361 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:56.523328    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:56.951597 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:56.951561    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-548fd67646-ldmh4"]
Sep 10 21:04:56.954788 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:56.954770    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-548fd67646-ldmh4"
Sep 10 21:04:56.957617 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:56.957588    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 21:04:56.957752 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:56.957665    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 21:04:56.957752 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:56.957680    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 21:04:56.957752 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:56.957702    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-98fwg\""
Sep 10 21:04:56.957903 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:56.957750    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 21:04:56.958136 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:56.958102    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 21:04:56.961974 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:56.961621    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8rzj\" (UniqueName: \"kubernetes.io/projected/bc305b9b-0ab4-4760-a348-e17f66bd10b4-kube-api-access-z8rzj\") pod \"lws-controller-manager-548fd67646-ldmh4\" (UID: \"bc305b9b-0ab4-4760-a348-e17f66bd10b4\") " pod="openshift-lws-operator/lws-controller-manager-548fd67646-ldmh4"
Sep 10 21:04:56.961974 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:56.961719    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bc305b9b-0ab4-4760-a348-e17f66bd10b4-manager-config\") pod \"lws-controller-manager-548fd67646-ldmh4\" (UID: \"bc305b9b-0ab4-4760-a348-e17f66bd10b4\") " pod="openshift-lws-operator/lws-controller-manager-548fd67646-ldmh4"
Sep 10 21:04:56.961974 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:56.961807    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc305b9b-0ab4-4760-a348-e17f66bd10b4-metrics-cert\") pod \"lws-controller-manager-548fd67646-ldmh4\" (UID: \"bc305b9b-0ab4-4760-a348-e17f66bd10b4\") " pod="openshift-lws-operator/lws-controller-manager-548fd67646-ldmh4"
Sep 10 21:04:56.961974 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:56.961879    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc305b9b-0ab4-4760-a348-e17f66bd10b4-cert\") pod \"lws-controller-manager-548fd67646-ldmh4\" (UID: \"bc305b9b-0ab4-4760-a348-e17f66bd10b4\") " pod="openshift-lws-operator/lws-controller-manager-548fd67646-ldmh4"
Sep 10 21:04:56.963002 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:56.962982    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-548fd67646-ldmh4"]
Sep 10 21:04:57.062418 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:57.062387    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8rzj\" (UniqueName: \"kubernetes.io/projected/bc305b9b-0ab4-4760-a348-e17f66bd10b4-kube-api-access-z8rzj\") pod \"lws-controller-manager-548fd67646-ldmh4\" (UID: \"bc305b9b-0ab4-4760-a348-e17f66bd10b4\") " pod="openshift-lws-operator/lws-controller-manager-548fd67646-ldmh4"
Sep 10 21:04:57.062585 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:57.062451    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bc305b9b-0ab4-4760-a348-e17f66bd10b4-manager-config\") pod \"lws-controller-manager-548fd67646-ldmh4\" (UID: \"bc305b9b-0ab4-4760-a348-e17f66bd10b4\") " pod="openshift-lws-operator/lws-controller-manager-548fd67646-ldmh4"
Sep 10 21:04:57.062585 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:57.062489    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc305b9b-0ab4-4760-a348-e17f66bd10b4-metrics-cert\") pod \"lws-controller-manager-548fd67646-ldmh4\" (UID: \"bc305b9b-0ab4-4760-a348-e17f66bd10b4\") " pod="openshift-lws-operator/lws-controller-manager-548fd67646-ldmh4"
Sep 10 21:04:57.062585 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:57.062527    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc305b9b-0ab4-4760-a348-e17f66bd10b4-cert\") pod \"lws-controller-manager-548fd67646-ldmh4\" (UID: \"bc305b9b-0ab4-4760-a348-e17f66bd10b4\") " pod="openshift-lws-operator/lws-controller-manager-548fd67646-ldmh4"
Sep 10 21:04:57.063104 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:57.063074    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bc305b9b-0ab4-4760-a348-e17f66bd10b4-manager-config\") pod \"lws-controller-manager-548fd67646-ldmh4\" (UID: \"bc305b9b-0ab4-4760-a348-e17f66bd10b4\") " pod="openshift-lws-operator/lws-controller-manager-548fd67646-ldmh4"
Sep 10 21:04:57.065050 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:57.065027    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc305b9b-0ab4-4760-a348-e17f66bd10b4-cert\") pod \"lws-controller-manager-548fd67646-ldmh4\" (UID: \"bc305b9b-0ab4-4760-a348-e17f66bd10b4\") " pod="openshift-lws-operator/lws-controller-manager-548fd67646-ldmh4"
Sep 10 21:04:57.065120 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:57.065086    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc305b9b-0ab4-4760-a348-e17f66bd10b4-metrics-cert\") pod \"lws-controller-manager-548fd67646-ldmh4\" (UID: \"bc305b9b-0ab4-4760-a348-e17f66bd10b4\") " pod="openshift-lws-operator/lws-controller-manager-548fd67646-ldmh4"
Sep 10 21:04:57.070968 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:57.070946    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8rzj\" (UniqueName: \"kubernetes.io/projected/bc305b9b-0ab4-4760-a348-e17f66bd10b4-kube-api-access-z8rzj\") pod \"lws-controller-manager-548fd67646-ldmh4\" (UID: \"bc305b9b-0ab4-4760-a348-e17f66bd10b4\") " pod="openshift-lws-operator/lws-controller-manager-548fd67646-ldmh4"
Sep 10 21:04:57.264537 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:57.264441    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-548fd67646-ldmh4"
Sep 10 21:04:57.310176 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:57.310144    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:57.311239 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:57.311211    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c994s2"
Sep 10 21:04:57.540792 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:57.540588    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-548fd67646-ldmh4"]
Sep 10 21:04:57.543348 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:04:57.543321    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc305b9b_0ab4_4760_a348_e17f66bd10b4.slice/crio-9baca3d336fff33da674405fc21e525bb3635ecc9f1f259b63464079fd73ff3f WatchSource:0}: Error finding container 9baca3d336fff33da674405fc21e525bb3635ecc9f1f259b63464079fd73ff3f: Status 404 returned error can't find the container with id 9baca3d336fff33da674405fc21e525bb3635ecc9f1f259b63464079fd73ff3f
Sep 10 21:04:58.317786 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:04:58.317750    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-548fd67646-ldmh4" event={"ID":"bc305b9b-0ab4-4760-a348-e17f66bd10b4","Type":"ContainerStarted","Data":"9baca3d336fff33da674405fc21e525bb3635ecc9f1f259b63464079fd73ff3f"}
Sep 10 21:05:00.328918 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:05:00.328831    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-548fd67646-ldmh4" event={"ID":"bc305b9b-0ab4-4760-a348-e17f66bd10b4","Type":"ContainerStarted","Data":"d8350b84cf94287c3aebd678d5014eeb6e9958210b19ad7e96af62fcdc87e2c1"}
Sep 10 21:05:00.329353 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:05:00.328956    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-548fd67646-ldmh4"
Sep 10 21:05:00.346818 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:05:00.346762    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-548fd67646-ldmh4" podStartSLOduration=1.959283369 podStartE2EDuration="4.346723521s" totalImagesPullingTime="2.387440152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:04:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:05:00.346337205 +0000 UTC m=+360.011362004" watchObservedRunningTime="2026-09-10 21:05:00.346723521 +0000 UTC m=+360.011748326"
Sep 10 21:05:04.296883 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:05:04.296851    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-x66cz"
Sep 10 21:05:11.335876 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:05:11.335840    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-548fd67646-ldmh4"
Sep 10 21:06:21.583481 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:06:21.583444    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-c7qg4"]
Sep 10 21:06:21.586484 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:06:21.586461    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-c7qg4"
Sep 10 21:06:21.588771 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:06:21.588747    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cn77b\""
Sep 10 21:06:21.588875 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:06:21.588863    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 10 21:06:21.588926 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:06:21.588886    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 10 21:06:21.589404 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:06:21.589390    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 21:06:21.589498 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:06:21.589392    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 21:06:21.599457 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:06:21.599415    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-c7qg4"]
Sep 10 21:06:21.682618 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:06:21.682579    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d9094ca4-097a-4074-8753-05239f5fc252-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-c7qg4\" (UID: \"d9094ca4-097a-4074-8753-05239f5fc252\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-c7qg4"
Sep 10 21:06:21.682618 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:06:21.682635    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9094ca4-097a-4074-8753-05239f5fc252-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-c7qg4\" (UID: \"d9094ca4-097a-4074-8753-05239f5fc252\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-c7qg4"
Sep 10 21:06:21.682837 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:06:21.682657    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9dmg\" (UniqueName: \"kubernetes.io/projected/d9094ca4-097a-4074-8753-05239f5fc252-kube-api-access-s9dmg\") pod \"kuadrant-console-plugin-fbdd4f445-c7qg4\" (UID: \"d9094ca4-097a-4074-8753-05239f5fc252\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-c7qg4"
Sep 10 21:06:21.783380 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:06:21.783344    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d9094ca4-097a-4074-8753-05239f5fc252-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-c7qg4\" (UID: \"d9094ca4-097a-4074-8753-05239f5fc252\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-c7qg4"
Sep 10 21:06:21.783585 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:06:21.783399    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9094ca4-097a-4074-8753-05239f5fc252-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-c7qg4\" (UID: \"d9094ca4-097a-4074-8753-05239f5fc252\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-c7qg4"
Sep 10 21:06:21.783585 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:06:21.783450    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9dmg\" (UniqueName: \"kubernetes.io/projected/d9094ca4-097a-4074-8753-05239f5fc252-kube-api-access-s9dmg\") pod \"kuadrant-console-plugin-fbdd4f445-c7qg4\" (UID: \"d9094ca4-097a-4074-8753-05239f5fc252\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-c7qg4"
Sep 10 21:06:21.783955 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:06:21.783931    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d9094ca4-097a-4074-8753-05239f5fc252-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-c7qg4\" (UID: \"d9094ca4-097a-4074-8753-05239f5fc252\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-c7qg4"
Sep 10 21:06:21.785975 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:06:21.785954    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9094ca4-097a-4074-8753-05239f5fc252-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-c7qg4\" (UID: \"d9094ca4-097a-4074-8753-05239f5fc252\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-c7qg4"
Sep 10 21:06:21.791214 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:06:21.791195    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9dmg\" (UniqueName: \"kubernetes.io/projected/d9094ca4-097a-4074-8753-05239f5fc252-kube-api-access-s9dmg\") pod \"kuadrant-console-plugin-fbdd4f445-c7qg4\" (UID: \"d9094ca4-097a-4074-8753-05239f5fc252\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-c7qg4"
Sep 10 21:06:21.913297 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:06:21.913269    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-c7qg4"
Sep 10 21:06:22.184832 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:06:22.184758    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-c7qg4"]
Sep 10 21:06:22.188215 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:06:22.188178    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9094ca4_097a_4074_8753_05239f5fc252.slice/crio-9803cf0b6c191b4752f7918fde707c66dd839ed42951c7b4167f6afba340c0e1 WatchSource:0}: Error finding container 9803cf0b6c191b4752f7918fde707c66dd839ed42951c7b4167f6afba340c0e1: Status 404 returned error can't find the container with id 9803cf0b6c191b4752f7918fde707c66dd839ed42951c7b4167f6afba340c0e1
Sep 10 21:06:22.680992 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:06:22.680957    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-c7qg4" event={"ID":"d9094ca4-097a-4074-8753-05239f5fc252","Type":"ContainerStarted","Data":"9803cf0b6c191b4752f7918fde707c66dd839ed42951c7b4167f6afba340c0e1"}
Sep 10 21:06:28.712106 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:06:28.712066    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-c7qg4" event={"ID":"d9094ca4-097a-4074-8753-05239f5fc252","Type":"ContainerStarted","Data":"4bd724783fdf5fc68a78d5d57d26021de9728d1fd9cf5d6605888189694ce30d"}
Sep 10 21:06:28.728612 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:06:28.728561    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-c7qg4" podStartSLOduration=2.21969238 podStartE2EDuration="7.728547456s" totalImagesPullingTime="5.508855076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:06:28.727115071 +0000 UTC m=+448.392139870" watchObservedRunningTime="2026-09-10 21:06:28.728547456 +0000 UTC m=+448.393572256"
Sep 10 21:07:04.547824 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.547782    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"]
Sep 10 21:07:04.551072 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.551048    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.553113 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.553092    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-c9mj9\""
Sep 10 21:07:04.562522 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.562494    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"]
Sep 10 21:07:04.631720 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.631673    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.631913 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.631729    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcrtk\" (UniqueName: \"kubernetes.io/projected/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-kube-api-access-lcrtk\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.631913 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.631772    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.631913 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.631798    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.632084 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.631981    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.632084 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.632009    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.632084 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.632036    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.632084 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.632063    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.632246 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.632085    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.733587 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.733553    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.733587 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.733588    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.733829 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.733609    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.733829 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.733624    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.733829 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.733639    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.733829 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.733672    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.733829 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.733696    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcrtk\" (UniqueName: \"kubernetes.io/projected/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-kube-api-access-lcrtk\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.733829 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.733721    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.733829 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.733749    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.734159 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.734027    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.734159 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.734077    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.734159 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.734087    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.734252 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.734232    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.734650 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.734631    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.736033 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.736006    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.736129 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.736073    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.741822 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.741798    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcrtk\" (UniqueName: \"kubernetes.io/projected/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-kube-api-access-lcrtk\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.741899 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.741868    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c45383bb-f8d6-4467-8aa4-af2e1a9e18fc-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-wqb75\" (UID: \"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:04.861217 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:04.861142    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:05.138750 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:05.138722    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"]
Sep 10 21:07:05.141935 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:07:05.141908    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45383bb_f8d6_4467_8aa4_af2e1a9e18fc.slice/crio-6490d772966cc03cb2ea8b0b58603baf9d0854412514987e786b047031cd2993 WatchSource:0}: Error finding container 6490d772966cc03cb2ea8b0b58603baf9d0854412514987e786b047031cd2993: Status 404 returned error can't find the container with id 6490d772966cc03cb2ea8b0b58603baf9d0854412514987e786b047031cd2993
Sep 10 21:07:05.144716 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:05.144677    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:07:05.144818 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:05.144749    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:07:05.144818 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:05.144781    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:07:05.865746 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:05.865708    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75" event={"ID":"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc","Type":"ContainerStarted","Data":"98f148da17e767dd929b2ea7b33c2d027651a27374213f9063bd2bdf0df4818c"}
Sep 10 21:07:05.866143 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:05.865764    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75" event={"ID":"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc","Type":"ContainerStarted","Data":"6490d772966cc03cb2ea8b0b58603baf9d0854412514987e786b047031cd2993"}
Sep 10 21:07:05.896152 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:05.896094    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75" podStartSLOduration=1.896075417 podStartE2EDuration="1.896075417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:07:05.893355568 +0000 UTC m=+485.558380368" watchObservedRunningTime="2026-09-10 21:07:05.896075417 +0000 UTC m=+485.561100216"
Sep 10 21:07:06.861671 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:06.861639    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:06.866566 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:06.866545    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:06.870039 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:06.870013    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:06.870767 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:06.870747    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:07:22.198692 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:22.198657    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-sqx5t"]
Sep 10 21:07:22.201982 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:22.201966    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-sqx5t"
Sep 10 21:07:22.204369 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:22.204347    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-29nr7\""
Sep 10 21:07:22.208924 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:22.208899    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-sqx5t"]
Sep 10 21:07:22.296102 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:22.296069    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkl56\" (UniqueName: \"kubernetes.io/projected/07e5ab7c-b35c-4729-8a0a-d09f96b2a31b-kube-api-access-fkl56\") pod \"authorino-7bcfd8f9cd-sqx5t\" (UID: \"07e5ab7c-b35c-4729-8a0a-d09f96b2a31b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-sqx5t"
Sep 10 21:07:22.397127 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:22.397091    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkl56\" (UniqueName: \"kubernetes.io/projected/07e5ab7c-b35c-4729-8a0a-d09f96b2a31b-kube-api-access-fkl56\") pod \"authorino-7bcfd8f9cd-sqx5t\" (UID: \"07e5ab7c-b35c-4729-8a0a-d09f96b2a31b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-sqx5t"
Sep 10 21:07:22.405309 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:22.405282    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkl56\" (UniqueName: \"kubernetes.io/projected/07e5ab7c-b35c-4729-8a0a-d09f96b2a31b-kube-api-access-fkl56\") pod \"authorino-7bcfd8f9cd-sqx5t\" (UID: \"07e5ab7c-b35c-4729-8a0a-d09f96b2a31b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-sqx5t"
Sep 10 21:07:22.512282 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:22.512188    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-sqx5t"
Sep 10 21:07:22.778602 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:22.778527    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-sqx5t"]
Sep 10 21:07:22.781901 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:07:22.781876    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07e5ab7c_b35c_4729_8a0a_d09f96b2a31b.slice/crio-dcf23758dc6a8ad23faaf8f61bbc7501cd70947cb0923ceb4cfbac6c248c2f21 WatchSource:0}: Error finding container dcf23758dc6a8ad23faaf8f61bbc7501cd70947cb0923ceb4cfbac6c248c2f21: Status 404 returned error can't find the container with id dcf23758dc6a8ad23faaf8f61bbc7501cd70947cb0923ceb4cfbac6c248c2f21
Sep 10 21:07:22.938497 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:22.938461    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-sqx5t" event={"ID":"07e5ab7c-b35c-4729-8a0a-d09f96b2a31b","Type":"ContainerStarted","Data":"dcf23758dc6a8ad23faaf8f61bbc7501cd70947cb0923ceb4cfbac6c248c2f21"}
Sep 10 21:07:24.954005 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:24.953977    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-sqx5t"]
Sep 10 21:07:25.963750 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:25.963713    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-sqx5t" event={"ID":"07e5ab7c-b35c-4729-8a0a-d09f96b2a31b","Type":"ContainerStarted","Data":"37d8cb0afc7b9a60c5f44d5ea0dfabe48c794e738b40dbc95bfd9367a63b19a4"}
Sep 10 21:07:25.964205 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:25.963810    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-sqx5t" podUID="07e5ab7c-b35c-4729-8a0a-d09f96b2a31b" containerName="authorino" containerID="cri-o://37d8cb0afc7b9a60c5f44d5ea0dfabe48c794e738b40dbc95bfd9367a63b19a4" gracePeriod=30
Sep 10 21:07:25.979517 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:25.979474    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-sqx5t" podStartSLOduration=1.483632527 podStartE2EDuration="3.979458591s" totalImagesPullingTime="2.495826064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:07:25.977583937 +0000 UTC m=+505.642608738" watchObservedRunningTime="2026-09-10 21:07:25.979458591 +0000 UTC m=+505.644483391"
Sep 10 21:07:26.212843 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:26.212820    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-sqx5t"
Sep 10 21:07:26.336197 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:26.336110    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkl56\" (UniqueName: \"kubernetes.io/projected/07e5ab7c-b35c-4729-8a0a-d09f96b2a31b-kube-api-access-fkl56\") pod \"07e5ab7c-b35c-4729-8a0a-d09f96b2a31b\" (UID: \"07e5ab7c-b35c-4729-8a0a-d09f96b2a31b\") "
Sep 10 21:07:26.338770 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:26.338732    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e5ab7c-b35c-4729-8a0a-d09f96b2a31b-kube-api-access-fkl56" (OuterVolumeSpecName: "kube-api-access-fkl56") pod "07e5ab7c-b35c-4729-8a0a-d09f96b2a31b" (UID: "07e5ab7c-b35c-4729-8a0a-d09f96b2a31b"). InnerVolumeSpecName "kube-api-access-fkl56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:07:26.437594 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:26.437560    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkl56\" (UniqueName: \"kubernetes.io/projected/07e5ab7c-b35c-4729-8a0a-d09f96b2a31b-kube-api-access-fkl56\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:07:26.969617 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:26.969584    2584 generic.go:356] "Generic (PLEG): container finished" podID="07e5ab7c-b35c-4729-8a0a-d09f96b2a31b" containerID="37d8cb0afc7b9a60c5f44d5ea0dfabe48c794e738b40dbc95bfd9367a63b19a4" exitCode=0
Sep 10 21:07:26.970014 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:26.969628    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-sqx5t"
Sep 10 21:07:26.970014 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:26.969674    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-sqx5t" event={"ID":"07e5ab7c-b35c-4729-8a0a-d09f96b2a31b","Type":"ContainerDied","Data":"37d8cb0afc7b9a60c5f44d5ea0dfabe48c794e738b40dbc95bfd9367a63b19a4"}
Sep 10 21:07:26.970014 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:26.969720    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-sqx5t" event={"ID":"07e5ab7c-b35c-4729-8a0a-d09f96b2a31b","Type":"ContainerDied","Data":"dcf23758dc6a8ad23faaf8f61bbc7501cd70947cb0923ceb4cfbac6c248c2f21"}
Sep 10 21:07:26.970014 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:26.969738    2584 scope.go:117] "RemoveContainer" containerID="37d8cb0afc7b9a60c5f44d5ea0dfabe48c794e738b40dbc95bfd9367a63b19a4"
Sep 10 21:07:26.978212 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:26.978195    2584 scope.go:117] "RemoveContainer" containerID="37d8cb0afc7b9a60c5f44d5ea0dfabe48c794e738b40dbc95bfd9367a63b19a4"
Sep 10 21:07:26.978485 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:07:26.978454    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d8cb0afc7b9a60c5f44d5ea0dfabe48c794e738b40dbc95bfd9367a63b19a4\": container with ID starting with 37d8cb0afc7b9a60c5f44d5ea0dfabe48c794e738b40dbc95bfd9367a63b19a4 not found: ID does not exist" containerID="37d8cb0afc7b9a60c5f44d5ea0dfabe48c794e738b40dbc95bfd9367a63b19a4"
Sep 10 21:07:26.978577 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:26.978491    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d8cb0afc7b9a60c5f44d5ea0dfabe48c794e738b40dbc95bfd9367a63b19a4"} err="failed to get container status \"37d8cb0afc7b9a60c5f44d5ea0dfabe48c794e738b40dbc95bfd9367a63b19a4\": rpc error: code = NotFound desc = could not find container \"37d8cb0afc7b9a60c5f44d5ea0dfabe48c794e738b40dbc95bfd9367a63b19a4\": container with ID starting with 37d8cb0afc7b9a60c5f44d5ea0dfabe48c794e738b40dbc95bfd9367a63b19a4 not found: ID does not exist"
Sep 10 21:07:26.986064 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:26.986040    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-sqx5t"]
Sep 10 21:07:26.989712 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:26.989686    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-sqx5t"]
Sep 10 21:07:28.839414 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:28.839369    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07e5ab7c-b35c-4729-8a0a-d09f96b2a31b" path="/var/lib/kubelet/pods/07e5ab7c-b35c-4729-8a0a-d09f96b2a31b/volumes"
Sep 10 21:07:56.676166 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:56.676079    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-wgb7w"]
Sep 10 21:07:56.676678 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:56.676591    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07e5ab7c-b35c-4729-8a0a-d09f96b2a31b" containerName="authorino"
Sep 10 21:07:56.676678 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:56.676610    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="07e5ab7c-b35c-4729-8a0a-d09f96b2a31b" containerName="authorino"
Sep 10 21:07:56.676847 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:56.676718    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="07e5ab7c-b35c-4729-8a0a-d09f96b2a31b" containerName="authorino"
Sep 10 21:07:56.679609 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:56.679589    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-wgb7w"
Sep 10 21:07:56.681945 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:56.681928    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-29nr7\""
Sep 10 21:07:56.688027 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:56.688007    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-wgb7w"]
Sep 10 21:07:56.793683 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:56.793655    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vsp7\" (UniqueName: \"kubernetes.io/projected/98409c54-4a47-4b46-8ed2-951c3e43790a-kube-api-access-7vsp7\") pod \"authorino-557f876f69-wgb7w\" (UID: \"98409c54-4a47-4b46-8ed2-951c3e43790a\") " pod="kuadrant-system/authorino-557f876f69-wgb7w"
Sep 10 21:07:56.802452 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:56.802413    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-wgb7w"]
Sep 10 21:07:56.802647 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:07:56.802626    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-7vsp7], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-wgb7w" podUID="98409c54-4a47-4b46-8ed2-951c3e43790a"
Sep 10 21:07:56.894949 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:56.894914    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vsp7\" (UniqueName: \"kubernetes.io/projected/98409c54-4a47-4b46-8ed2-951c3e43790a-kube-api-access-7vsp7\") pod \"authorino-557f876f69-wgb7w\" (UID: \"98409c54-4a47-4b46-8ed2-951c3e43790a\") " pod="kuadrant-system/authorino-557f876f69-wgb7w"
Sep 10 21:07:56.907116 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:56.907088    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vsp7\" (UniqueName: \"kubernetes.io/projected/98409c54-4a47-4b46-8ed2-951c3e43790a-kube-api-access-7vsp7\") pod \"authorino-557f876f69-wgb7w\" (UID: \"98409c54-4a47-4b46-8ed2-951c3e43790a\") " pod="kuadrant-system/authorino-557f876f69-wgb7w"
Sep 10 21:07:56.951542 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:56.951477    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6b558ff6f7-krwnj"]
Sep 10 21:07:56.954701 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:56.954686    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b558ff6f7-krwnj"
Sep 10 21:07:56.957104 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:56.957084    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 21:07:56.964805 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:56.964784    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b558ff6f7-krwnj"]
Sep 10 21:07:56.996000 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:56.995967    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2mt4\" (UniqueName: \"kubernetes.io/projected/47a71315-a0b2-4c7f-9006-c9081a90a31a-kube-api-access-j2mt4\") pod \"authorino-6b558ff6f7-krwnj\" (UID: \"47a71315-a0b2-4c7f-9006-c9081a90a31a\") " pod="kuadrant-system/authorino-6b558ff6f7-krwnj"
Sep 10 21:07:56.996180 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:56.996052    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/47a71315-a0b2-4c7f-9006-c9081a90a31a-tls-cert\") pod \"authorino-6b558ff6f7-krwnj\" (UID: \"47a71315-a0b2-4c7f-9006-c9081a90a31a\") " pod="kuadrant-system/authorino-6b558ff6f7-krwnj"
Sep 10 21:07:57.096575 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:57.096547    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/47a71315-a0b2-4c7f-9006-c9081a90a31a-tls-cert\") pod \"authorino-6b558ff6f7-krwnj\" (UID: \"47a71315-a0b2-4c7f-9006-c9081a90a31a\") " pod="kuadrant-system/authorino-6b558ff6f7-krwnj"
Sep 10 21:07:57.096748 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:57.096619    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2mt4\" (UniqueName: \"kubernetes.io/projected/47a71315-a0b2-4c7f-9006-c9081a90a31a-kube-api-access-j2mt4\") pod \"authorino-6b558ff6f7-krwnj\" (UID: \"47a71315-a0b2-4c7f-9006-c9081a90a31a\") " pod="kuadrant-system/authorino-6b558ff6f7-krwnj"
Sep 10 21:07:57.098957 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:57.098935    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-wgb7w"
Sep 10 21:07:57.099061 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:57.099008    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/47a71315-a0b2-4c7f-9006-c9081a90a31a-tls-cert\") pod \"authorino-6b558ff6f7-krwnj\" (UID: \"47a71315-a0b2-4c7f-9006-c9081a90a31a\") " pod="kuadrant-system/authorino-6b558ff6f7-krwnj"
Sep 10 21:07:57.104680 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:57.104662    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2mt4\" (UniqueName: \"kubernetes.io/projected/47a71315-a0b2-4c7f-9006-c9081a90a31a-kube-api-access-j2mt4\") pod \"authorino-6b558ff6f7-krwnj\" (UID: \"47a71315-a0b2-4c7f-9006-c9081a90a31a\") " pod="kuadrant-system/authorino-6b558ff6f7-krwnj"
Sep 10 21:07:57.119136 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:57.119117    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-wgb7w"
Sep 10 21:07:57.197879 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:57.197851    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vsp7\" (UniqueName: \"kubernetes.io/projected/98409c54-4a47-4b46-8ed2-951c3e43790a-kube-api-access-7vsp7\") pod \"98409c54-4a47-4b46-8ed2-951c3e43790a\" (UID: \"98409c54-4a47-4b46-8ed2-951c3e43790a\") "
Sep 10 21:07:57.199846 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:57.199822    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98409c54-4a47-4b46-8ed2-951c3e43790a-kube-api-access-7vsp7" (OuterVolumeSpecName: "kube-api-access-7vsp7") pod "98409c54-4a47-4b46-8ed2-951c3e43790a" (UID: "98409c54-4a47-4b46-8ed2-951c3e43790a"). InnerVolumeSpecName "kube-api-access-7vsp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:07:57.264039 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:57.264005    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b558ff6f7-krwnj"
Sep 10 21:07:57.299563 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:57.299535    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vsp7\" (UniqueName: \"kubernetes.io/projected/98409c54-4a47-4b46-8ed2-951c3e43790a-kube-api-access-7vsp7\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:07:57.536475 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:57.536260    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b558ff6f7-krwnj"]
Sep 10 21:07:57.539062 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:07:57.539033    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47a71315_a0b2_4c7f_9006_c9081a90a31a.slice/crio-8930185d4d2527d0146e57eeec199ff9e5d741b6192f094ca13610432da4c427 WatchSource:0}: Error finding container 8930185d4d2527d0146e57eeec199ff9e5d741b6192f094ca13610432da4c427: Status 404 returned error can't find the container with id 8930185d4d2527d0146e57eeec199ff9e5d741b6192f094ca13610432da4c427
Sep 10 21:07:58.104375 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:58.104345    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-wgb7w"
Sep 10 21:07:58.104812 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:58.104380    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b558ff6f7-krwnj" event={"ID":"47a71315-a0b2-4c7f-9006-c9081a90a31a","Type":"ContainerStarted","Data":"8930185d4d2527d0146e57eeec199ff9e5d741b6192f094ca13610432da4c427"}
Sep 10 21:07:58.135644 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:58.135611    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-wgb7w"]
Sep 10 21:07:58.139445 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:58.138658    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-wgb7w"]
Sep 10 21:07:58.839911 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:58.839878    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98409c54-4a47-4b46-8ed2-951c3e43790a" path="/var/lib/kubelet/pods/98409c54-4a47-4b46-8ed2-951c3e43790a/volumes"
Sep 10 21:07:59.111372 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:59.111283    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b558ff6f7-krwnj" event={"ID":"47a71315-a0b2-4c7f-9006-c9081a90a31a","Type":"ContainerStarted","Data":"e796f405d52c3d119243fdd79393e8097abbfc44bf897d9243790e939d32d9a3"}
Sep 10 21:07:59.138209 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:07:59.138161    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6b558ff6f7-krwnj" podStartSLOduration=2.392288103 podStartE2EDuration="3.138146197s" totalImagesPullingTime="745.858094ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:07:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:07:59.137082076 +0000 UTC m=+538.802106876" watchObservedRunningTime="2026-09-10 21:07:59.138146197 +0000 UTC m=+538.803170996"
Sep 10 21:08:52.911623 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:08:52.911577    2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-wqb75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 21:08:52.912068 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:08:52.911676    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75" podUID="c45383bb-f8d6-4467-8aa4-af2e1a9e18fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 21:08:53.362489 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:08:53.362397    2584 generic.go:356] "Generic (PLEG): container finished" podID="c45383bb-f8d6-4467-8aa4-af2e1a9e18fc" containerID="98f148da17e767dd929b2ea7b33c2d027651a27374213f9063bd2bdf0df4818c" exitCode=0
Sep 10 21:08:53.362489 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:08:53.362467    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75" event={"ID":"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc","Type":"ContainerDied","Data":"98f148da17e767dd929b2ea7b33c2d027651a27374213f9063bd2bdf0df4818c"}
Sep 10 21:08:53.362870 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:08:53.362857    2584 scope.go:117] "RemoveContainer" containerID="98f148da17e767dd929b2ea7b33c2d027651a27374213f9063bd2bdf0df4818c"
Sep 10 21:08:53.363888 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:08:53.363864    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:08:53.363966 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:08:53.363954    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:08:53.364005 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:08:53.363989    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:08:54.368534 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:08:54.368498    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75" event={"ID":"c45383bb-f8d6-4467-8aa4-af2e1a9e18fc","Type":"ContainerStarted","Data":"4d3f35128a5a0cc32ff24a05464ce4053fa918cc98ad94d7a8b0c8cac0279b6d"}
Sep 10 21:08:54.861361 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:08:54.861324    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:08:54.861585 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:08:54.861554    2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-wqb75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 10 21:08:54.861667 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:08:54.861624    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75" podUID="c45383bb-f8d6-4467-8aa4-af2e1a9e18fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 10 21:08:55.862232 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:08:55.862196    2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-wqb75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 10 21:08:55.862692 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:08:55.862256    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75" podUID="c45383bb-f8d6-4467-8aa4-af2e1a9e18fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 10 21:08:56.861545 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:08:56.861514    2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-wqb75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 10 21:08:56.861719 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:08:56.861571    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75" podUID="c45383bb-f8d6-4467-8aa4-af2e1a9e18fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 10 21:08:57.861535 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:08:57.861501    2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-wqb75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 10 21:08:57.861909 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:08:57.861567    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75" podUID="c45383bb-f8d6-4467-8aa4-af2e1a9e18fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 10 21:08:58.861964 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:08:58.861926    2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-wqb75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 10 21:08:58.862327 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:08:58.861995    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75" podUID="c45383bb-f8d6-4467-8aa4-af2e1a9e18fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 10 21:08:59.865606 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:08:59.865569    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:08:59.866062 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:08:59.865951    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:08:59.866664 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:08:59.866641    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-wqb75"
Sep 10 21:09:00.749114 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:00.749087    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-jkkfm_fd81843e-6283-43e3-a571-d5e47be26e38/console-operator/1.log"
Sep 10 21:09:00.749287 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:00.749126    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-jkkfm_fd81843e-6283-43e3-a571-d5e47be26e38/console-operator/1.log"
Sep 10 21:09:05.565134 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.565097    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"]
Sep 10 21:09:05.568645 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.568625    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"
Sep 10 21:09:05.571294 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.571274    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 21:09:05.571398 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.571274    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 21:09:05.571980 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.571967    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 10 21:09:05.572026 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.571965    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kxt89\""
Sep 10 21:09:05.578745 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.578720    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"]
Sep 10 21:09:05.696590 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.696548    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6cvn\" (UniqueName: \"kubernetes.io/projected/54f7bcbf-c532-44ad-9044-95f58c58a93d-kube-api-access-n6cvn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75\" (UID: \"54f7bcbf-c532-44ad-9044-95f58c58a93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"
Sep 10 21:09:05.696757 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.696614    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54f7bcbf-c532-44ad-9044-95f58c58a93d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75\" (UID: \"54f7bcbf-c532-44ad-9044-95f58c58a93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"
Sep 10 21:09:05.696757 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.696669    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54f7bcbf-c532-44ad-9044-95f58c58a93d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75\" (UID: \"54f7bcbf-c532-44ad-9044-95f58c58a93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"
Sep 10 21:09:05.696757 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.696689    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54f7bcbf-c532-44ad-9044-95f58c58a93d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75\" (UID: \"54f7bcbf-c532-44ad-9044-95f58c58a93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"
Sep 10 21:09:05.696757 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.696710    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f7bcbf-c532-44ad-9044-95f58c58a93d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75\" (UID: \"54f7bcbf-c532-44ad-9044-95f58c58a93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"
Sep 10 21:09:05.696757 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.696752    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54f7bcbf-c532-44ad-9044-95f58c58a93d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75\" (UID: \"54f7bcbf-c532-44ad-9044-95f58c58a93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"
Sep 10 21:09:05.798151 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.798096    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6cvn\" (UniqueName: \"kubernetes.io/projected/54f7bcbf-c532-44ad-9044-95f58c58a93d-kube-api-access-n6cvn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75\" (UID: \"54f7bcbf-c532-44ad-9044-95f58c58a93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"
Sep 10 21:09:05.798364 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.798166    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54f7bcbf-c532-44ad-9044-95f58c58a93d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75\" (UID: \"54f7bcbf-c532-44ad-9044-95f58c58a93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"
Sep 10 21:09:05.798364 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.798210    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54f7bcbf-c532-44ad-9044-95f58c58a93d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75\" (UID: \"54f7bcbf-c532-44ad-9044-95f58c58a93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"
Sep 10 21:09:05.798364 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.798226    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54f7bcbf-c532-44ad-9044-95f58c58a93d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75\" (UID: \"54f7bcbf-c532-44ad-9044-95f58c58a93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"
Sep 10 21:09:05.798364 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.798245    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f7bcbf-c532-44ad-9044-95f58c58a93d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75\" (UID: \"54f7bcbf-c532-44ad-9044-95f58c58a93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"
Sep 10 21:09:05.798614 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.798378    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54f7bcbf-c532-44ad-9044-95f58c58a93d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75\" (UID: \"54f7bcbf-c532-44ad-9044-95f58c58a93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"
Sep 10 21:09:05.798702 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.798679    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54f7bcbf-c532-44ad-9044-95f58c58a93d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75\" (UID: \"54f7bcbf-c532-44ad-9044-95f58c58a93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"
Sep 10 21:09:05.798761 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.798725    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54f7bcbf-c532-44ad-9044-95f58c58a93d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75\" (UID: \"54f7bcbf-c532-44ad-9044-95f58c58a93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"
Sep 10 21:09:05.798836 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.798818    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54f7bcbf-c532-44ad-9044-95f58c58a93d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75\" (UID: \"54f7bcbf-c532-44ad-9044-95f58c58a93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"
Sep 10 21:09:05.800675 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.800649    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54f7bcbf-c532-44ad-9044-95f58c58a93d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75\" (UID: \"54f7bcbf-c532-44ad-9044-95f58c58a93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"
Sep 10 21:09:05.800924 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.800906    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f7bcbf-c532-44ad-9044-95f58c58a93d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75\" (UID: \"54f7bcbf-c532-44ad-9044-95f58c58a93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"
Sep 10 21:09:05.807683 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.807652    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6cvn\" (UniqueName: \"kubernetes.io/projected/54f7bcbf-c532-44ad-9044-95f58c58a93d-kube-api-access-n6cvn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75\" (UID: \"54f7bcbf-c532-44ad-9044-95f58c58a93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"
Sep 10 21:09:05.878917 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:05.878883    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"
Sep 10 21:09:06.163735 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:06.163710    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"]
Sep 10 21:09:06.165875 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:09:06.165843    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f7bcbf_c532_44ad_9044_95f58c58a93d.slice/crio-b0c8eb280f4cee33ed796f9c624ccb2e1e9033e77231a55baff052676d8b7d98 WatchSource:0}: Error finding container b0c8eb280f4cee33ed796f9c624ccb2e1e9033e77231a55baff052676d8b7d98: Status 404 returned error can't find the container with id b0c8eb280f4cee33ed796f9c624ccb2e1e9033e77231a55baff052676d8b7d98
Sep 10 21:09:06.426811 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:06.426723    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75" event={"ID":"54f7bcbf-c532-44ad-9044-95f58c58a93d","Type":"ContainerStarted","Data":"b0c8eb280f4cee33ed796f9c624ccb2e1e9033e77231a55baff052676d8b7d98"}
Sep 10 21:09:07.433178 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:07.433136    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75" event={"ID":"54f7bcbf-c532-44ad-9044-95f58c58a93d","Type":"ContainerStarted","Data":"b5c64499e71a1ef4d71cfd9bd668b93a13cabbbf054709a467d88b2b58543bb2"}
Sep 10 21:09:07.451833 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:07.451785    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75" podStartSLOduration=1.368644902 podStartE2EDuration="2.451770609s" totalImagesPullingTime="1.083125707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:09:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:09:07.450088819 +0000 UTC m=+607.115113642" watchObservedRunningTime="2026-09-10 21:09:07.451770609 +0000 UTC m=+607.116795409"
Sep 10 21:09:15.880002 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:15.879956    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"
Sep 10 21:09:15.880527 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:15.880024    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"
Sep 10 21:09:15.892449 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:15.892409    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"
Sep 10 21:09:16.484848 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:16.484815    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75"
Sep 10 21:09:45.563019 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:45.562983    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"]
Sep 10 21:09:45.566318 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:45.566296    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"
Sep 10 21:09:45.568520 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:45.568498    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 10 21:09:45.577081 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:45.577060    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"]
Sep 10 21:09:45.646050 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:45.646019    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b21604b-6ee2-43f9-bf26-757932579b52-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b58zm\" (UID: \"2b21604b-6ee2-43f9-bf26-757932579b52\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"
Sep 10 21:09:45.646050 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:45.646053    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b21604b-6ee2-43f9-bf26-757932579b52-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b58zm\" (UID: \"2b21604b-6ee2-43f9-bf26-757932579b52\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"
Sep 10 21:09:45.646228 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:45.646072    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b21604b-6ee2-43f9-bf26-757932579b52-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b58zm\" (UID: \"2b21604b-6ee2-43f9-bf26-757932579b52\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"
Sep 10 21:09:45.646228 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:45.646144    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b21604b-6ee2-43f9-bf26-757932579b52-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b58zm\" (UID: \"2b21604b-6ee2-43f9-bf26-757932579b52\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"
Sep 10 21:09:45.646299 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:45.646230    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b21604b-6ee2-43f9-bf26-757932579b52-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b58zm\" (UID: \"2b21604b-6ee2-43f9-bf26-757932579b52\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"
Sep 10 21:09:45.646299 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:45.646259    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml7n7\" (UniqueName: \"kubernetes.io/projected/2b21604b-6ee2-43f9-bf26-757932579b52-kube-api-access-ml7n7\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b58zm\" (UID: \"2b21604b-6ee2-43f9-bf26-757932579b52\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"
Sep 10 21:09:45.747546 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:45.747515    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b21604b-6ee2-43f9-bf26-757932579b52-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b58zm\" (UID: \"2b21604b-6ee2-43f9-bf26-757932579b52\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"
Sep 10 21:09:45.747546 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:45.747547    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml7n7\" (UniqueName: \"kubernetes.io/projected/2b21604b-6ee2-43f9-bf26-757932579b52-kube-api-access-ml7n7\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b58zm\" (UID: \"2b21604b-6ee2-43f9-bf26-757932579b52\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"
Sep 10 21:09:45.747736 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:45.747569    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b21604b-6ee2-43f9-bf26-757932579b52-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b58zm\" (UID: \"2b21604b-6ee2-43f9-bf26-757932579b52\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"
Sep 10 21:09:45.747736 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:45.747688    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b21604b-6ee2-43f9-bf26-757932579b52-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b58zm\" (UID: \"2b21604b-6ee2-43f9-bf26-757932579b52\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"
Sep 10 21:09:45.747736 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:45.747725    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b21604b-6ee2-43f9-bf26-757932579b52-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b58zm\" (UID: \"2b21604b-6ee2-43f9-bf26-757932579b52\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"
Sep 10 21:09:45.747860 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:45.747786    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b21604b-6ee2-43f9-bf26-757932579b52-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b58zm\" (UID: \"2b21604b-6ee2-43f9-bf26-757932579b52\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"
Sep 10 21:09:45.747966 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:45.747946    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b21604b-6ee2-43f9-bf26-757932579b52-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b58zm\" (UID: \"2b21604b-6ee2-43f9-bf26-757932579b52\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"
Sep 10 21:09:45.748032 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:45.747975    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b21604b-6ee2-43f9-bf26-757932579b52-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b58zm\" (UID: \"2b21604b-6ee2-43f9-bf26-757932579b52\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"
Sep 10 21:09:45.748032 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:45.748013    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b21604b-6ee2-43f9-bf26-757932579b52-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b58zm\" (UID: \"2b21604b-6ee2-43f9-bf26-757932579b52\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"
Sep 10 21:09:45.749820 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:45.749796    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b21604b-6ee2-43f9-bf26-757932579b52-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b58zm\" (UID: \"2b21604b-6ee2-43f9-bf26-757932579b52\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"
Sep 10 21:09:45.750218 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:45.750194    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b21604b-6ee2-43f9-bf26-757932579b52-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b58zm\" (UID: \"2b21604b-6ee2-43f9-bf26-757932579b52\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"
Sep 10 21:09:45.756027 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:45.756003    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml7n7\" (UniqueName: \"kubernetes.io/projected/2b21604b-6ee2-43f9-bf26-757932579b52-kube-api-access-ml7n7\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b58zm\" (UID: \"2b21604b-6ee2-43f9-bf26-757932579b52\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"
Sep 10 21:09:45.876561 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:45.876532    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"
Sep 10 21:09:46.147536 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:46.147503    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"]
Sep 10 21:09:46.150255 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:09:46.150227    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b21604b_6ee2_43f9_bf26_757932579b52.slice/crio-5959fcd9efffeac8b1c34228576b3f48004c8674b0c9aa7f2451b9efc8bba0d3 WatchSource:0}: Error finding container 5959fcd9efffeac8b1c34228576b3f48004c8674b0c9aa7f2451b9efc8bba0d3: Status 404 returned error can't find the container with id 5959fcd9efffeac8b1c34228576b3f48004c8674b0c9aa7f2451b9efc8bba0d3
Sep 10 21:09:46.151756 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:46.151736    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:09:46.604599 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:46.604503    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm" event={"ID":"2b21604b-6ee2-43f9-bf26-757932579b52","Type":"ContainerStarted","Data":"19f723b47b2d3ba29830f628111ae72bf4b3d27f5bfb9dfbcb28d61c24cdb8c1"}
Sep 10 21:09:46.604599 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:46.604575    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm" event={"ID":"2b21604b-6ee2-43f9-bf26-757932579b52","Type":"ContainerStarted","Data":"5959fcd9efffeac8b1c34228576b3f48004c8674b0c9aa7f2451b9efc8bba0d3"}
Sep 10 21:09:46.624675 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:46.624625    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm" podStartSLOduration=1.446463941 podStartE2EDuration="1.624608401s" totalImagesPullingTime="178.14446ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:09:46.623257759 +0000 UTC m=+646.288282559" watchObservedRunningTime="2026-09-10 21:09:46.624608401 +0000 UTC m=+646.289633200"
Sep 10 21:09:55.061355 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.060917    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"]
Sep 10 21:09:55.065070 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.065046    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"
Sep 10 21:09:55.067132 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.067112    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 10 21:09:55.075191 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.075169    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"]
Sep 10 21:09:55.234678 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.234636    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6aa2cff9-04d4-4eaf-8df6-e7310cf89623-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf\" (UID: \"6aa2cff9-04d4-4eaf-8df6-e7310cf89623\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"
Sep 10 21:09:55.234866 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.234684    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpqbd\" (UniqueName: \"kubernetes.io/projected/6aa2cff9-04d4-4eaf-8df6-e7310cf89623-kube-api-access-kpqbd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf\" (UID: \"6aa2cff9-04d4-4eaf-8df6-e7310cf89623\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"
Sep 10 21:09:55.234866 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.234795    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6aa2cff9-04d4-4eaf-8df6-e7310cf89623-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf\" (UID: \"6aa2cff9-04d4-4eaf-8df6-e7310cf89623\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"
Sep 10 21:09:55.234866 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.234829    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aa2cff9-04d4-4eaf-8df6-e7310cf89623-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf\" (UID: \"6aa2cff9-04d4-4eaf-8df6-e7310cf89623\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"
Sep 10 21:09:55.235009 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.234901    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6aa2cff9-04d4-4eaf-8df6-e7310cf89623-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf\" (UID: \"6aa2cff9-04d4-4eaf-8df6-e7310cf89623\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"
Sep 10 21:09:55.235009 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.234983    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6aa2cff9-04d4-4eaf-8df6-e7310cf89623-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf\" (UID: \"6aa2cff9-04d4-4eaf-8df6-e7310cf89623\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"
Sep 10 21:09:55.335677 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.335587    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6aa2cff9-04d4-4eaf-8df6-e7310cf89623-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf\" (UID: \"6aa2cff9-04d4-4eaf-8df6-e7310cf89623\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"
Sep 10 21:09:55.335677 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.335627    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpqbd\" (UniqueName: \"kubernetes.io/projected/6aa2cff9-04d4-4eaf-8df6-e7310cf89623-kube-api-access-kpqbd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf\" (UID: \"6aa2cff9-04d4-4eaf-8df6-e7310cf89623\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"
Sep 10 21:09:55.335864 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.335683    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6aa2cff9-04d4-4eaf-8df6-e7310cf89623-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf\" (UID: \"6aa2cff9-04d4-4eaf-8df6-e7310cf89623\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"
Sep 10 21:09:55.335864 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.335706    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aa2cff9-04d4-4eaf-8df6-e7310cf89623-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf\" (UID: \"6aa2cff9-04d4-4eaf-8df6-e7310cf89623\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"
Sep 10 21:09:55.335864 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.335754    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6aa2cff9-04d4-4eaf-8df6-e7310cf89623-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf\" (UID: \"6aa2cff9-04d4-4eaf-8df6-e7310cf89623\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"
Sep 10 21:09:55.335864 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.335799    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6aa2cff9-04d4-4eaf-8df6-e7310cf89623-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf\" (UID: \"6aa2cff9-04d4-4eaf-8df6-e7310cf89623\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"
Sep 10 21:09:55.336066 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.336038    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6aa2cff9-04d4-4eaf-8df6-e7310cf89623-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf\" (UID: \"6aa2cff9-04d4-4eaf-8df6-e7310cf89623\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"
Sep 10 21:09:55.336114 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.336066    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6aa2cff9-04d4-4eaf-8df6-e7310cf89623-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf\" (UID: \"6aa2cff9-04d4-4eaf-8df6-e7310cf89623\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"
Sep 10 21:09:55.336156 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.336109    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6aa2cff9-04d4-4eaf-8df6-e7310cf89623-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf\" (UID: \"6aa2cff9-04d4-4eaf-8df6-e7310cf89623\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"
Sep 10 21:09:55.338077 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.338048    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6aa2cff9-04d4-4eaf-8df6-e7310cf89623-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf\" (UID: \"6aa2cff9-04d4-4eaf-8df6-e7310cf89623\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"
Sep 10 21:09:55.338337 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.338320    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aa2cff9-04d4-4eaf-8df6-e7310cf89623-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf\" (UID: \"6aa2cff9-04d4-4eaf-8df6-e7310cf89623\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"
Sep 10 21:09:55.345401 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.345374    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpqbd\" (UniqueName: \"kubernetes.io/projected/6aa2cff9-04d4-4eaf-8df6-e7310cf89623-kube-api-access-kpqbd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf\" (UID: \"6aa2cff9-04d4-4eaf-8df6-e7310cf89623\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"
Sep 10 21:09:55.378282 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.378259    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"
Sep 10 21:09:55.649074 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.649050    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"]
Sep 10 21:09:55.651801 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:09:55.651773    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa2cff9_04d4_4eaf_8df6_e7310cf89623.slice/crio-2991755276b9ad9e1abc89d870c7303192ab9bfcabf64206ce55f9fd41f34cd7 WatchSource:0}: Error finding container 2991755276b9ad9e1abc89d870c7303192ab9bfcabf64206ce55f9fd41f34cd7: Status 404 returned error can't find the container with id 2991755276b9ad9e1abc89d870c7303192ab9bfcabf64206ce55f9fd41f34cd7
Sep 10 21:09:55.877091 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.877060    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"
Sep 10 21:09:55.877200 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.877131    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"
Sep 10 21:09:55.890024 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:55.890004    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"
Sep 10 21:09:56.649195 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:56.649160    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf" event={"ID":"6aa2cff9-04d4-4eaf-8df6-e7310cf89623","Type":"ContainerStarted","Data":"8362f02a501b9efe50f2aa84a411935eb3a249ddda15c4b58e5e5e748258b305"}
Sep 10 21:09:56.649640 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:56.649218    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf" event={"ID":"6aa2cff9-04d4-4eaf-8df6-e7310cf89623","Type":"ContainerStarted","Data":"2991755276b9ad9e1abc89d870c7303192ab9bfcabf64206ce55f9fd41f34cd7"}
Sep 10 21:09:56.661652 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:56.661618    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b58zm"
Sep 10 21:09:56.667214 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:09:56.667162    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf" podStartSLOduration=1.4876858419999999 podStartE2EDuration="1.66714395s" totalImagesPullingTime="179.458108ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:09:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:09:56.666632833 +0000 UTC m=+656.331657634" watchObservedRunningTime="2026-09-10 21:09:56.66714395 +0000 UTC m=+656.332168752"
Sep 10 21:10:05.379151 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:05.379107    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"
Sep 10 21:10:05.379744 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:05.379200    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"
Sep 10 21:10:05.391785 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:05.391759    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"
Sep 10 21:10:05.703797 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:05.703714    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf"
Sep 10 21:10:12.562725 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:12.562687    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"]
Sep 10 21:10:12.566898 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:12.566875    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"
Sep 10 21:10:12.568968 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:12.568943    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 21:10:12.576030 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:12.576007    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"]
Sep 10 21:10:12.588512 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:12.588484    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6228bd72-6ae7-4078-852d-24dffc5e437f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz\" (UID: \"6228bd72-6ae7-4078-852d-24dffc5e437f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"
Sep 10 21:10:12.588619 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:12.588525    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llwgm\" (UniqueName: \"kubernetes.io/projected/6228bd72-6ae7-4078-852d-24dffc5e437f-kube-api-access-llwgm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz\" (UID: \"6228bd72-6ae7-4078-852d-24dffc5e437f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"
Sep 10 21:10:12.588619 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:12.588548    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6228bd72-6ae7-4078-852d-24dffc5e437f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz\" (UID: \"6228bd72-6ae7-4078-852d-24dffc5e437f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"
Sep 10 21:10:12.588619 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:12.588607    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6228bd72-6ae7-4078-852d-24dffc5e437f-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz\" (UID: \"6228bd72-6ae7-4078-852d-24dffc5e437f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"
Sep 10 21:10:12.588727 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:12.588632    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6228bd72-6ae7-4078-852d-24dffc5e437f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz\" (UID: \"6228bd72-6ae7-4078-852d-24dffc5e437f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"
Sep 10 21:10:12.588727 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:12.588675    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6228bd72-6ae7-4078-852d-24dffc5e437f-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz\" (UID: \"6228bd72-6ae7-4078-852d-24dffc5e437f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"
Sep 10 21:10:12.690037 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:12.690009    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6228bd72-6ae7-4078-852d-24dffc5e437f-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz\" (UID: \"6228bd72-6ae7-4078-852d-24dffc5e437f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"
Sep 10 21:10:12.690203 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:12.690080    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6228bd72-6ae7-4078-852d-24dffc5e437f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz\" (UID: \"6228bd72-6ae7-4078-852d-24dffc5e437f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"
Sep 10 21:10:12.690203 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:12.690098    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llwgm\" (UniqueName: \"kubernetes.io/projected/6228bd72-6ae7-4078-852d-24dffc5e437f-kube-api-access-llwgm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz\" (UID: \"6228bd72-6ae7-4078-852d-24dffc5e437f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"
Sep 10 21:10:12.690203 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:12.690120    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6228bd72-6ae7-4078-852d-24dffc5e437f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz\" (UID: \"6228bd72-6ae7-4078-852d-24dffc5e437f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"
Sep 10 21:10:12.690203 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:12.690145    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6228bd72-6ae7-4078-852d-24dffc5e437f-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz\" (UID: \"6228bd72-6ae7-4078-852d-24dffc5e437f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"
Sep 10 21:10:12.690203 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:12.690165    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6228bd72-6ae7-4078-852d-24dffc5e437f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz\" (UID: \"6228bd72-6ae7-4078-852d-24dffc5e437f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"
Sep 10 21:10:12.690511 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:12.690487    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6228bd72-6ae7-4078-852d-24dffc5e437f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz\" (UID: \"6228bd72-6ae7-4078-852d-24dffc5e437f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"
Sep 10 21:10:12.690592 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:12.690532    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6228bd72-6ae7-4078-852d-24dffc5e437f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz\" (UID: \"6228bd72-6ae7-4078-852d-24dffc5e437f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"
Sep 10 21:10:12.690592 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:12.690554    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6228bd72-6ae7-4078-852d-24dffc5e437f-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz\" (UID: \"6228bd72-6ae7-4078-852d-24dffc5e437f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"
Sep 10 21:10:12.692246 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:12.692223    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6228bd72-6ae7-4078-852d-24dffc5e437f-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz\" (UID: \"6228bd72-6ae7-4078-852d-24dffc5e437f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"
Sep 10 21:10:12.692575 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:12.692556    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6228bd72-6ae7-4078-852d-24dffc5e437f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz\" (UID: \"6228bd72-6ae7-4078-852d-24dffc5e437f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"
Sep 10 21:10:12.697816 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:12.697791    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llwgm\" (UniqueName: \"kubernetes.io/projected/6228bd72-6ae7-4078-852d-24dffc5e437f-kube-api-access-llwgm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz\" (UID: \"6228bd72-6ae7-4078-852d-24dffc5e437f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"
Sep 10 21:10:12.876811 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:12.876765    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"
Sep 10 21:10:13.147997 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:13.147805    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"]
Sep 10 21:10:13.150983 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:10:13.150956    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6228bd72_6ae7_4078_852d_24dffc5e437f.slice/crio-0af009eec33786405901f950feadba931e430bb36d205f886a45539a1657e7bb WatchSource:0}: Error finding container 0af009eec33786405901f950feadba931e430bb36d205f886a45539a1657e7bb: Status 404 returned error can't find the container with id 0af009eec33786405901f950feadba931e430bb36d205f886a45539a1657e7bb
Sep 10 21:10:13.730580 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:13.730545    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz" event={"ID":"6228bd72-6ae7-4078-852d-24dffc5e437f","Type":"ContainerStarted","Data":"cb34ad3b2c854252316e184717db117448202452b154893c46c2c8aa602aac5d"}
Sep 10 21:10:13.730580 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:13.730590    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz" event={"ID":"6228bd72-6ae7-4078-852d-24dffc5e437f","Type":"ContainerStarted","Data":"0af009eec33786405901f950feadba931e430bb36d205f886a45539a1657e7bb"}
Sep 10 21:10:13.748495 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:13.748450    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz" podStartSLOduration=1.5605729849999999 podStartE2EDuration="1.748419404s" totalImagesPullingTime="187.846419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:10:13.746957835 +0000 UTC m=+673.411982635" watchObservedRunningTime="2026-09-10 21:10:13.748419404 +0000 UTC m=+673.413444204"
Sep 10 21:10:22.877603 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:22.877557    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"
Sep 10 21:10:22.877603 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:22.877620    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"
Sep 10 21:10:22.890262 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:22.890229    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"
Sep 10 21:10:23.786467 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:10:23.786436    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz"
Sep 10 21:11:32.288396 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:32.288357    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b558ff6f7-krwnj"]
Sep 10 21:11:32.289022 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:32.288576    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6b558ff6f7-krwnj" podUID="47a71315-a0b2-4c7f-9006-c9081a90a31a" containerName="authorino" containerID="cri-o://e796f405d52c3d119243fdd79393e8097abbfc44bf897d9243790e939d32d9a3" gracePeriod=30
Sep 10 21:11:32.532253 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:32.532226    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b558ff6f7-krwnj"
Sep 10 21:11:32.625581 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:32.625551    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/47a71315-a0b2-4c7f-9006-c9081a90a31a-tls-cert\") pod \"47a71315-a0b2-4c7f-9006-c9081a90a31a\" (UID: \"47a71315-a0b2-4c7f-9006-c9081a90a31a\") "
Sep 10 21:11:32.625716 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:32.625604    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2mt4\" (UniqueName: \"kubernetes.io/projected/47a71315-a0b2-4c7f-9006-c9081a90a31a-kube-api-access-j2mt4\") pod \"47a71315-a0b2-4c7f-9006-c9081a90a31a\" (UID: \"47a71315-a0b2-4c7f-9006-c9081a90a31a\") "
Sep 10 21:11:32.627631 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:32.627606    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a71315-a0b2-4c7f-9006-c9081a90a31a-kube-api-access-j2mt4" (OuterVolumeSpecName: "kube-api-access-j2mt4") pod "47a71315-a0b2-4c7f-9006-c9081a90a31a" (UID: "47a71315-a0b2-4c7f-9006-c9081a90a31a"). InnerVolumeSpecName "kube-api-access-j2mt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:11:32.635461 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:32.635439    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47a71315-a0b2-4c7f-9006-c9081a90a31a-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "47a71315-a0b2-4c7f-9006-c9081a90a31a" (UID: "47a71315-a0b2-4c7f-9006-c9081a90a31a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:11:32.726569 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:32.726532    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2mt4\" (UniqueName: \"kubernetes.io/projected/47a71315-a0b2-4c7f-9006-c9081a90a31a-kube-api-access-j2mt4\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:11:32.726569 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:32.726563    2584 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/47a71315-a0b2-4c7f-9006-c9081a90a31a-tls-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:11:33.092239 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:33.092145    2584 generic.go:356] "Generic (PLEG): container finished" podID="47a71315-a0b2-4c7f-9006-c9081a90a31a" containerID="e796f405d52c3d119243fdd79393e8097abbfc44bf897d9243790e939d32d9a3" exitCode=0
Sep 10 21:11:33.092239 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:33.092197    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b558ff6f7-krwnj"
Sep 10 21:11:33.092239 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:33.092210    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b558ff6f7-krwnj" event={"ID":"47a71315-a0b2-4c7f-9006-c9081a90a31a","Type":"ContainerDied","Data":"e796f405d52c3d119243fdd79393e8097abbfc44bf897d9243790e939d32d9a3"}
Sep 10 21:11:33.092239 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:33.092241    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b558ff6f7-krwnj" event={"ID":"47a71315-a0b2-4c7f-9006-c9081a90a31a","Type":"ContainerDied","Data":"8930185d4d2527d0146e57eeec199ff9e5d741b6192f094ca13610432da4c427"}
Sep 10 21:11:33.092599 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:33.092257    2584 scope.go:117] "RemoveContainer" containerID="e796f405d52c3d119243fdd79393e8097abbfc44bf897d9243790e939d32d9a3"
Sep 10 21:11:33.102696 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:33.102677    2584 scope.go:117] "RemoveContainer" containerID="e796f405d52c3d119243fdd79393e8097abbfc44bf897d9243790e939d32d9a3"
Sep 10 21:11:33.103096 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:11:33.103075    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e796f405d52c3d119243fdd79393e8097abbfc44bf897d9243790e939d32d9a3\": container with ID starting with e796f405d52c3d119243fdd79393e8097abbfc44bf897d9243790e939d32d9a3 not found: ID does not exist" containerID="e796f405d52c3d119243fdd79393e8097abbfc44bf897d9243790e939d32d9a3"
Sep 10 21:11:33.103178 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:33.103105    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e796f405d52c3d119243fdd79393e8097abbfc44bf897d9243790e939d32d9a3"} err="failed to get container status \"e796f405d52c3d119243fdd79393e8097abbfc44bf897d9243790e939d32d9a3\": rpc error: code = NotFound desc = could not find container \"e796f405d52c3d119243fdd79393e8097abbfc44bf897d9243790e939d32d9a3\": container with ID starting with e796f405d52c3d119243fdd79393e8097abbfc44bf897d9243790e939d32d9a3 not found: ID does not exist"
Sep 10 21:11:33.109903 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:33.109881    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b558ff6f7-krwnj"]
Sep 10 21:11:33.113974 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:33.113950    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6b558ff6f7-krwnj"]
Sep 10 21:11:34.840151 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:34.840118    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47a71315-a0b2-4c7f-9006-c9081a90a31a" path="/var/lib/kubelet/pods/47a71315-a0b2-4c7f-9006-c9081a90a31a/volumes"
Sep 10 21:11:50.227474 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.227440    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"]
Sep 10 21:11:50.227933 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.227892    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47a71315-a0b2-4c7f-9006-c9081a90a31a" containerName="authorino"
Sep 10 21:11:50.227933 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.227904    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="47a71315-a0b2-4c7f-9006-c9081a90a31a" containerName="authorino"
Sep 10 21:11:50.228006 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.227973    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="47a71315-a0b2-4c7f-9006-c9081a90a31a" containerName="authorino"
Sep 10 21:11:50.232525 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.232508    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.235029 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.234998    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b22a78-openshift-default-dockercfg-hp2c2\""
Sep 10 21:11:50.248510 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.248486    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"]
Sep 10 21:11:50.284139 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.284106    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7315ba-55f9-495a-bd7e-a917794e9997-istio-podinfo\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.284290 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.284145    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-istio-envoy\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.284290 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.284180    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-credential-socket\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.284290 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.284275    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-workload-certs\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.284415 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.284383    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7315ba-55f9-495a-bd7e-a917794e9997-istiod-ca-cert\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.284494 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.284446    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk8fb\" (UniqueName: \"kubernetes.io/projected/2a7315ba-55f9-495a-bd7e-a917794e9997-kube-api-access-rk8fb\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.284494 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.284465    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7315ba-55f9-495a-bd7e-a917794e9997-istio-token\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.284568 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.284493    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-workload-socket\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.284568 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.284508    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-istio-data\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.385700 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.385660    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7315ba-55f9-495a-bd7e-a917794e9997-istio-podinfo\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.385700 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.385701    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-istio-envoy\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.385929 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.385723    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-credential-socket\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.385929 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.385769    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-workload-certs\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.385929 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.385811    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7315ba-55f9-495a-bd7e-a917794e9997-istiod-ca-cert\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.385929 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.385841    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk8fb\" (UniqueName: \"kubernetes.io/projected/2a7315ba-55f9-495a-bd7e-a917794e9997-kube-api-access-rk8fb\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.385929 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.385864    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7315ba-55f9-495a-bd7e-a917794e9997-istio-token\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.385929 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.385902    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-workload-socket\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.385929 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.385924    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-istio-data\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.386248 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.386185    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-credential-socket\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.386248 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.386215    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-workload-certs\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.386386 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.386369    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-istio-data\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.386659 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.386445    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-workload-socket\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.386787 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.386765    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7315ba-55f9-495a-bd7e-a917794e9997-istiod-ca-cert\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.388171 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.388149    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-istio-envoy\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.388498 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.388478    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7315ba-55f9-495a-bd7e-a917794e9997-istio-podinfo\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.393731 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.393700    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7315ba-55f9-495a-bd7e-a917794e9997-istio-token\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.393820 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.393750    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk8fb\" (UniqueName: \"kubernetes.io/projected/2a7315ba-55f9-495a-bd7e-a917794e9997-kube-api-access-rk8fb\") pod \"e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") " pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.545975 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.545880    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:50.822436 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.822398    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"]
Sep 10 21:11:50.824723 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:11:50.824697    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7315ba_55f9_495a_bd7e_a917794e9997.slice/crio-0374c9fa7491e77aa7451cafd67cf6075adc89b60e46cb4b7f4da09ba405bce4 WatchSource:0}: Error finding container 0374c9fa7491e77aa7451cafd67cf6075adc89b60e46cb4b7f4da09ba405bce4: Status 404 returned error can't find the container with id 0374c9fa7491e77aa7451cafd67cf6075adc89b60e46cb4b7f4da09ba405bce4
Sep 10 21:11:50.827381 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.827345    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:11:50.827478 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.827413    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:11:50.827478 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:50.827470    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:11:51.176302 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:51.176267    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l" event={"ID":"2a7315ba-55f9-495a-bd7e-a917794e9997","Type":"ContainerStarted","Data":"362827d42b99805bb0307779666882da2153b8cdf8d8bedd3a42eb306824c89c"}
Sep 10 21:11:51.176495 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:51.176334    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l" event={"ID":"2a7315ba-55f9-495a-bd7e-a917794e9997","Type":"ContainerStarted","Data":"0374c9fa7491e77aa7451cafd67cf6075adc89b60e46cb4b7f4da09ba405bce4"}
Sep 10 21:11:51.203188 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:51.203124    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l" podStartSLOduration=1.203106716 podStartE2EDuration="1.203106716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:51.201280235 +0000 UTC m=+770.866305035" watchObservedRunningTime="2026-09-10 21:11:51.203106716 +0000 UTC m=+770.868131516"
Sep 10 21:11:51.546131 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:51.546046    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:51.551041 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:51.551014    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:52.181064 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:52.181028    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:52.181962 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:52.181944    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:11:58.551147 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.551108    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"]
Sep 10 21:11:58.554579 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.554558    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.556578 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.556560    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8086f00e-openshift-default-dockercfg-wnpsh\""
Sep 10 21:11:58.565033 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.565011    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"]
Sep 10 21:11:58.670230 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.670199    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-istio-data\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.670465 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.670247    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-workload-certs\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.670465 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.670290    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1208af-cb0e-4d1e-a90a-5725e61df387-istio-podinfo\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.670465 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.670310    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1208af-cb0e-4d1e-a90a-5725e61df387-istiod-ca-cert\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.670465 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.670337    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86tsx\" (UniqueName: \"kubernetes.io/projected/ce1208af-cb0e-4d1e-a90a-5725e61df387-kube-api-access-86tsx\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.670465 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.670409    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-workload-socket\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.670658 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.670488    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-credential-socket\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.670658 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.670508    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-istio-envoy\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.670658 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.670535    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1208af-cb0e-4d1e-a90a-5725e61df387-istio-token\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.771736 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.771702    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-workload-certs\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.771736 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.771739    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1208af-cb0e-4d1e-a90a-5725e61df387-istio-podinfo\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.771981 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.771766    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1208af-cb0e-4d1e-a90a-5725e61df387-istiod-ca-cert\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.771981 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.771873    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86tsx\" (UniqueName: \"kubernetes.io/projected/ce1208af-cb0e-4d1e-a90a-5725e61df387-kube-api-access-86tsx\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.771981 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.771952    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-workload-socket\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.772156 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.772003    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-credential-socket\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.772156 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.772032    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-istio-envoy\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.772156 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.772077    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1208af-cb0e-4d1e-a90a-5725e61df387-istio-token\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.772315 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.772174    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-istio-data\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.772548 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.772520    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-workload-socket\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.772645 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.772526    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-workload-certs\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.772645 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.772528    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-credential-socket\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.772645 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.772568    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-istio-data\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.772801 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.772700    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1208af-cb0e-4d1e-a90a-5725e61df387-istiod-ca-cert\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.774275 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.774252    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-istio-envoy\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.774364 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.774347    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1208af-cb0e-4d1e-a90a-5725e61df387-istio-podinfo\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.783138 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.783117    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86tsx\" (UniqueName: \"kubernetes.io/projected/ce1208af-cb0e-4d1e-a90a-5725e61df387-kube-api-access-86tsx\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.783488 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.783472    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1208af-cb0e-4d1e-a90a-5725e61df387-istio-token\") pod \"e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") " pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:58.866887 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:58.866803    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:11:59.149792 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:59.149765    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"]
Sep 10 21:11:59.151709 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:11:59.151681    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1208af_cb0e_4d1e_a90a_5725e61df387.slice/crio-fb8ae6e55b507589aa10aa511f7fa6019f746af36a1e74cb0c138f10fe12d082 WatchSource:0}: Error finding container fb8ae6e55b507589aa10aa511f7fa6019f746af36a1e74cb0c138f10fe12d082: Status 404 returned error can't find the container with id fb8ae6e55b507589aa10aa511f7fa6019f746af36a1e74cb0c138f10fe12d082
Sep 10 21:11:59.153935 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:59.153908    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:11:59.154012 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:59.153969    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:11:59.154050 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:59.154024    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:11:59.219064 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:11:59.219020    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv" event={"ID":"ce1208af-cb0e-4d1e-a90a-5725e61df387","Type":"ContainerStarted","Data":"fb8ae6e55b507589aa10aa511f7fa6019f746af36a1e74cb0c138f10fe12d082"}
Sep 10 21:12:00.226739 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:00.226701    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv" event={"ID":"ce1208af-cb0e-4d1e-a90a-5725e61df387","Type":"ContainerStarted","Data":"66625d0bf8e76e58671330a32cafbb14b800a797e1f2ebbc32adba3824b3c265"}
Sep 10 21:12:00.248657 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:00.248598    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv" podStartSLOduration=2.248580817 podStartE2EDuration="2.248580817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:12:00.247829704 +0000 UTC m=+779.912854504" watchObservedRunningTime="2026-09-10 21:12:00.248580817 +0000 UTC m=+779.913605617"
Sep 10 21:12:00.867319 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:00.867286    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:12:00.873826 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:00.873797    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:12:01.231301 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:01.231272    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:12:01.232413 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:01.232394    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:12:08.224036 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:08.223620    2584 patch_prober.go:28] interesting pod/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 21:12:08.224036 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:08.223713    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l" podUID="2a7315ba-55f9-495a-bd7e-a917794e9997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 21:12:10.277165 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:10.277131    2584 generic.go:356] "Generic (PLEG): container finished" podID="2a7315ba-55f9-495a-bd7e-a917794e9997" containerID="362827d42b99805bb0307779666882da2153b8cdf8d8bedd3a42eb306824c89c" exitCode=0
Sep 10 21:12:10.277582 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:10.277201    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l" event={"ID":"2a7315ba-55f9-495a-bd7e-a917794e9997","Type":"ContainerDied","Data":"362827d42b99805bb0307779666882da2153b8cdf8d8bedd3a42eb306824c89c"}
Sep 10 21:12:10.277634 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:10.277594    2584 scope.go:117] "RemoveContainer" containerID="362827d42b99805bb0307779666882da2153b8cdf8d8bedd3a42eb306824c89c"
Sep 10 21:12:10.278648 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:10.278624    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:12:10.278721 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:10.278688    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:12:10.278721 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:10.278718    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:12:11.284185 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:11.284152    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l" event={"ID":"2a7315ba-55f9-495a-bd7e-a917794e9997","Type":"ContainerStarted","Data":"a05f75ef41098a3e98f0fad32a660f7a335dc056743fc4d8b0855b1b5716d17e"}
Sep 10 21:12:11.546994 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:11.546880    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:12:11.547573 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:11.547520    2584 patch_prober.go:28] interesting pod/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 10 21:12:11.547701 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:11.547628    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l" podUID="2a7315ba-55f9-495a-bd7e-a917794e9997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 10 21:12:11.941830 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:11.941800    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"]
Sep 10 21:12:11.944310 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:11.944292    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:12:11.946403 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:11.946383    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-1c65017b-kserve-self-signed-certs\""
Sep 10 21:12:11.953571 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:11.953550    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"]
Sep 10 21:12:12.024923 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:12.024869    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b71d537-0290-43f9-9277-5f2c946a55b4-dshm\") pod \"e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg\" (UID: \"9b71d537-0290-43f9-9277-5f2c946a55b4\") " pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:12:12.025118 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:12.024941    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6r5d\" (UniqueName: \"kubernetes.io/projected/9b71d537-0290-43f9-9277-5f2c946a55b4-kube-api-access-l6r5d\") pod \"e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg\" (UID: \"9b71d537-0290-43f9-9277-5f2c946a55b4\") " pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:12:12.025118 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:12.024973    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b71d537-0290-43f9-9277-5f2c946a55b4-tmp-dir\") pod \"e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg\" (UID: \"9b71d537-0290-43f9-9277-5f2c946a55b4\") " pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:12:12.025118 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:12.025019    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b71d537-0290-43f9-9277-5f2c946a55b4-model-cache\") pod \"e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg\" (UID: \"9b71d537-0290-43f9-9277-5f2c946a55b4\") " pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:12:12.025118 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:12.025050    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b71d537-0290-43f9-9277-5f2c946a55b4-home\") pod \"e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg\" (UID: \"9b71d537-0290-43f9-9277-5f2c946a55b4\") " pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:12:12.025118 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:12.025085    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b71d537-0290-43f9-9277-5f2c946a55b4-tls-certs\") pod \"e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg\" (UID: \"9b71d537-0290-43f9-9277-5f2c946a55b4\") " pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:12:12.125942 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:12.125897    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b71d537-0290-43f9-9277-5f2c946a55b4-home\") pod \"e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg\" (UID: \"9b71d537-0290-43f9-9277-5f2c946a55b4\") " pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:12:12.126127 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:12.125963    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b71d537-0290-43f9-9277-5f2c946a55b4-tls-certs\") pod \"e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg\" (UID: \"9b71d537-0290-43f9-9277-5f2c946a55b4\") " pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:12:12.126127 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:12.126081    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b71d537-0290-43f9-9277-5f2c946a55b4-dshm\") pod \"e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg\" (UID: \"9b71d537-0290-43f9-9277-5f2c946a55b4\") " pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:12:12.126127 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:12.126116    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6r5d\" (UniqueName: \"kubernetes.io/projected/9b71d537-0290-43f9-9277-5f2c946a55b4-kube-api-access-l6r5d\") pod \"e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg\" (UID: \"9b71d537-0290-43f9-9277-5f2c946a55b4\") " pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:12:12.126314 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:12.126145    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b71d537-0290-43f9-9277-5f2c946a55b4-tmp-dir\") pod \"e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg\" (UID: \"9b71d537-0290-43f9-9277-5f2c946a55b4\") " pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:12:12.126314 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:12.126193    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b71d537-0290-43f9-9277-5f2c946a55b4-model-cache\") pod \"e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg\" (UID: \"9b71d537-0290-43f9-9277-5f2c946a55b4\") " pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:12:12.126679 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:12.126646    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b71d537-0290-43f9-9277-5f2c946a55b4-model-cache\") pod \"e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg\" (UID: \"9b71d537-0290-43f9-9277-5f2c946a55b4\") " pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:12:12.127072 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:12.127045    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b71d537-0290-43f9-9277-5f2c946a55b4-home\") pod \"e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg\" (UID: \"9b71d537-0290-43f9-9277-5f2c946a55b4\") " pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:12:12.127375 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:12.127352    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b71d537-0290-43f9-9277-5f2c946a55b4-tmp-dir\") pod \"e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg\" (UID: \"9b71d537-0290-43f9-9277-5f2c946a55b4\") " pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:12:12.129443 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:12.129401    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b71d537-0290-43f9-9277-5f2c946a55b4-tls-certs\") pod \"e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg\" (UID: \"9b71d537-0290-43f9-9277-5f2c946a55b4\") " pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:12:12.131355 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:12.131333    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b71d537-0290-43f9-9277-5f2c946a55b4-dshm\") pod \"e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg\" (UID: \"9b71d537-0290-43f9-9277-5f2c946a55b4\") " pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:12:12.136007 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:12.135982    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6r5d\" (UniqueName: \"kubernetes.io/projected/9b71d537-0290-43f9-9277-5f2c946a55b4-kube-api-access-l6r5d\") pod \"e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg\" (UID: \"9b71d537-0290-43f9-9277-5f2c946a55b4\") " pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:12:12.254720 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:12.254641    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:12:12.546604 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:12.546525    2584 patch_prober.go:28] interesting pod/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 10 21:12:12.546604 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:12.546580    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l" podUID="2a7315ba-55f9-495a-bd7e-a917794e9997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 10 21:12:12.550758 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:12.550692    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"]
Sep 10 21:12:13.297262 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:13.297222    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg" event={"ID":"9b71d537-0290-43f9-9277-5f2c946a55b4","Type":"ContainerStarted","Data":"29161a3678342f4b4babe553cded88cfb67e1071c20f074372d58fc1b265c49e"}
Sep 10 21:12:13.297262 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:13.297269    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg" event={"ID":"9b71d537-0290-43f9-9277-5f2c946a55b4","Type":"ContainerStarted","Data":"588cb425f9b431a0adccfe0f647dff518c5d64887431401e9867e0f790be0be8"}
Sep 10 21:12:13.315536 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:13.315472    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg" podStartSLOduration=2.315450998 podStartE2EDuration="2.315450998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:12:13.313621295 +0000 UTC m=+792.978646095" watchObservedRunningTime="2026-09-10 21:12:13.315450998 +0000 UTC m=+792.980475798"
Sep 10 21:12:13.547130 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:13.547096    2584 patch_prober.go:28] interesting pod/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 10 21:12:13.547553 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:13.547156    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l" podUID="2a7315ba-55f9-495a-bd7e-a917794e9997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 10 21:12:14.547733 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:14.547265    2584 patch_prober.go:28] interesting pod/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 10 21:12:14.547733 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:14.547323    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l" podUID="2a7315ba-55f9-495a-bd7e-a917794e9997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 10 21:12:15.546859 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:15.546828    2584 patch_prober.go:28] interesting pod/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 10 21:12:15.547869 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:15.546889    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l" podUID="2a7315ba-55f9-495a-bd7e-a917794e9997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 10 21:12:16.232805 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:16.232764    2584 patch_prober.go:28] interesting pod/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 10 21:12:16.232988 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:16.232818    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv" podUID="ce1208af-cb0e-4d1e-a90a-5725e61df387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 10 21:12:16.323257 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:16.323220    2584 generic.go:356] "Generic (PLEG): container finished" podID="ce1208af-cb0e-4d1e-a90a-5725e61df387" containerID="66625d0bf8e76e58671330a32cafbb14b800a797e1f2ebbc32adba3824b3c265" exitCode=0
Sep 10 21:12:16.323614 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:16.323304    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv" event={"ID":"ce1208af-cb0e-4d1e-a90a-5725e61df387","Type":"ContainerDied","Data":"66625d0bf8e76e58671330a32cafbb14b800a797e1f2ebbc32adba3824b3c265"}
Sep 10 21:12:16.324377 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:16.324356    2584 scope.go:117] "RemoveContainer" containerID="66625d0bf8e76e58671330a32cafbb14b800a797e1f2ebbc32adba3824b3c265"
Sep 10 21:12:16.325906 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:16.325872    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:12:16.326170 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:16.326145    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:12:16.326351 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:16.326329    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:12:16.546914 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:16.546812    2584 patch_prober.go:28] interesting pod/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 10 21:12:16.546914 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:16.546905    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l" podUID="2a7315ba-55f9-495a-bd7e-a917794e9997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 10 21:12:17.331867 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:17.331833    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv" event={"ID":"ce1208af-cb0e-4d1e-a90a-5725e61df387","Type":"ContainerStarted","Data":"7fb62e8a2d6354c645f0edfce940b8c9cf2b6686ecb1081e9474f99f1e5111bd"}
Sep 10 21:12:17.550765 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:17.550732    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:12:17.551118 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:17.551082    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:12:17.551879 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:17.551861    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:12:17.867526 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:17.867478    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:12:17.867728 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:17.867674    2584 patch_prober.go:28] interesting pod/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 10 21:12:17.867795 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:17.867727    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv" podUID="ce1208af-cb0e-4d1e-a90a-5725e61df387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 10 21:12:18.768566 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.768527    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"]
Sep 10 21:12:18.772164 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.772142    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.774278 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.774256    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-414d5f-openshift-default-dockercfg-rm7hw\""
Sep 10 21:12:18.783857 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.783832    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"]
Sep 10 21:12:18.823446 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.817951    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-workload-certs\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.823446 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.818017    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76d25ff-57e2-4366-a081-675f880e63b1-istiod-ca-cert\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.823446 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.818089    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-istio-envoy\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.823446 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.818145    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76d25ff-57e2-4366-a081-675f880e63b1-istio-podinfo\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.823446 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.818197    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76d25ff-57e2-4366-a081-675f880e63b1-istio-token\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.823446 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.818246    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-credential-socket\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.823446 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.818277    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-istio-data\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.823446 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.818335    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlxdn\" (UniqueName: \"kubernetes.io/projected/c76d25ff-57e2-4366-a081-675f880e63b1-kube-api-access-dlxdn\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.823446 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.818364    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-workload-socket\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.867363 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.867328    2584 patch_prober.go:28] interesting pod/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 10 21:12:18.867546 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.867415    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv" podUID="ce1208af-cb0e-4d1e-a90a-5725e61df387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 10 21:12:18.922775 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.919934    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-istio-data\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.922775 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.920023    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlxdn\" (UniqueName: \"kubernetes.io/projected/c76d25ff-57e2-4366-a081-675f880e63b1-kube-api-access-dlxdn\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.922775 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.920054    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-workload-socket\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.922775 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.920100    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-workload-certs\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.922775 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.920134    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76d25ff-57e2-4366-a081-675f880e63b1-istiod-ca-cert\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.922775 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.920206    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-istio-envoy\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.922775 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.920263    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76d25ff-57e2-4366-a081-675f880e63b1-istio-podinfo\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.922775 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.920319    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76d25ff-57e2-4366-a081-675f880e63b1-istio-token\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.922775 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.920364    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-credential-socket\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.922775 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.920800    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-credential-socket\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.922775 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.921040    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-istio-data\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.922775 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.921568    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-workload-socket\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.922775 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.921798    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-workload-certs\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.922775 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.922464    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76d25ff-57e2-4366-a081-675f880e63b1-istiod-ca-cert\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.928351 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.928327    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-istio-envoy\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.928579 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.928558    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76d25ff-57e2-4366-a081-675f880e63b1-istio-podinfo\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.929110 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.929091    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlxdn\" (UniqueName: \"kubernetes.io/projected/c76d25ff-57e2-4366-a081-675f880e63b1-kube-api-access-dlxdn\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:18.932437 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:18.932402    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76d25ff-57e2-4366-a081-675f880e63b1-istio-token\") pod \"e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") " pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:19.083731 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:19.083646    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:19.383527 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:19.383495    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"]
Sep 10 21:12:19.386647 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:12:19.386621    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76d25ff_57e2_4366_a081_675f880e63b1.slice/crio-6e82b891d30255ee581bd7123fe3c00893c4669f3afae132653ad862144e3069 WatchSource:0}: Error finding container 6e82b891d30255ee581bd7123fe3c00893c4669f3afae132653ad862144e3069: Status 404 returned error can't find the container with id 6e82b891d30255ee581bd7123fe3c00893c4669f3afae132653ad862144e3069
Sep 10 21:12:19.388800 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:19.388771    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:12:19.388884 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:19.388847    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:12:19.388884 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:19.388873    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:12:19.868051 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:19.867964    2584 patch_prober.go:28] interesting pod/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 10 21:12:19.868051 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:19.868033    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv" podUID="ce1208af-cb0e-4d1e-a90a-5725e61df387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 10 21:12:20.351567 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:20.351528    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt" event={"ID":"c76d25ff-57e2-4366-a081-675f880e63b1","Type":"ContainerStarted","Data":"4a45d119bc0d88279c9c6ced47273ff6b2f4149e58221edf5c735933db89c86d"}
Sep 10 21:12:20.351752 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:20.351584    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt" event={"ID":"c76d25ff-57e2-4366-a081-675f880e63b1","Type":"ContainerStarted","Data":"6e82b891d30255ee581bd7123fe3c00893c4669f3afae132653ad862144e3069"}
Sep 10 21:12:20.372788 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:20.372734    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt" podStartSLOduration=2.372716784 podStartE2EDuration="2.372716784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:12:20.37046973 +0000 UTC m=+800.035494528" watchObservedRunningTime="2026-09-10 21:12:20.372716784 +0000 UTC m=+800.037741583"
Sep 10 21:12:20.867701 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:20.867667    2584 patch_prober.go:28] interesting pod/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 10 21:12:20.867874 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:20.867723    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv" podUID="ce1208af-cb0e-4d1e-a90a-5725e61df387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 10 21:12:21.084627 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:21.084586    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:21.089602 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:21.089575    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:21.357225 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:21.357191    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:21.358131 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:21.358110    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:21.867969 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:21.867929    2584 patch_prober.go:28] interesting pod/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 10 21:12:21.868158 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:21.867988    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv" podUID="ce1208af-cb0e-4d1e-a90a-5725e61df387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 10 21:12:22.255748 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:22.255709    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:12:22.256169 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:22.255764    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:12:22.269120 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:22.269097    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:12:22.374661 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:22.374632    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:12:22.871521 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:22.871477    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:12:22.871994 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:22.871972    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:12:22.872697 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:22.872675    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:12:27.241183 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:27.241152    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8086f00e-xlhr4"]
Sep 10 21:12:27.243713 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:27.243694    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8086f00e-xlhr4"
Sep 10 21:12:27.245772 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:27.245745    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:12:27.245772 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:27.245752    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4gs5\""
Sep 10 21:12:27.251950 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:27.251551    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8086f00e-xlhr4"]
Sep 10 21:12:27.415239 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:27.415208    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34e28eae-6b06-450b-969b-1d25f7602ae2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8086f00e-xlhr4\" (UID: \"34e28eae-6b06-450b-969b-1d25f7602ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8086f00e-xlhr4"
Sep 10 21:12:27.516544 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:27.516457    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34e28eae-6b06-450b-969b-1d25f7602ae2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8086f00e-xlhr4\" (UID: \"34e28eae-6b06-450b-969b-1d25f7602ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8086f00e-xlhr4"
Sep 10 21:12:27.517058 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:27.517039    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34e28eae-6b06-450b-969b-1d25f7602ae2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8086f00e-xlhr4\" (UID: \"34e28eae-6b06-450b-969b-1d25f7602ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8086f00e-xlhr4"
Sep 10 21:12:27.556522 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:27.556493    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8086f00e-xlhr4"
Sep 10 21:12:27.824243 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:27.824218    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8086f00e-xlhr4"]
Sep 10 21:12:27.825930 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:12:27.825898    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e28eae_6b06_450b_969b_1d25f7602ae2.slice/crio-02d39df3165c088f49b5f7edcbe873caebfc0e93808e8a04ab5960b5f67614fd WatchSource:0}: Error finding container 02d39df3165c088f49b5f7edcbe873caebfc0e93808e8a04ab5960b5f67614fd: Status 404 returned error can't find the container with id 02d39df3165c088f49b5f7edcbe873caebfc0e93808e8a04ab5960b5f67614fd
Sep 10 21:12:28.394354 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:28.394318    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8086f00e-xlhr4" event={"ID":"34e28eae-6b06-450b-969b-1d25f7602ae2","Type":"ContainerStarted","Data":"02d39df3165c088f49b5f7edcbe873caebfc0e93808e8a04ab5960b5f67614fd"}
Sep 10 21:12:30.406285 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:30.406250    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8086f00e-xlhr4" event={"ID":"34e28eae-6b06-450b-969b-1d25f7602ae2","Type":"ContainerStarted","Data":"73b1be22ed39fe3273a51c42c33054d62786f5c44c5027bb287bef47a9666145"}
Sep 10 21:12:30.421169 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:30.421121    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8086f00e-xlhr4" podStartSLOduration=1.255337229 podStartE2EDuration="3.421107059s" totalImagesPullingTime="2.16576983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:12:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:12:30.420239713 +0000 UTC m=+810.085264514" watchObservedRunningTime="2026-09-10 21:12:30.421107059 +0000 UTC m=+810.086131858"
Sep 10 21:12:31.412690 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:31.412659    2584 generic.go:356] "Generic (PLEG): container finished" podID="34e28eae-6b06-450b-969b-1d25f7602ae2" containerID="73b1be22ed39fe3273a51c42c33054d62786f5c44c5027bb287bef47a9666145" exitCode=0
Sep 10 21:12:31.413042 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:31.412744    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8086f00e-xlhr4" event={"ID":"34e28eae-6b06-450b-969b-1d25f7602ae2","Type":"ContainerDied","Data":"73b1be22ed39fe3273a51c42c33054d62786f5c44c5027bb287bef47a9666145"}
Sep 10 21:12:32.545928 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:32.545903    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8086f00e-xlhr4"
Sep 10 21:12:32.669952 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:32.669874    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34e28eae-6b06-450b-969b-1d25f7602ae2-maas-api-service-ca\") pod \"34e28eae-6b06-450b-969b-1d25f7602ae2\" (UID: \"34e28eae-6b06-450b-969b-1d25f7602ae2\") "
Sep 10 21:12:32.670257 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:32.670227    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34e28eae-6b06-450b-969b-1d25f7602ae2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "34e28eae-6b06-450b-969b-1d25f7602ae2" (UID: "34e28eae-6b06-450b-969b-1d25f7602ae2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:12:32.771012 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:32.770978    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34e28eae-6b06-450b-969b-1d25f7602ae2-maas-api-service-ca\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:12:33.423653 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:33.423617    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8086f00e-xlhr4" event={"ID":"34e28eae-6b06-450b-969b-1d25f7602ae2","Type":"ContainerDied","Data":"02d39df3165c088f49b5f7edcbe873caebfc0e93808e8a04ab5960b5f67614fd"}
Sep 10 21:12:33.423653 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:33.423645    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8086f00e-xlhr4"
Sep 10 21:12:33.423848 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:33.423669    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02d39df3165c088f49b5f7edcbe873caebfc0e93808e8a04ab5960b5f67614fd"
Sep 10 21:12:42.475111 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:42.475017    2584 generic.go:356] "Generic (PLEG): container finished" podID="c76d25ff-57e2-4366-a081-675f880e63b1" containerID="4a45d119bc0d88279c9c6ced47273ff6b2f4149e58221edf5c735933db89c86d" exitCode=0
Sep 10 21:12:42.475521 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:42.475093    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt" event={"ID":"c76d25ff-57e2-4366-a081-675f880e63b1","Type":"ContainerDied","Data":"4a45d119bc0d88279c9c6ced47273ff6b2f4149e58221edf5c735933db89c86d"}
Sep 10 21:12:42.475651 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:42.475634    2584 scope.go:117] "RemoveContainer" containerID="4a45d119bc0d88279c9c6ced47273ff6b2f4149e58221edf5c735933db89c86d"
Sep 10 21:12:42.476774 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:42.476748    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:12:42.476861 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:42.476813    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:12:42.476861 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:42.476846    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:12:43.482948 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:43.482907    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt" event={"ID":"c76d25ff-57e2-4366-a081-675f880e63b1","Type":"ContainerStarted","Data":"4a0a3839eeecfa03c61c5cbefa4e86de5c843ac4de173836700824846dcf38e1"}
Sep 10 21:12:43.906516 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:43.906477    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8086f00e-xlhr4"]
Sep 10 21:12:43.909771 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:43.909731    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8086f00e-xlhr4"]
Sep 10 21:12:44.047393 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:44.047345    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"]
Sep 10 21:12:44.047696 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:44.047665    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv" podUID="ce1208af-cb0e-4d1e-a90a-5725e61df387" containerName="istio-proxy" containerID="cri-o://7fb62e8a2d6354c645f0edfce940b8c9cf2b6686ecb1081e9474f99f1e5111bd" gracePeriod=30
Sep 10 21:12:44.084507 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:44.084479    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:44.084670 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:44.084638    2584 patch_prober.go:28] interesting pod/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 10 21:12:44.084744 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:44.084691    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt" podUID="c76d25ff-57e2-4366-a081-675f880e63b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 10 21:12:44.840022 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:44.839945    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34e28eae-6b06-450b-969b-1d25f7602ae2" path="/var/lib/kubelet/pods/34e28eae-6b06-450b-969b-1d25f7602ae2/volumes"
Sep 10 21:12:45.085003 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.084966    2584 patch_prober.go:28] interesting pod/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 10 21:12:45.085163 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.085025    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt" podUID="c76d25ff-57e2-4366-a081-675f880e63b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 10 21:12:45.110228 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.110141    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"]
Sep 10 21:12:45.111175 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.111152    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34e28eae-6b06-450b-969b-1d25f7602ae2" containerName="revoke-keys"
Sep 10 21:12:45.111340 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.111326    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="34e28eae-6b06-450b-969b-1d25f7602ae2" containerName="revoke-keys"
Sep 10 21:12:45.118255 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.118222    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="34e28eae-6b06-450b-969b-1d25f7602ae2" containerName="revoke-keys"
Sep 10 21:12:45.129975 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.126572    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.132364 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.132339    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-065d3bbf-gw-openshift-default-dockercfg-vxfkf\""
Sep 10 21:12:45.132978 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.132954    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"]
Sep 10 21:12:45.232113 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.232044    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-workload-certs\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.232400 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.232373    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istiod-ca-cert\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.232931 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.232525    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-credential-socket\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.232931 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.232560    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istio-token\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.232931 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.232602    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-workload-socket\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.232931 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.232645    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8l2f\" (UniqueName: \"kubernetes.io/projected/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-kube-api-access-f8l2f\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.232931 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.232701    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istio-podinfo\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.232931 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.232762    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istio-data\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.232931 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.232799    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istio-envoy\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.333660 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.333622    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-workload-certs\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.333660 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.333667    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istiod-ca-cert\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.333912 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.333685    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-credential-socket\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.333912 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.333712    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istio-token\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.333912 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.333747    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-workload-socket\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.333912 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.333787    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8l2f\" (UniqueName: \"kubernetes.io/projected/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-kube-api-access-f8l2f\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.333912 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.333835    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istio-podinfo\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.333912 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.333894    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istio-data\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.334219 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.333927    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istio-envoy\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.334219 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.334199    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-credential-socket\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.334574 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.334547    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istiod-ca-cert\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.334677 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.334581    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-workload-certs\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.334791 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.334746    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istio-data\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.334791 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.334767    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-workload-socket\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.336787 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.336762    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istio-envoy\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.336885 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.336825    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istio-podinfo\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.341907 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.341881    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8l2f\" (UniqueName: \"kubernetes.io/projected/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-kube-api-access-f8l2f\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.342353 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.342330    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istio-token\") pod \"e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") " pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.445829 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.445792    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:45.731709 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.731685    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"]
Sep 10 21:12:45.733019 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:12:45.732993    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a139c2_bbf1_4f3e_a5c8_4ecc0e5da0cc.slice/crio-78eea7c1ae0112cf7a38fadae77dbd8c01be688867536aa999771c075e1e0a79 WatchSource:0}: Error finding container 78eea7c1ae0112cf7a38fadae77dbd8c01be688867536aa999771c075e1e0a79: Status 404 returned error can't find the container with id 78eea7c1ae0112cf7a38fadae77dbd8c01be688867536aa999771c075e1e0a79
Sep 10 21:12:45.735457 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.735397    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:12:45.735555 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.735493    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:12:45.735555 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:45.735523    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:12:46.085104 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:46.085020    2584 patch_prober.go:28] interesting pod/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 10 21:12:46.085481 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:46.085092    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt" podUID="c76d25ff-57e2-4366-a081-675f880e63b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 10 21:12:46.499667 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:46.499627    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf" event={"ID":"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc","Type":"ContainerStarted","Data":"9d52883388467d68192e93519dcd524de9691f8da756060a2c6b0bd88e2eae67"}
Sep 10 21:12:46.499867 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:46.499684    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf" event={"ID":"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc","Type":"ContainerStarted","Data":"78eea7c1ae0112cf7a38fadae77dbd8c01be688867536aa999771c075e1e0a79"}
Sep 10 21:12:46.522959 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:46.522893    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf" podStartSLOduration=1.522870596 podStartE2EDuration="1.522870596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:12:46.52108515 +0000 UTC m=+826.186109951" watchObservedRunningTime="2026-09-10 21:12:46.522870596 +0000 UTC m=+826.187895397"
Sep 10 21:12:47.084978 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:47.084944    2584 patch_prober.go:28] interesting pod/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 10 21:12:47.085149 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:47.084999    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt" podUID="c76d25ff-57e2-4366-a081-675f880e63b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 10 21:12:47.446757 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:47.446716    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:47.452479 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:47.452450    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:47.506345 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:47.506281    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:47.512655 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:47.512631    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:12:48.084534 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:48.084496    2584 patch_prober.go:28] interesting pod/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 10 21:12:48.084701 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:48.084555    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt" podUID="c76d25ff-57e2-4366-a081-675f880e63b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 10 21:12:49.088516 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.088490    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:49.088909 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.088854    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:49.089571 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.089552    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:12:49.307413 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.307381    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:12:49.377194 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.377168    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1208af-cb0e-4d1e-a90a-5725e61df387-istio-podinfo\") pod \"ce1208af-cb0e-4d1e-a90a-5725e61df387\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") "
Sep 10 21:12:49.377365 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.377207    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-istio-data\") pod \"ce1208af-cb0e-4d1e-a90a-5725e61df387\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") "
Sep 10 21:12:49.377365 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.377225    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-workload-socket\") pod \"ce1208af-cb0e-4d1e-a90a-5725e61df387\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") "
Sep 10 21:12:49.377365 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.377288    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86tsx\" (UniqueName: \"kubernetes.io/projected/ce1208af-cb0e-4d1e-a90a-5725e61df387-kube-api-access-86tsx\") pod \"ce1208af-cb0e-4d1e-a90a-5725e61df387\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") "
Sep 10 21:12:49.377365 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.377318    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-istio-envoy\") pod \"ce1208af-cb0e-4d1e-a90a-5725e61df387\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") "
Sep 10 21:12:49.377622 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.377390    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1208af-cb0e-4d1e-a90a-5725e61df387-istio-token\") pod \"ce1208af-cb0e-4d1e-a90a-5725e61df387\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") "
Sep 10 21:12:49.377622 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.377416    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1208af-cb0e-4d1e-a90a-5725e61df387-istiod-ca-cert\") pod \"ce1208af-cb0e-4d1e-a90a-5725e61df387\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") "
Sep 10 21:12:49.377622 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.377479    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-credential-socket\") pod \"ce1208af-cb0e-4d1e-a90a-5725e61df387\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") "
Sep 10 21:12:49.377622 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.377484    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-istio-data" (OuterVolumeSpecName: "istio-data") pod "ce1208af-cb0e-4d1e-a90a-5725e61df387" (UID: "ce1208af-cb0e-4d1e-a90a-5725e61df387"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:12:49.377622 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.377586    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-workload-certs\") pod \"ce1208af-cb0e-4d1e-a90a-5725e61df387\" (UID: \"ce1208af-cb0e-4d1e-a90a-5725e61df387\") "
Sep 10 21:12:49.377873 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.377582    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ce1208af-cb0e-4d1e-a90a-5725e61df387" (UID: "ce1208af-cb0e-4d1e-a90a-5725e61df387"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:12:49.377873 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.377753    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ce1208af-cb0e-4d1e-a90a-5725e61df387" (UID: "ce1208af-cb0e-4d1e-a90a-5725e61df387"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:12:49.377977 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.377890    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ce1208af-cb0e-4d1e-a90a-5725e61df387" (UID: "ce1208af-cb0e-4d1e-a90a-5725e61df387"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:12:49.378049 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.378023    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-credential-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:12:49.378104 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.378049    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-workload-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:12:49.378104 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.378064    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-istio-data\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:12:49.378104 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.378072    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce1208af-cb0e-4d1e-a90a-5725e61df387-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ce1208af-cb0e-4d1e-a90a-5725e61df387" (UID: "ce1208af-cb0e-4d1e-a90a-5725e61df387"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:12:49.378104 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.378078    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-workload-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:12:49.379682 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.379652    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce1208af-cb0e-4d1e-a90a-5725e61df387-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ce1208af-cb0e-4d1e-a90a-5725e61df387" (UID: "ce1208af-cb0e-4d1e-a90a-5725e61df387"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:12:49.379795 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.379777    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1208af-cb0e-4d1e-a90a-5725e61df387-kube-api-access-86tsx" (OuterVolumeSpecName: "kube-api-access-86tsx") pod "ce1208af-cb0e-4d1e-a90a-5725e61df387" (UID: "ce1208af-cb0e-4d1e-a90a-5725e61df387"). InnerVolumeSpecName "kube-api-access-86tsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:12:49.380268 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.380242    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1208af-cb0e-4d1e-a90a-5725e61df387-istio-token" (OuterVolumeSpecName: "istio-token") pod "ce1208af-cb0e-4d1e-a90a-5725e61df387" (UID: "ce1208af-cb0e-4d1e-a90a-5725e61df387"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:12:49.380373 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.380336    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ce1208af-cb0e-4d1e-a90a-5725e61df387" (UID: "ce1208af-cb0e-4d1e-a90a-5725e61df387"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:12:49.479572 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.479536    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86tsx\" (UniqueName: \"kubernetes.io/projected/ce1208af-cb0e-4d1e-a90a-5725e61df387-kube-api-access-86tsx\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:12:49.479572 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.479565    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1208af-cb0e-4d1e-a90a-5725e61df387-istio-envoy\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:12:49.479572 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.479574    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1208af-cb0e-4d1e-a90a-5725e61df387-istio-token\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:12:49.479572 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.479582    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1208af-cb0e-4d1e-a90a-5725e61df387-istiod-ca-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:12:49.479832 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.479590    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1208af-cb0e-4d1e-a90a-5725e61df387-istio-podinfo\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:12:49.518025 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.517990    2584 generic.go:356] "Generic (PLEG): container finished" podID="ce1208af-cb0e-4d1e-a90a-5725e61df387" containerID="7fb62e8a2d6354c645f0edfce940b8c9cf2b6686ecb1081e9474f99f1e5111bd" exitCode=0
Sep 10 21:12:49.518199 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.518067    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"
Sep 10 21:12:49.518199 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.518081    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv" event={"ID":"ce1208af-cb0e-4d1e-a90a-5725e61df387","Type":"ContainerDied","Data":"7fb62e8a2d6354c645f0edfce940b8c9cf2b6686ecb1081e9474f99f1e5111bd"}
Sep 10 21:12:49.518199 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.518142    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv" event={"ID":"ce1208af-cb0e-4d1e-a90a-5725e61df387","Type":"ContainerDied","Data":"fb8ae6e55b507589aa10aa511f7fa6019f746af36a1e74cb0c138f10fe12d082"}
Sep 10 21:12:49.518199 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.518170    2584 scope.go:117] "RemoveContainer" containerID="7fb62e8a2d6354c645f0edfce940b8c9cf2b6686ecb1081e9474f99f1e5111bd"
Sep 10 21:12:49.528456 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.528412    2584 scope.go:117] "RemoveContainer" containerID="66625d0bf8e76e58671330a32cafbb14b800a797e1f2ebbc32adba3824b3c265"
Sep 10 21:12:49.536911 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.536893    2584 scope.go:117] "RemoveContainer" containerID="7fb62e8a2d6354c645f0edfce940b8c9cf2b6686ecb1081e9474f99f1e5111bd"
Sep 10 21:12:49.537177 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:12:49.537157    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb62e8a2d6354c645f0edfce940b8c9cf2b6686ecb1081e9474f99f1e5111bd\": container with ID starting with 7fb62e8a2d6354c645f0edfce940b8c9cf2b6686ecb1081e9474f99f1e5111bd not found: ID does not exist" containerID="7fb62e8a2d6354c645f0edfce940b8c9cf2b6686ecb1081e9474f99f1e5111bd"
Sep 10 21:12:49.537258 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.537191    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb62e8a2d6354c645f0edfce940b8c9cf2b6686ecb1081e9474f99f1e5111bd"} err="failed to get container status \"7fb62e8a2d6354c645f0edfce940b8c9cf2b6686ecb1081e9474f99f1e5111bd\": rpc error: code = NotFound desc = could not find container \"7fb62e8a2d6354c645f0edfce940b8c9cf2b6686ecb1081e9474f99f1e5111bd\": container with ID starting with 7fb62e8a2d6354c645f0edfce940b8c9cf2b6686ecb1081e9474f99f1e5111bd not found: ID does not exist"
Sep 10 21:12:49.537258 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.537215    2584 scope.go:117] "RemoveContainer" containerID="66625d0bf8e76e58671330a32cafbb14b800a797e1f2ebbc32adba3824b3c265"
Sep 10 21:12:49.537564 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:12:49.537544    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66625d0bf8e76e58671330a32cafbb14b800a797e1f2ebbc32adba3824b3c265\": container with ID starting with 66625d0bf8e76e58671330a32cafbb14b800a797e1f2ebbc32adba3824b3c265 not found: ID does not exist" containerID="66625d0bf8e76e58671330a32cafbb14b800a797e1f2ebbc32adba3824b3c265"
Sep 10 21:12:49.537631 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.537570    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66625d0bf8e76e58671330a32cafbb14b800a797e1f2ebbc32adba3824b3c265"} err="failed to get container status \"66625d0bf8e76e58671330a32cafbb14b800a797e1f2ebbc32adba3824b3c265\": rpc error: code = NotFound desc = could not find container \"66625d0bf8e76e58671330a32cafbb14b800a797e1f2ebbc32adba3824b3c265\": container with ID starting with 66625d0bf8e76e58671330a32cafbb14b800a797e1f2ebbc32adba3824b3c265 not found: ID does not exist"
Sep 10 21:12:49.543138 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.543118    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"]
Sep 10 21:12:49.547245 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:49.547226    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8086f00e-openshift-default-6fb4c8d5fd-tt5fv"]
Sep 10 21:12:50.841312 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:12:50.841276    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce1208af-cb0e-4d1e-a90a-5725e61df387" path="/var/lib/kubelet/pods/ce1208af-cb0e-4d1e-a90a-5725e61df387/volumes"
Sep 10 21:13:07.363378 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.363341    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"]
Sep 10 21:13:07.364973 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.363941    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce1208af-cb0e-4d1e-a90a-5725e61df387" containerName="istio-proxy"
Sep 10 21:13:07.364973 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.363959    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce1208af-cb0e-4d1e-a90a-5725e61df387" containerName="istio-proxy"
Sep 10 21:13:07.364973 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.363978    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce1208af-cb0e-4d1e-a90a-5725e61df387" containerName="istio-proxy"
Sep 10 21:13:07.364973 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.363984    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce1208af-cb0e-4d1e-a90a-5725e61df387" containerName="istio-proxy"
Sep 10 21:13:07.364973 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.364052    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce1208af-cb0e-4d1e-a90a-5725e61df387" containerName="istio-proxy"
Sep 10 21:13:07.364973 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.364060    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce1208af-cb0e-4d1e-a90a-5725e61df387" containerName="istio-proxy"
Sep 10 21:13:07.365889 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.365869    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:07.368674 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.368650    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-5e2fa5\"/\"default-dockercfg-qdgzb\""
Sep 10 21:13:07.368784 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.368729    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-5e2fa5\"/\"autores-model-5e2fa5-kserve-self-signed-certs\""
Sep 10 21:13:07.368784 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.368772    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-5e2fa5\"/\"kube-root-ca.crt\""
Sep 10 21:13:07.368909 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.368785    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-5e2fa5\"/\"openshift-service-ca.crt\""
Sep 10 21:13:07.377391 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.377370    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"]
Sep 10 21:13:07.466711 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.466674    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca610e0-374c-4895-a3e1-935015e1a861-model-cache\") pod \"autores-model-5e2fa5-kserve-7966c9c587-rrgl7\" (UID: \"4ca610e0-374c-4895-a3e1-935015e1a861\") " pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:07.466866 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.466714    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca610e0-374c-4895-a3e1-935015e1a861-tmp-dir\") pod \"autores-model-5e2fa5-kserve-7966c9c587-rrgl7\" (UID: \"4ca610e0-374c-4895-a3e1-935015e1a861\") " pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:07.466866 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.466794    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k5r7\" (UniqueName: \"kubernetes.io/projected/4ca610e0-374c-4895-a3e1-935015e1a861-kube-api-access-9k5r7\") pod \"autores-model-5e2fa5-kserve-7966c9c587-rrgl7\" (UID: \"4ca610e0-374c-4895-a3e1-935015e1a861\") " pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:07.466959 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.466864    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca610e0-374c-4895-a3e1-935015e1a861-home\") pod \"autores-model-5e2fa5-kserve-7966c9c587-rrgl7\" (UID: \"4ca610e0-374c-4895-a3e1-935015e1a861\") " pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:07.467001 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.466971    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca610e0-374c-4895-a3e1-935015e1a861-dshm\") pod \"autores-model-5e2fa5-kserve-7966c9c587-rrgl7\" (UID: \"4ca610e0-374c-4895-a3e1-935015e1a861\") " pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:07.467053 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.467040    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca610e0-374c-4895-a3e1-935015e1a861-tls-certs\") pod \"autores-model-5e2fa5-kserve-7966c9c587-rrgl7\" (UID: \"4ca610e0-374c-4895-a3e1-935015e1a861\") " pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:07.568497 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.568379    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k5r7\" (UniqueName: \"kubernetes.io/projected/4ca610e0-374c-4895-a3e1-935015e1a861-kube-api-access-9k5r7\") pod \"autores-model-5e2fa5-kserve-7966c9c587-rrgl7\" (UID: \"4ca610e0-374c-4895-a3e1-935015e1a861\") " pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:07.568497 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.568451    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca610e0-374c-4895-a3e1-935015e1a861-home\") pod \"autores-model-5e2fa5-kserve-7966c9c587-rrgl7\" (UID: \"4ca610e0-374c-4895-a3e1-935015e1a861\") " pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:07.568718 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.568561    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca610e0-374c-4895-a3e1-935015e1a861-dshm\") pod \"autores-model-5e2fa5-kserve-7966c9c587-rrgl7\" (UID: \"4ca610e0-374c-4895-a3e1-935015e1a861\") " pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:07.568718 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.568633    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca610e0-374c-4895-a3e1-935015e1a861-tls-certs\") pod \"autores-model-5e2fa5-kserve-7966c9c587-rrgl7\" (UID: \"4ca610e0-374c-4895-a3e1-935015e1a861\") " pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:07.568718 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.568677    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca610e0-374c-4895-a3e1-935015e1a861-model-cache\") pod \"autores-model-5e2fa5-kserve-7966c9c587-rrgl7\" (UID: \"4ca610e0-374c-4895-a3e1-935015e1a861\") " pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:07.568718 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.568702    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca610e0-374c-4895-a3e1-935015e1a861-tmp-dir\") pod \"autores-model-5e2fa5-kserve-7966c9c587-rrgl7\" (UID: \"4ca610e0-374c-4895-a3e1-935015e1a861\") " pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:07.568920 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.568819    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca610e0-374c-4895-a3e1-935015e1a861-home\") pod \"autores-model-5e2fa5-kserve-7966c9c587-rrgl7\" (UID: \"4ca610e0-374c-4895-a3e1-935015e1a861\") " pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:07.568981 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.568945    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca610e0-374c-4895-a3e1-935015e1a861-model-cache\") pod \"autores-model-5e2fa5-kserve-7966c9c587-rrgl7\" (UID: \"4ca610e0-374c-4895-a3e1-935015e1a861\") " pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:07.569073 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.569055    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca610e0-374c-4895-a3e1-935015e1a861-tmp-dir\") pod \"autores-model-5e2fa5-kserve-7966c9c587-rrgl7\" (UID: \"4ca610e0-374c-4895-a3e1-935015e1a861\") " pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:07.570750 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.570732    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca610e0-374c-4895-a3e1-935015e1a861-dshm\") pod \"autores-model-5e2fa5-kserve-7966c9c587-rrgl7\" (UID: \"4ca610e0-374c-4895-a3e1-935015e1a861\") " pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:07.570923 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.570907    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca610e0-374c-4895-a3e1-935015e1a861-tls-certs\") pod \"autores-model-5e2fa5-kserve-7966c9c587-rrgl7\" (UID: \"4ca610e0-374c-4895-a3e1-935015e1a861\") " pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:07.576636 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.576616    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k5r7\" (UniqueName: \"kubernetes.io/projected/4ca610e0-374c-4895-a3e1-935015e1a861-kube-api-access-9k5r7\") pod \"autores-model-5e2fa5-kserve-7966c9c587-rrgl7\" (UID: \"4ca610e0-374c-4895-a3e1-935015e1a861\") " pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:07.608100 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.608071    2584 generic.go:356] "Generic (PLEG): container finished" podID="a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc" containerID="9d52883388467d68192e93519dcd524de9691f8da756060a2c6b0bd88e2eae67" exitCode=0
Sep 10 21:13:07.608211 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.608143    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf" event={"ID":"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc","Type":"ContainerDied","Data":"9d52883388467d68192e93519dcd524de9691f8da756060a2c6b0bd88e2eae67"}
Sep 10 21:13:07.608522 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.608508    2584 scope.go:117] "RemoveContainer" containerID="9d52883388467d68192e93519dcd524de9691f8da756060a2c6b0bd88e2eae67"
Sep 10 21:13:07.609716 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.609689    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:13:07.609780 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.609759    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:13:07.609819 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.609794    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:13:07.678297 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.678267    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:07.959138 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:07.959113    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"]
Sep 10 21:13:07.961287 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:13:07.961264    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ca610e0_374c_4895_a3e1_935015e1a861.slice/crio-8eb8deee203cc82fdb55bd3703c81a9c0a3b77208855a5cec2f668e1314ab8a1 WatchSource:0}: Error finding container 8eb8deee203cc82fdb55bd3703c81a9c0a3b77208855a5cec2f668e1314ab8a1: Status 404 returned error can't find the container with id 8eb8deee203cc82fdb55bd3703c81a9c0a3b77208855a5cec2f668e1314ab8a1
Sep 10 21:13:08.618707 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:08.618669    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf" event={"ID":"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc","Type":"ContainerStarted","Data":"b7cad2087783bdf7fb1e1dbe7dd81b4aa9cafd13c8ca15bd9dc1096bc3bdf249"}
Sep 10 21:13:08.621692 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:08.621661    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7" event={"ID":"4ca610e0-374c-4895-a3e1-935015e1a861","Type":"ContainerStarted","Data":"f6cd85f3234ec06706d51bfc813bb7bf30e6fe642a9b409e3957f526744aaa23"}
Sep 10 21:13:08.621890 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:08.621873    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7" event={"ID":"4ca610e0-374c-4895-a3e1-935015e1a861","Type":"ContainerStarted","Data":"8eb8deee203cc82fdb55bd3703c81a9c0a3b77208855a5cec2f668e1314ab8a1"}
Sep 10 21:13:08.653864 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:08.653815    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7" podStartSLOduration=1.653801166 podStartE2EDuration="1.653801166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:13:08.652246878 +0000 UTC m=+848.317271677" watchObservedRunningTime="2026-09-10 21:13:08.653801166 +0000 UTC m=+848.318825966"
Sep 10 21:13:09.446562 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:09.446530    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:13:09.446864 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:09.446760    2584 patch_prober.go:28] interesting pod/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 10 21:13:09.446954 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:09.446903    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf" podUID="a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 10 21:13:10.446950 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:10.446909    2584 patch_prober.go:28] interesting pod/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 10 21:13:10.447338 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:10.446971    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf" podUID="a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 10 21:13:11.447268 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:11.447233    2584 patch_prober.go:28] interesting pod/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 10 21:13:11.447775 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:11.447289    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf" podUID="a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 10 21:13:12.446662 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:12.446625    2584 patch_prober.go:28] interesting pod/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 10 21:13:12.446850 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:12.446683    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf" podUID="a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 10 21:13:12.999688 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:12.999637    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-065d3bbf-8z7q6"]
Sep 10 21:13:13.002917 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:13.002887    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-065d3bbf-8z7q6"
Sep 10 21:13:13.005342 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:13.005311    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4gs5\""
Sep 10 21:13:13.005648 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:13.005628    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:13:13.016273 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:13.016243    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-065d3bbf-8z7q6"]
Sep 10 21:13:13.040435 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:13.039265    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c297ecf4-69a8-4a46-8250-373d278a4186-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-065d3bbf-8z7q6\" (UID: \"c297ecf4-69a8-4a46-8250-373d278a4186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-065d3bbf-8z7q6"
Sep 10 21:13:13.142950 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:13.142912    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c297ecf4-69a8-4a46-8250-373d278a4186-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-065d3bbf-8z7q6\" (UID: \"c297ecf4-69a8-4a46-8250-373d278a4186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-065d3bbf-8z7q6"
Sep 10 21:13:13.143575 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:13.143554    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c297ecf4-69a8-4a46-8250-373d278a4186-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-065d3bbf-8z7q6\" (UID: \"c297ecf4-69a8-4a46-8250-373d278a4186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-065d3bbf-8z7q6"
Sep 10 21:13:13.329693 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:13.329585    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-065d3bbf-8z7q6"
Sep 10 21:13:13.447141 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:13.447101    2584 patch_prober.go:28] interesting pod/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 10 21:13:13.447321 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:13.447164    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf" podUID="a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 10 21:13:13.599865 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:13.599769    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-065d3bbf-8z7q6"]
Sep 10 21:13:13.602197 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:13:13.602170    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc297ecf4_69a8_4a46_8250_373d278a4186.slice/crio-ed78c452940e0171d9a003503d7ccedf40d6087b699d90efc79246519338aa9f WatchSource:0}: Error finding container ed78c452940e0171d9a003503d7ccedf40d6087b699d90efc79246519338aa9f: Status 404 returned error can't find the container with id ed78c452940e0171d9a003503d7ccedf40d6087b699d90efc79246519338aa9f
Sep 10 21:13:13.650503 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:13.650471    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-065d3bbf-8z7q6" event={"ID":"c297ecf4-69a8-4a46-8250-373d278a4186","Type":"ContainerStarted","Data":"ed78c452940e0171d9a003503d7ccedf40d6087b699d90efc79246519338aa9f"}
Sep 10 21:13:14.450567 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:14.450540    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:13:14.450942 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:14.450921    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:13:14.451633 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:14.451613    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:13:14.657986 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:14.657896    2584 generic.go:356] "Generic (PLEG): container finished" podID="c297ecf4-69a8-4a46-8250-373d278a4186" containerID="0aee2177c5a7fc19e43d43139a924999fbbf4798cd68da6484fed4a31274308c" exitCode=0
Sep 10 21:13:14.657986 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:14.657954    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-065d3bbf-8z7q6" event={"ID":"c297ecf4-69a8-4a46-8250-373d278a4186","Type":"ContainerDied","Data":"0aee2177c5a7fc19e43d43139a924999fbbf4798cd68da6484fed4a31274308c"}
Sep 10 21:13:15.799583 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:15.799556    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-065d3bbf-8z7q6"
Sep 10 21:13:15.868883 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:15.868854    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c297ecf4-69a8-4a46-8250-373d278a4186-maas-api-service-ca\") pod \"c297ecf4-69a8-4a46-8250-373d278a4186\" (UID: \"c297ecf4-69a8-4a46-8250-373d278a4186\") "
Sep 10 21:13:15.869203 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:15.869177    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c297ecf4-69a8-4a46-8250-373d278a4186-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c297ecf4-69a8-4a46-8250-373d278a4186" (UID: "c297ecf4-69a8-4a46-8250-373d278a4186"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:13:15.969964 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:15.969888    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c297ecf4-69a8-4a46-8250-373d278a4186-maas-api-service-ca\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:13:16.669419 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:16.669395    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-065d3bbf-8z7q6"
Sep 10 21:13:16.669419 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:16.669394    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-065d3bbf-8z7q6" event={"ID":"c297ecf4-69a8-4a46-8250-373d278a4186","Type":"ContainerDied","Data":"ed78c452940e0171d9a003503d7ccedf40d6087b699d90efc79246519338aa9f"}
Sep 10 21:13:16.669637 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:16.669481    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed78c452940e0171d9a003503d7ccedf40d6087b699d90efc79246519338aa9f"
Sep 10 21:13:17.679395 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:17.679361    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:17.679842 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:17.679410    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:17.692009 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:17.691983    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:18.690622 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:18.690597    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:19.758140 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:19.758105    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-065d3bbf-8z7q6"]
Sep 10 21:13:19.760241 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:19.760219    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-065d3bbf-8z7q6"]
Sep 10 21:13:19.896294 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:19.896262    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"]
Sep 10 21:13:19.896532 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:19.896487    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf" podUID="a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc" containerName="istio-proxy" containerID="cri-o://b7cad2087783bdf7fb1e1dbe7dd81b4aa9cafd13c8ca15bd9dc1096bc3bdf249" gracePeriod=30
Sep 10 21:13:20.840714 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:20.840681    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c297ecf4-69a8-4a46-8250-373d278a4186" path="/var/lib/kubelet/pods/c297ecf4-69a8-4a46-8250-373d278a4186/volumes"
Sep 10 21:13:22.456485 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.456447    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"]
Sep 10 21:13:22.457072 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.457048    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c297ecf4-69a8-4a46-8250-373d278a4186" containerName="revoke-keys"
Sep 10 21:13:22.457116 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.457078    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c297ecf4-69a8-4a46-8250-373d278a4186" containerName="revoke-keys"
Sep 10 21:13:22.457199 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.457187    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c297ecf4-69a8-4a46-8250-373d278a4186" containerName="revoke-keys"
Sep 10 21:13:22.459784 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.459763    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:13:22.462286 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.462265    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b22a78\"/\"openshift-service-ca.crt\""
Sep 10 21:13:22.462870 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.462846    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b22a78\"/\"test-model-b22a78-kserve-self-signed-certs\""
Sep 10 21:13:22.462985 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.462907    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b22a78\"/\"default-dockercfg-ffq6m\""
Sep 10 21:13:22.462985 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.462907    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b22a78\"/\"kube-root-ca.crt\""
Sep 10 21:13:22.469336 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.469313    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"]
Sep 10 21:13:22.535754 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.535712    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97fd7ec2-999c-460a-87e5-4dc53d15d224-tmp-dir\") pod \"test-model-b22a78-kserve-78bb494cdf-49nkg\" (UID: \"97fd7ec2-999c-460a-87e5-4dc53d15d224\") " pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:13:22.535754 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.535754    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97fd7ec2-999c-460a-87e5-4dc53d15d224-tls-certs\") pod \"test-model-b22a78-kserve-78bb494cdf-49nkg\" (UID: \"97fd7ec2-999c-460a-87e5-4dc53d15d224\") " pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:13:22.536005 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.535861    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97fd7ec2-999c-460a-87e5-4dc53d15d224-dshm\") pod \"test-model-b22a78-kserve-78bb494cdf-49nkg\" (UID: \"97fd7ec2-999c-460a-87e5-4dc53d15d224\") " pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:13:22.536005 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.535903    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74rx8\" (UniqueName: \"kubernetes.io/projected/97fd7ec2-999c-460a-87e5-4dc53d15d224-kube-api-access-74rx8\") pod \"test-model-b22a78-kserve-78bb494cdf-49nkg\" (UID: \"97fd7ec2-999c-460a-87e5-4dc53d15d224\") " pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:13:22.536005 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.535973    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97fd7ec2-999c-460a-87e5-4dc53d15d224-home\") pod \"test-model-b22a78-kserve-78bb494cdf-49nkg\" (UID: \"97fd7ec2-999c-460a-87e5-4dc53d15d224\") " pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:13:22.536130 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.536007    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97fd7ec2-999c-460a-87e5-4dc53d15d224-model-cache\") pod \"test-model-b22a78-kserve-78bb494cdf-49nkg\" (UID: \"97fd7ec2-999c-460a-87e5-4dc53d15d224\") " pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:13:22.637317 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.637269    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97fd7ec2-999c-460a-87e5-4dc53d15d224-model-cache\") pod \"test-model-b22a78-kserve-78bb494cdf-49nkg\" (UID: \"97fd7ec2-999c-460a-87e5-4dc53d15d224\") " pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:13:22.637513 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.637451    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97fd7ec2-999c-460a-87e5-4dc53d15d224-tmp-dir\") pod \"test-model-b22a78-kserve-78bb494cdf-49nkg\" (UID: \"97fd7ec2-999c-460a-87e5-4dc53d15d224\") " pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:13:22.637513 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.637489    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97fd7ec2-999c-460a-87e5-4dc53d15d224-tls-certs\") pod \"test-model-b22a78-kserve-78bb494cdf-49nkg\" (UID: \"97fd7ec2-999c-460a-87e5-4dc53d15d224\") " pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:13:22.637604 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.637573    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97fd7ec2-999c-460a-87e5-4dc53d15d224-dshm\") pod \"test-model-b22a78-kserve-78bb494cdf-49nkg\" (UID: \"97fd7ec2-999c-460a-87e5-4dc53d15d224\") " pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:13:22.637640 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.637604    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74rx8\" (UniqueName: \"kubernetes.io/projected/97fd7ec2-999c-460a-87e5-4dc53d15d224-kube-api-access-74rx8\") pod \"test-model-b22a78-kserve-78bb494cdf-49nkg\" (UID: \"97fd7ec2-999c-460a-87e5-4dc53d15d224\") " pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:13:22.637685 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.637659    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97fd7ec2-999c-460a-87e5-4dc53d15d224-home\") pod \"test-model-b22a78-kserve-78bb494cdf-49nkg\" (UID: \"97fd7ec2-999c-460a-87e5-4dc53d15d224\") " pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:13:22.637778 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.637754    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97fd7ec2-999c-460a-87e5-4dc53d15d224-model-cache\") pod \"test-model-b22a78-kserve-78bb494cdf-49nkg\" (UID: \"97fd7ec2-999c-460a-87e5-4dc53d15d224\") " pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:13:22.637778 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.637770    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97fd7ec2-999c-460a-87e5-4dc53d15d224-tmp-dir\") pod \"test-model-b22a78-kserve-78bb494cdf-49nkg\" (UID: \"97fd7ec2-999c-460a-87e5-4dc53d15d224\") " pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:13:22.638035 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.638011    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97fd7ec2-999c-460a-87e5-4dc53d15d224-home\") pod \"test-model-b22a78-kserve-78bb494cdf-49nkg\" (UID: \"97fd7ec2-999c-460a-87e5-4dc53d15d224\") " pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:13:22.639950 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.639923    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97fd7ec2-999c-460a-87e5-4dc53d15d224-dshm\") pod \"test-model-b22a78-kserve-78bb494cdf-49nkg\" (UID: \"97fd7ec2-999c-460a-87e5-4dc53d15d224\") " pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:13:22.640171 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.640154    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97fd7ec2-999c-460a-87e5-4dc53d15d224-tls-certs\") pod \"test-model-b22a78-kserve-78bb494cdf-49nkg\" (UID: \"97fd7ec2-999c-460a-87e5-4dc53d15d224\") " pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:13:22.645776 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.645747    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74rx8\" (UniqueName: \"kubernetes.io/projected/97fd7ec2-999c-460a-87e5-4dc53d15d224-kube-api-access-74rx8\") pod \"test-model-b22a78-kserve-78bb494cdf-49nkg\" (UID: \"97fd7ec2-999c-460a-87e5-4dc53d15d224\") " pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:13:22.770166 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:22.770076    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:13:23.067828 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:23.067749    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"]
Sep 10 21:13:23.070797 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:13:23.070772    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97fd7ec2_999c_460a_87e5_4dc53d15d224.slice/crio-22ee44e1721282b23974d7e69c874b1465765bbe7227df744b232be987094316 WatchSource:0}: Error finding container 22ee44e1721282b23974d7e69c874b1465765bbe7227df744b232be987094316: Status 404 returned error can't find the container with id 22ee44e1721282b23974d7e69c874b1465765bbe7227df744b232be987094316
Sep 10 21:13:23.706285 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:23.706243    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg" event={"ID":"97fd7ec2-999c-460a-87e5-4dc53d15d224","Type":"ContainerStarted","Data":"9c1d87b9ccbdb7600bd0e1cbf6f98bd8f49856a34cf2c6bc628dedc71bfe30e6"}
Sep 10 21:13:23.706285 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:23.706290    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg" event={"ID":"97fd7ec2-999c-460a-87e5-4dc53d15d224","Type":"ContainerStarted","Data":"22ee44e1721282b23974d7e69c874b1465765bbe7227df744b232be987094316"}
Sep 10 21:13:23.725804 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:23.725751    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg" podStartSLOduration=1.7257355589999999 podStartE2EDuration="1.725735559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:13:23.7236981 +0000 UTC m=+863.388722900" watchObservedRunningTime="2026-09-10 21:13:23.725735559 +0000 UTC m=+863.390760359"
Sep 10 21:13:25.166503 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.166155    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:13:25.267726 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.267693    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8l2f\" (UniqueName: \"kubernetes.io/projected/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-kube-api-access-f8l2f\") pod \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") "
Sep 10 21:13:25.267885 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.267763    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istio-token\") pod \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") "
Sep 10 21:13:25.267885 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.267791    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istio-data\") pod \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") "
Sep 10 21:13:25.267885 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.267845    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-workload-socket\") pod \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") "
Sep 10 21:13:25.268020 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.267908    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-workload-certs\") pod \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") "
Sep 10 21:13:25.268020 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.267935    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-credential-socket\") pod \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") "
Sep 10 21:13:25.268020 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.267965    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istiod-ca-cert\") pod \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") "
Sep 10 21:13:25.268020 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.268004    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istio-envoy\") pod \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") "
Sep 10 21:13:25.268224 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.268029    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istio-podinfo\") pod \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\" (UID: \"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc\") "
Sep 10 21:13:25.268224 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.268093    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istio-data" (OuterVolumeSpecName: "istio-data") pod "a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc" (UID: "a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:25.268224 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.268214    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc" (UID: "a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:25.268362 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.268227    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc" (UID: "a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:25.268528 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.268470    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-workload-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:13:25.268528 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.268490    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc" (UID: "a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:13:25.268528 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.268499    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-workload-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:13:25.268793 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.268536    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istio-data\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:13:25.268793 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.268664    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc" (UID: "a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:25.270016 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.269987    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istio-token" (OuterVolumeSpecName: "istio-token") pod "a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc" (UID: "a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:13:25.270098 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.270041    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-kube-api-access-f8l2f" (OuterVolumeSpecName: "kube-api-access-f8l2f") pod "a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc" (UID: "a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc"). InnerVolumeSpecName "kube-api-access-f8l2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:13:25.270187 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.270169    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc" (UID: "a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:13:25.270276 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.270257    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc" (UID: "a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:25.369272 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.369243    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8l2f\" (UniqueName: \"kubernetes.io/projected/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-kube-api-access-f8l2f\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:13:25.369272 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.369270    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istio-token\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:13:25.369272 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.369279    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-credential-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:13:25.369511 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.369289    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istiod-ca-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:13:25.369511 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.369298    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istio-envoy\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:13:25.369511 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.369305    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc-istio-podinfo\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:13:25.717279 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.717247    2584 generic.go:356] "Generic (PLEG): container finished" podID="a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc" containerID="b7cad2087783bdf7fb1e1dbe7dd81b4aa9cafd13c8ca15bd9dc1096bc3bdf249" exitCode=0
Sep 10 21:13:25.717492 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.717296    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"
Sep 10 21:13:25.717492 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.717329    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf" event={"ID":"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc","Type":"ContainerDied","Data":"b7cad2087783bdf7fb1e1dbe7dd81b4aa9cafd13c8ca15bd9dc1096bc3bdf249"}
Sep 10 21:13:25.717492 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.717396    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf" event={"ID":"a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc","Type":"ContainerDied","Data":"78eea7c1ae0112cf7a38fadae77dbd8c01be688867536aa999771c075e1e0a79"}
Sep 10 21:13:25.717492 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.717443    2584 scope.go:117] "RemoveContainer" containerID="b7cad2087783bdf7fb1e1dbe7dd81b4aa9cafd13c8ca15bd9dc1096bc3bdf249"
Sep 10 21:13:25.729014 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.728990    2584 scope.go:117] "RemoveContainer" containerID="9d52883388467d68192e93519dcd524de9691f8da756060a2c6b0bd88e2eae67"
Sep 10 21:13:25.737030 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.737009    2584 scope.go:117] "RemoveContainer" containerID="b7cad2087783bdf7fb1e1dbe7dd81b4aa9cafd13c8ca15bd9dc1096bc3bdf249"
Sep 10 21:13:25.737288 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:13:25.737269    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7cad2087783bdf7fb1e1dbe7dd81b4aa9cafd13c8ca15bd9dc1096bc3bdf249\": container with ID starting with b7cad2087783bdf7fb1e1dbe7dd81b4aa9cafd13c8ca15bd9dc1096bc3bdf249 not found: ID does not exist" containerID="b7cad2087783bdf7fb1e1dbe7dd81b4aa9cafd13c8ca15bd9dc1096bc3bdf249"
Sep 10 21:13:25.737348 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.737295    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7cad2087783bdf7fb1e1dbe7dd81b4aa9cafd13c8ca15bd9dc1096bc3bdf249"} err="failed to get container status \"b7cad2087783bdf7fb1e1dbe7dd81b4aa9cafd13c8ca15bd9dc1096bc3bdf249\": rpc error: code = NotFound desc = could not find container \"b7cad2087783bdf7fb1e1dbe7dd81b4aa9cafd13c8ca15bd9dc1096bc3bdf249\": container with ID starting with b7cad2087783bdf7fb1e1dbe7dd81b4aa9cafd13c8ca15bd9dc1096bc3bdf249 not found: ID does not exist"
Sep 10 21:13:25.737348 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.737312    2584 scope.go:117] "RemoveContainer" containerID="9d52883388467d68192e93519dcd524de9691f8da756060a2c6b0bd88e2eae67"
Sep 10 21:13:25.737601 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:13:25.737583    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d52883388467d68192e93519dcd524de9691f8da756060a2c6b0bd88e2eae67\": container with ID starting with 9d52883388467d68192e93519dcd524de9691f8da756060a2c6b0bd88e2eae67 not found: ID does not exist" containerID="9d52883388467d68192e93519dcd524de9691f8da756060a2c6b0bd88e2eae67"
Sep 10 21:13:25.737650 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.737609    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d52883388467d68192e93519dcd524de9691f8da756060a2c6b0bd88e2eae67"} err="failed to get container status \"9d52883388467d68192e93519dcd524de9691f8da756060a2c6b0bd88e2eae67\": rpc error: code = NotFound desc = could not find container \"9d52883388467d68192e93519dcd524de9691f8da756060a2c6b0bd88e2eae67\": container with ID starting with 9d52883388467d68192e93519dcd524de9691f8da756060a2c6b0bd88e2eae67 not found: ID does not exist"
Sep 10 21:13:25.741627 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.741606    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"]
Sep 10 21:13:25.743411 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:25.743391    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-065d3bbf-gw-openshift-default-95fdd7768-7cmgf"]
Sep 10 21:13:26.844048 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:26.844013    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc" path="/var/lib/kubelet/pods/a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc/volumes"
Sep 10 21:13:32.770203 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:32.770171    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:13:32.770796 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:32.770222    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:13:32.782805 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:32.782779    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:13:33.778542 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:33.778513    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:13:34.428773 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:34.428738    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"]
Sep 10 21:13:34.563820 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:34.563780    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5e2fa5-vns2p"]
Sep 10 21:13:34.564274 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:34.564255    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc" containerName="istio-proxy"
Sep 10 21:13:34.564274 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:34.564273    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc" containerName="istio-proxy"
Sep 10 21:13:34.564457 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:34.564304    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc" containerName="istio-proxy"
Sep 10 21:13:34.564457 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:34.564310    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc" containerName="istio-proxy"
Sep 10 21:13:34.564457 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:34.564374    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc" containerName="istio-proxy"
Sep 10 21:13:34.567690 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:34.567669    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5e2fa5-vns2p"
Sep 10 21:13:34.570173 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:34.570149    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:13:34.570289 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:34.570259    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4gs5\""
Sep 10 21:13:34.576612 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:34.576577    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5e2fa5-vns2p"]
Sep 10 21:13:34.667624 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:34.667592    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d717750d-b08a-49c1-935e-3ce0a1a2edb5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-5e2fa5-vns2p\" (UID: \"d717750d-b08a-49c1-935e-3ce0a1a2edb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5e2fa5-vns2p"
Sep 10 21:13:34.770907 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:34.770810    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d717750d-b08a-49c1-935e-3ce0a1a2edb5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-5e2fa5-vns2p\" (UID: \"d717750d-b08a-49c1-935e-3ce0a1a2edb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5e2fa5-vns2p"
Sep 10 21:13:34.771550 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:34.771529    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d717750d-b08a-49c1-935e-3ce0a1a2edb5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-5e2fa5-vns2p\" (UID: \"d717750d-b08a-49c1-935e-3ce0a1a2edb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5e2fa5-vns2p"
Sep 10 21:13:34.879520 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:34.879489    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5e2fa5-vns2p"
Sep 10 21:13:35.149874 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:35.149839    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5e2fa5-vns2p"]
Sep 10 21:13:35.153091 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:13:35.153058    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd717750d_b08a_49c1_935e_3ce0a1a2edb5.slice/crio-ed916f62a5570e1bf4cc68d35d783f2046ae2cd9c9dd573c60465053db6beafb WatchSource:0}: Error finding container ed916f62a5570e1bf4cc68d35d783f2046ae2cd9c9dd573c60465053db6beafb: Status 404 returned error can't find the container with id ed916f62a5570e1bf4cc68d35d783f2046ae2cd9c9dd573c60465053db6beafb
Sep 10 21:13:35.446409 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:35.446375    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"]
Sep 10 21:13:35.778031 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:35.777944    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5e2fa5-vns2p" event={"ID":"d717750d-b08a-49c1-935e-3ce0a1a2edb5","Type":"ContainerStarted","Data":"ce55d03a794a334ef8569aa200dc63cc0eb6353cd263ac084bfa2efc758b2cac"}
Sep 10 21:13:35.778031 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:35.777999    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5e2fa5-vns2p" event={"ID":"d717750d-b08a-49c1-935e-3ce0a1a2edb5","Type":"ContainerStarted","Data":"ed916f62a5570e1bf4cc68d35d783f2046ae2cd9c9dd573c60465053db6beafb"}
Sep 10 21:13:35.794066 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:35.794017    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5e2fa5-vns2p" podStartSLOduration=1.7940036529999999 podStartE2EDuration="1.794003653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:13:35.792781182 +0000 UTC m=+875.457805993" watchObservedRunningTime="2026-09-10 21:13:35.794003653 +0000 UTC m=+875.459028453"
Sep 10 21:13:36.784544 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:36.784508    2584 generic.go:356] "Generic (PLEG): container finished" podID="d717750d-b08a-49c1-935e-3ce0a1a2edb5" containerID="ce55d03a794a334ef8569aa200dc63cc0eb6353cd263ac084bfa2efc758b2cac" exitCode=0
Sep 10 21:13:36.784977 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:36.784584    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5e2fa5-vns2p" event={"ID":"d717750d-b08a-49c1-935e-3ce0a1a2edb5","Type":"ContainerDied","Data":"ce55d03a794a334ef8569aa200dc63cc0eb6353cd263ac084bfa2efc758b2cac"}
Sep 10 21:13:37.922182 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:37.922156    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5e2fa5-vns2p"
Sep 10 21:13:38.105291 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:38.105202    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d717750d-b08a-49c1-935e-3ce0a1a2edb5-maas-api-service-ca\") pod \"d717750d-b08a-49c1-935e-3ce0a1a2edb5\" (UID: \"d717750d-b08a-49c1-935e-3ce0a1a2edb5\") "
Sep 10 21:13:38.105573 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:38.105547    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d717750d-b08a-49c1-935e-3ce0a1a2edb5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d717750d-b08a-49c1-935e-3ce0a1a2edb5" (UID: "d717750d-b08a-49c1-935e-3ce0a1a2edb5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:13:38.206801 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:38.206764    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d717750d-b08a-49c1-935e-3ce0a1a2edb5-maas-api-service-ca\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:13:38.796517 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:38.796481    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5e2fa5-vns2p"
Sep 10 21:13:38.796517 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:38.796493    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5e2fa5-vns2p" event={"ID":"d717750d-b08a-49c1-935e-3ce0a1a2edb5","Type":"ContainerDied","Data":"ed916f62a5570e1bf4cc68d35d783f2046ae2cd9c9dd573c60465053db6beafb"}
Sep 10 21:13:38.797041 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:38.796543    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed916f62a5570e1bf4cc68d35d783f2046ae2cd9c9dd573c60465053db6beafb"
Sep 10 21:13:45.349772 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:45.349734    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5e2fa5-vns2p"]
Sep 10 21:13:45.352871 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:45.352833    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5e2fa5-vns2p"]
Sep 10 21:13:46.843948 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:46.843909    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d717750d-b08a-49c1-935e-3ce0a1a2edb5" path="/var/lib/kubelet/pods/d717750d-b08a-49c1-935e-3ce0a1a2edb5/volumes"
Sep 10 21:13:49.173762 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.173726    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-169213ba-r4trd"]
Sep 10 21:13:49.174301 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.174279    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d717750d-b08a-49c1-935e-3ce0a1a2edb5" containerName="revoke-keys"
Sep 10 21:13:49.174385 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.174303    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d717750d-b08a-49c1-935e-3ce0a1a2edb5" containerName="revoke-keys"
Sep 10 21:13:49.174477 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.174410    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7a139c2-bbf1-4f3e-a5c8-4ecc0e5da0cc" containerName="istio-proxy"
Sep 10 21:13:49.174477 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.174439    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d717750d-b08a-49c1-935e-3ce0a1a2edb5" containerName="revoke-keys"
Sep 10 21:13:49.178727 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.178707    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-169213ba-r4trd"
Sep 10 21:13:49.180994 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.180970    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:13:49.181249 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.181228    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4gs5\""
Sep 10 21:13:49.184885 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.184863    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-169213ba-r4trd"]
Sep 10 21:13:49.215139 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.215109    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d88a8f0-33e6-42fb-8641-39e520983961-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-169213ba-r4trd\" (UID: \"9d88a8f0-33e6-42fb-8641-39e520983961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-169213ba-r4trd"
Sep 10 21:13:49.316858 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.316827    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d88a8f0-33e6-42fb-8641-39e520983961-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-169213ba-r4trd\" (UID: \"9d88a8f0-33e6-42fb-8641-39e520983961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-169213ba-r4trd"
Sep 10 21:13:49.317411 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.317389    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d88a8f0-33e6-42fb-8641-39e520983961-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-169213ba-r4trd\" (UID: \"9d88a8f0-33e6-42fb-8641-39e520983961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-169213ba-r4trd"
Sep 10 21:13:49.451266 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.451176    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7" podUID="4ca610e0-374c-4895-a3e1-935015e1a861" containerName="main" containerID="cri-o://f6cd85f3234ec06706d51bfc813bb7bf30e6fe642a9b409e3957f526744aaa23" gracePeriod=45
Sep 10 21:13:49.507995 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.507971    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-169213ba-r4trd"
Sep 10 21:13:49.705391 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.705329    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:49.777441 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.777392    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-169213ba-r4trd"]
Sep 10 21:13:49.780382 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:13:49.780355    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d88a8f0_33e6_42fb_8641_39e520983961.slice/crio-090cbabe2cc1dd0ad36e2f4aa872bf437f1390e82b81c64aaba11fc34ce2e9d5 WatchSource:0}: Error finding container 090cbabe2cc1dd0ad36e2f4aa872bf437f1390e82b81c64aaba11fc34ce2e9d5: Status 404 returned error can't find the container with id 090cbabe2cc1dd0ad36e2f4aa872bf437f1390e82b81c64aaba11fc34ce2e9d5
Sep 10 21:13:49.821300 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.821275    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca610e0-374c-4895-a3e1-935015e1a861-home\") pod \"4ca610e0-374c-4895-a3e1-935015e1a861\" (UID: \"4ca610e0-374c-4895-a3e1-935015e1a861\") "
Sep 10 21:13:49.821412 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.821321    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca610e0-374c-4895-a3e1-935015e1a861-model-cache\") pod \"4ca610e0-374c-4895-a3e1-935015e1a861\" (UID: \"4ca610e0-374c-4895-a3e1-935015e1a861\") "
Sep 10 21:13:49.821412 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.821371    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca610e0-374c-4895-a3e1-935015e1a861-dshm\") pod \"4ca610e0-374c-4895-a3e1-935015e1a861\" (UID: \"4ca610e0-374c-4895-a3e1-935015e1a861\") "
Sep 10 21:13:49.821412 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.821400    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca610e0-374c-4895-a3e1-935015e1a861-tmp-dir\") pod \"4ca610e0-374c-4895-a3e1-935015e1a861\" (UID: \"4ca610e0-374c-4895-a3e1-935015e1a861\") "
Sep 10 21:13:49.821596 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.821465    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9k5r7\" (UniqueName: \"kubernetes.io/projected/4ca610e0-374c-4895-a3e1-935015e1a861-kube-api-access-9k5r7\") pod \"4ca610e0-374c-4895-a3e1-935015e1a861\" (UID: \"4ca610e0-374c-4895-a3e1-935015e1a861\") "
Sep 10 21:13:49.821596 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.821535    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca610e0-374c-4895-a3e1-935015e1a861-tls-certs\") pod \"4ca610e0-374c-4895-a3e1-935015e1a861\" (UID: \"4ca610e0-374c-4895-a3e1-935015e1a861\") "
Sep 10 21:13:49.821738 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.821533    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca610e0-374c-4895-a3e1-935015e1a861-home" (OuterVolumeSpecName: "home") pod "4ca610e0-374c-4895-a3e1-935015e1a861" (UID: "4ca610e0-374c-4895-a3e1-935015e1a861"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:49.821798 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.821643    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca610e0-374c-4895-a3e1-935015e1a861-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ca610e0-374c-4895-a3e1-935015e1a861" (UID: "4ca610e0-374c-4895-a3e1-935015e1a861"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:49.821798 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.821693    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca610e0-374c-4895-a3e1-935015e1a861-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ca610e0-374c-4895-a3e1-935015e1a861" (UID: "4ca610e0-374c-4895-a3e1-935015e1a861"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:49.821955 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.821936    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca610e0-374c-4895-a3e1-935015e1a861-home\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:13:49.822010 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.821962    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca610e0-374c-4895-a3e1-935015e1a861-model-cache\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:13:49.822010 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.821977    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca610e0-374c-4895-a3e1-935015e1a861-tmp-dir\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:13:49.823553 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.823532    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca610e0-374c-4895-a3e1-935015e1a861-dshm" (OuterVolumeSpecName: "dshm") pod "4ca610e0-374c-4895-a3e1-935015e1a861" (UID: "4ca610e0-374c-4895-a3e1-935015e1a861"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:49.823637 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.823573    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ca610e0-374c-4895-a3e1-935015e1a861-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ca610e0-374c-4895-a3e1-935015e1a861" (UID: "4ca610e0-374c-4895-a3e1-935015e1a861"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:13:49.823637 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.823597    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca610e0-374c-4895-a3e1-935015e1a861-kube-api-access-9k5r7" (OuterVolumeSpecName: "kube-api-access-9k5r7") pod "4ca610e0-374c-4895-a3e1-935015e1a861" (UID: "4ca610e0-374c-4895-a3e1-935015e1a861"). InnerVolumeSpecName "kube-api-access-9k5r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:13:49.859898 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.859866    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-169213ba-r4trd" event={"ID":"9d88a8f0-33e6-42fb-8641-39e520983961","Type":"ContainerStarted","Data":"090cbabe2cc1dd0ad36e2f4aa872bf437f1390e82b81c64aaba11fc34ce2e9d5"}
Sep 10 21:13:49.861199 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.861174    2584 generic.go:356] "Generic (PLEG): container finished" podID="4ca610e0-374c-4895-a3e1-935015e1a861" containerID="f6cd85f3234ec06706d51bfc813bb7bf30e6fe642a9b409e3957f526744aaa23" exitCode=0
Sep 10 21:13:49.861307 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.861220    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"
Sep 10 21:13:49.861366 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.861225    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7" event={"ID":"4ca610e0-374c-4895-a3e1-935015e1a861","Type":"ContainerDied","Data":"f6cd85f3234ec06706d51bfc813bb7bf30e6fe642a9b409e3957f526744aaa23"}
Sep 10 21:13:49.861366 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.861349    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7" event={"ID":"4ca610e0-374c-4895-a3e1-935015e1a861","Type":"ContainerDied","Data":"8eb8deee203cc82fdb55bd3703c81a9c0a3b77208855a5cec2f668e1314ab8a1"}
Sep 10 21:13:49.861504 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.861372    2584 scope.go:117] "RemoveContainer" containerID="f6cd85f3234ec06706d51bfc813bb7bf30e6fe642a9b409e3957f526744aaa23"
Sep 10 21:13:49.870079 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.870062    2584 scope.go:117] "RemoveContainer" containerID="f6cd85f3234ec06706d51bfc813bb7bf30e6fe642a9b409e3957f526744aaa23"
Sep 10 21:13:49.870333 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:13:49.870314    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6cd85f3234ec06706d51bfc813bb7bf30e6fe642a9b409e3957f526744aaa23\": container with ID starting with f6cd85f3234ec06706d51bfc813bb7bf30e6fe642a9b409e3957f526744aaa23 not found: ID does not exist" containerID="f6cd85f3234ec06706d51bfc813bb7bf30e6fe642a9b409e3957f526744aaa23"
Sep 10 21:13:49.870384 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.870341    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6cd85f3234ec06706d51bfc813bb7bf30e6fe642a9b409e3957f526744aaa23"} err="failed to get container status \"f6cd85f3234ec06706d51bfc813bb7bf30e6fe642a9b409e3957f526744aaa23\": rpc error: code = NotFound desc = could not find container \"f6cd85f3234ec06706d51bfc813bb7bf30e6fe642a9b409e3957f526744aaa23\": container with ID starting with f6cd85f3234ec06706d51bfc813bb7bf30e6fe642a9b409e3957f526744aaa23 not found: ID does not exist"
Sep 10 21:13:49.881387 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.881363    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"]
Sep 10 21:13:49.885659 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.885631    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-5e2fa5/autores-model-5e2fa5-kserve-7966c9c587-rrgl7"]
Sep 10 21:13:49.923260 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.923239    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca610e0-374c-4895-a3e1-935015e1a861-tls-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:13:49.923395 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.923378    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca610e0-374c-4895-a3e1-935015e1a861-dshm\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:13:49.923500 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:49.923401    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k5r7\" (UniqueName: \"kubernetes.io/projected/4ca610e0-374c-4895-a3e1-935015e1a861-kube-api-access-9k5r7\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:13:50.465888 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.465845    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg" podUID="9b71d537-0290-43f9-9277-5f2c946a55b4" containerName="main" containerID="cri-o://29161a3678342f4b4babe553cded88cfb67e1071c20f074372d58fc1b265c49e" gracePeriod=45
Sep 10 21:13:50.706164 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.706138    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:13:50.731338 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.731271    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b71d537-0290-43f9-9277-5f2c946a55b4-tls-certs\") pod \"9b71d537-0290-43f9-9277-5f2c946a55b4\" (UID: \"9b71d537-0290-43f9-9277-5f2c946a55b4\") "
Sep 10 21:13:50.731338 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.731308    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b71d537-0290-43f9-9277-5f2c946a55b4-dshm\") pod \"9b71d537-0290-43f9-9277-5f2c946a55b4\" (UID: \"9b71d537-0290-43f9-9277-5f2c946a55b4\") "
Sep 10 21:13:50.731338 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.731329    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b71d537-0290-43f9-9277-5f2c946a55b4-tmp-dir\") pod \"9b71d537-0290-43f9-9277-5f2c946a55b4\" (UID: \"9b71d537-0290-43f9-9277-5f2c946a55b4\") "
Sep 10 21:13:50.731606 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.731355    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b71d537-0290-43f9-9277-5f2c946a55b4-model-cache\") pod \"9b71d537-0290-43f9-9277-5f2c946a55b4\" (UID: \"9b71d537-0290-43f9-9277-5f2c946a55b4\") "
Sep 10 21:13:50.731606 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.731504    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6r5d\" (UniqueName: \"kubernetes.io/projected/9b71d537-0290-43f9-9277-5f2c946a55b4-kube-api-access-l6r5d\") pod \"9b71d537-0290-43f9-9277-5f2c946a55b4\" (UID: \"9b71d537-0290-43f9-9277-5f2c946a55b4\") "
Sep 10 21:13:50.731606 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.731548    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b71d537-0290-43f9-9277-5f2c946a55b4-home\") pod \"9b71d537-0290-43f9-9277-5f2c946a55b4\" (UID: \"9b71d537-0290-43f9-9277-5f2c946a55b4\") "
Sep 10 21:13:50.731772 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.731683    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b71d537-0290-43f9-9277-5f2c946a55b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b71d537-0290-43f9-9277-5f2c946a55b4" (UID: "9b71d537-0290-43f9-9277-5f2c946a55b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:50.731829 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.731793    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b71d537-0290-43f9-9277-5f2c946a55b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b71d537-0290-43f9-9277-5f2c946a55b4" (UID: "9b71d537-0290-43f9-9277-5f2c946a55b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:50.731966 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.731939    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b71d537-0290-43f9-9277-5f2c946a55b4-home" (OuterVolumeSpecName: "home") pod "9b71d537-0290-43f9-9277-5f2c946a55b4" (UID: "9b71d537-0290-43f9-9277-5f2c946a55b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:50.732077 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.731990    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b71d537-0290-43f9-9277-5f2c946a55b4-tmp-dir\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:13:50.732077 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.732011    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b71d537-0290-43f9-9277-5f2c946a55b4-model-cache\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:13:50.733631 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.733610    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b71d537-0290-43f9-9277-5f2c946a55b4-dshm" (OuterVolumeSpecName: "dshm") pod "9b71d537-0290-43f9-9277-5f2c946a55b4" (UID: "9b71d537-0290-43f9-9277-5f2c946a55b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:50.733763 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.733745    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b71d537-0290-43f9-9277-5f2c946a55b4-kube-api-access-l6r5d" (OuterVolumeSpecName: "kube-api-access-l6r5d") pod "9b71d537-0290-43f9-9277-5f2c946a55b4" (UID: "9b71d537-0290-43f9-9277-5f2c946a55b4"). InnerVolumeSpecName "kube-api-access-l6r5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:13:50.733869 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.733855    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b71d537-0290-43f9-9277-5f2c946a55b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b71d537-0290-43f9-9277-5f2c946a55b4" (UID: "9b71d537-0290-43f9-9277-5f2c946a55b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:13:50.832576 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.832540    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6r5d\" (UniqueName: \"kubernetes.io/projected/9b71d537-0290-43f9-9277-5f2c946a55b4-kube-api-access-l6r5d\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:13:50.832576 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.832569    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b71d537-0290-43f9-9277-5f2c946a55b4-home\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:13:50.832576 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.832578    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b71d537-0290-43f9-9277-5f2c946a55b4-tls-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:13:50.832576 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.832586    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b71d537-0290-43f9-9277-5f2c946a55b4-dshm\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:13:50.841625 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.841595    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ca610e0-374c-4895-a3e1-935015e1a861" path="/var/lib/kubelet/pods/4ca610e0-374c-4895-a3e1-935015e1a861/volumes"
Sep 10 21:13:50.873210 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.873174    2584 generic.go:356] "Generic (PLEG): container finished" podID="9b71d537-0290-43f9-9277-5f2c946a55b4" containerID="29161a3678342f4b4babe553cded88cfb67e1071c20f074372d58fc1b265c49e" exitCode=0
Sep 10 21:13:50.873379 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.873230    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"
Sep 10 21:13:50.873379 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.873249    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg" event={"ID":"9b71d537-0290-43f9-9277-5f2c946a55b4","Type":"ContainerDied","Data":"29161a3678342f4b4babe553cded88cfb67e1071c20f074372d58fc1b265c49e"}
Sep 10 21:13:50.873379 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.873290    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg" event={"ID":"9b71d537-0290-43f9-9277-5f2c946a55b4","Type":"ContainerDied","Data":"588cb425f9b431a0adccfe0f647dff518c5d64887431401e9867e0f790be0be8"}
Sep 10 21:13:50.873379 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.873312    2584 scope.go:117] "RemoveContainer" containerID="29161a3678342f4b4babe553cded88cfb67e1071c20f074372d58fc1b265c49e"
Sep 10 21:13:50.876181 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.876155    2584 generic.go:356] "Generic (PLEG): container finished" podID="9d88a8f0-33e6-42fb-8641-39e520983961" containerID="f830eaead0d3926b5a46d19257174e18a2b28a8510de8a231e295c47572b6cef" exitCode=0
Sep 10 21:13:50.876287 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.876244    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-169213ba-r4trd" event={"ID":"9d88a8f0-33e6-42fb-8641-39e520983961","Type":"ContainerDied","Data":"f830eaead0d3926b5a46d19257174e18a2b28a8510de8a231e295c47572b6cef"}
Sep 10 21:13:50.882216 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.882194    2584 scope.go:117] "RemoveContainer" containerID="29161a3678342f4b4babe553cded88cfb67e1071c20f074372d58fc1b265c49e"
Sep 10 21:13:50.882549 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:13:50.882528    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29161a3678342f4b4babe553cded88cfb67e1071c20f074372d58fc1b265c49e\": container with ID starting with 29161a3678342f4b4babe553cded88cfb67e1071c20f074372d58fc1b265c49e not found: ID does not exist" containerID="29161a3678342f4b4babe553cded88cfb67e1071c20f074372d58fc1b265c49e"
Sep 10 21:13:50.882600 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.882557    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29161a3678342f4b4babe553cded88cfb67e1071c20f074372d58fc1b265c49e"} err="failed to get container status \"29161a3678342f4b4babe553cded88cfb67e1071c20f074372d58fc1b265c49e\": rpc error: code = NotFound desc = could not find container \"29161a3678342f4b4babe553cded88cfb67e1071c20f074372d58fc1b265c49e\": container with ID starting with 29161a3678342f4b4babe553cded88cfb67e1071c20f074372d58fc1b265c49e not found: ID does not exist"
Sep 10 21:13:50.888791 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.888770    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"]
Sep 10 21:13:50.892343 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:50.892319    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-1c65017b-kserve-f6bddb68-6bckg"]
Sep 10 21:13:52.004713 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:52.004687    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-169213ba-r4trd"
Sep 10 21:13:52.044608 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:52.044576    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d88a8f0-33e6-42fb-8641-39e520983961-maas-api-service-ca\") pod \"9d88a8f0-33e6-42fb-8641-39e520983961\" (UID: \"9d88a8f0-33e6-42fb-8641-39e520983961\") "
Sep 10 21:13:52.044909 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:52.044888    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d88a8f0-33e6-42fb-8641-39e520983961-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9d88a8f0-33e6-42fb-8641-39e520983961" (UID: "9d88a8f0-33e6-42fb-8641-39e520983961"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:13:52.146232 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:52.146145    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d88a8f0-33e6-42fb-8641-39e520983961-maas-api-service-ca\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:13:52.840176 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:52.840126    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b71d537-0290-43f9-9277-5f2c946a55b4" path="/var/lib/kubelet/pods/9b71d537-0290-43f9-9277-5f2c946a55b4/volumes"
Sep 10 21:13:52.888058 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:52.888029    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-169213ba-r4trd"
Sep 10 21:13:52.888231 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:52.888030    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-169213ba-r4trd" event={"ID":"9d88a8f0-33e6-42fb-8641-39e520983961","Type":"ContainerDied","Data":"090cbabe2cc1dd0ad36e2f4aa872bf437f1390e82b81c64aaba11fc34ce2e9d5"}
Sep 10 21:13:52.888231 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:52.888160    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="090cbabe2cc1dd0ad36e2f4aa872bf437f1390e82b81c64aaba11fc34ce2e9d5"
Sep 10 21:13:53.309395 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.309363    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"]
Sep 10 21:13:53.309806 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.309790    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ca610e0-374c-4895-a3e1-935015e1a861" containerName="main"
Sep 10 21:13:53.309861 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.309810    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ca610e0-374c-4895-a3e1-935015e1a861" containerName="main"
Sep 10 21:13:53.309861 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.309823    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d88a8f0-33e6-42fb-8641-39e520983961" containerName="revoke-keys"
Sep 10 21:13:53.309861 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.309829    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d88a8f0-33e6-42fb-8641-39e520983961" containerName="revoke-keys"
Sep 10 21:13:53.309861 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.309844    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b71d537-0290-43f9-9277-5f2c946a55b4" containerName="main"
Sep 10 21:13:53.309861 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.309850    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b71d537-0290-43f9-9277-5f2c946a55b4" containerName="main"
Sep 10 21:13:53.310033 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.309910    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b71d537-0290-43f9-9277-5f2c946a55b4" containerName="main"
Sep 10 21:13:53.310033 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.309921    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d88a8f0-33e6-42fb-8641-39e520983961" containerName="revoke-keys"
Sep 10 21:13:53.310033 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.309932    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ca610e0-374c-4895-a3e1-935015e1a861" containerName="main"
Sep 10 21:13:53.313198 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.313177    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.315293 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.315275    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-2bbc78-openshift-default-dockercfg-kxqtg\""
Sep 10 21:13:53.325342 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.325319    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"]
Sep 10 21:13:53.358703 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.358669    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istio-envoy\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.358828 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.358718    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istio-podinfo\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.358828 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.358758    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5vlc\" (UniqueName: \"kubernetes.io/projected/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-kube-api-access-p5vlc\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.358909 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.358825    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istiod-ca-cert\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.358909 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.358855    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-workload-certs\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.358909 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.358872    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-credential-socket\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.359003 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.358909    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-workload-socket\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.359003 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.358935    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istio-data\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.359067 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.359003    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istio-token\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.459628 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.459591    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-workload-certs\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.459628 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.459629    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-credential-socket\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.459875 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.459649    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-workload-socket\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.459875 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.459672    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istio-data\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.459875 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.459731    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istio-token\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.459875 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.459804    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istio-envoy\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.459875 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.459855    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istio-podinfo\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.460091 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.459885    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5vlc\" (UniqueName: \"kubernetes.io/projected/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-kube-api-access-p5vlc\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.460091 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.459924    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istiod-ca-cert\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.460091 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.460020    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-workload-certs\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.460254 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.460197    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-credential-socket\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.460333 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.460311    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-workload-socket\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.460389 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.460359    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istio-data\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.460643 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.460624    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istiod-ca-cert\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.462121 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.462096    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istio-envoy\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.462328 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.462309    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istio-podinfo\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.467315 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.467287    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istio-token\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.467500 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.467483    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5vlc\" (UniqueName: \"kubernetes.io/projected/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-kube-api-access-p5vlc\") pod \"e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") " pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.624025 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.623987    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:53.904344 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.904321    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"]
Sep 10 21:13:53.906398 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:13:53.906373    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5387c1fa_e1d2_4704_87ad_d6f34f17f1a1.slice/crio-ed64756a4b858313396dad15a99b77233877fa1eaddd6aecb0b5680be55276c7 WatchSource:0}: Error finding container ed64756a4b858313396dad15a99b77233877fa1eaddd6aecb0b5680be55276c7: Status 404 returned error can't find the container with id ed64756a4b858313396dad15a99b77233877fa1eaddd6aecb0b5680be55276c7
Sep 10 21:13:53.908713 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.908680    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:13:53.908806 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.908745    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:13:53.908806 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:53.908784    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:13:54.899083 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:54.899044    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9" event={"ID":"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1","Type":"ContainerStarted","Data":"eb7651d83efb0c9fcd159807e2b5fe0fd04c62d582cae19862eada844ea5af61"}
Sep 10 21:13:54.899573 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:54.899105    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9" event={"ID":"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1","Type":"ContainerStarted","Data":"ed64756a4b858313396dad15a99b77233877fa1eaddd6aecb0b5680be55276c7"}
Sep 10 21:13:54.923144 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:54.923094    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9" podStartSLOduration=1.923077715 podStartE2EDuration="1.923077715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:13:54.921599133 +0000 UTC m=+894.586623955" watchObservedRunningTime="2026-09-10 21:13:54.923077715 +0000 UTC m=+894.588102579"
Sep 10 21:13:55.282120 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:55.282036    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-169213ba-r4trd"]
Sep 10 21:13:55.284760 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:55.284734    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-169213ba-r4trd"]
Sep 10 21:13:55.625090 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:55.625056    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:55.630102 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:55.630077    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:55.905111 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:55.905009    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:55.906222 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:55.906199    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:13:56.840606 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:13:56.840572    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d88a8f0-33e6-42fb-8641-39e520983961" path="/var/lib/kubelet/pods/9d88a8f0-33e6-42fb-8641-39e520983961/volumes"
Sep 10 21:14:00.785698 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:00.785654    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-jkkfm_fd81843e-6283-43e3-a571-d5e47be26e38/console-operator/1.log"
Sep 10 21:14:00.794241 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:00.794215    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-jkkfm_fd81843e-6283-43e3-a571-d5e47be26e38/console-operator/1.log"
Sep 10 21:14:11.946626 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:11.946574    2584 patch_prober.go:28] interesting pod/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 21:14:11.947210 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:11.946646    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9" podUID="5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 21:14:13.996247 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:13.996161    2584 generic.go:356] "Generic (PLEG): container finished" podID="5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" containerID="eb7651d83efb0c9fcd159807e2b5fe0fd04c62d582cae19862eada844ea5af61" exitCode=0
Sep 10 21:14:13.996697 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:13.996240    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9" event={"ID":"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1","Type":"ContainerDied","Data":"eb7651d83efb0c9fcd159807e2b5fe0fd04c62d582cae19862eada844ea5af61"}
Sep 10 21:14:13.996697 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:13.996688    2584 scope.go:117] "RemoveContainer" containerID="eb7651d83efb0c9fcd159807e2b5fe0fd04c62d582cae19862eada844ea5af61"
Sep 10 21:14:13.997751 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:13.997725    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:14:13.997827 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:13.997789    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:14:13.997827 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:13.997823    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:14:15.003069 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:15.003021    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9" event={"ID":"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1","Type":"ContainerStarted","Data":"79cb6da9562646ec21d6697b73c5dc7f4c8a959a5497dae76c4a460226c9c5f2"}
Sep 10 21:14:15.624881 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:15.624844    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:14:15.625262 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:15.625237    2584 patch_prober.go:28] interesting pod/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 10 21:14:15.625417 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:15.625391    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9" podUID="5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 10 21:14:16.625390 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:16.625345    2584 patch_prober.go:28] interesting pod/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 10 21:14:16.625918 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:16.625417    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9" podUID="5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 10 21:14:17.625543 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:17.625081    2584 patch_prober.go:28] interesting pod/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 10 21:14:17.625543 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:17.625155    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9" podUID="5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 10 21:14:18.625475 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:18.625411    2584 patch_prober.go:28] interesting pod/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 10 21:14:18.625807 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:18.625770    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9" podUID="5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 10 21:14:19.624563 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:19.624520    2584 patch_prober.go:28] interesting pod/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 10 21:14:19.624771 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:19.624623    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9" podUID="5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 10 21:14:20.628555 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:20.628525    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:14:20.628956 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:20.628863    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:14:20.629644 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:20.629626    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:14:24.458327 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:24.458295    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3002a381-hmfss"]
Sep 10 21:14:24.460890 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:24.460868    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3002a381-hmfss"
Sep 10 21:14:24.463009 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:24.462984    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:14:24.463127 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:24.463104    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4gs5\""
Sep 10 21:14:24.468995 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:24.468973    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3002a381-hmfss"]
Sep 10 21:14:24.485670 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:24.485641    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b88b47-bf5a-423d-a3d9-cfe788fd4af6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3002a381-hmfss\" (UID: \"54b88b47-bf5a-423d-a3d9-cfe788fd4af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3002a381-hmfss"
Sep 10 21:14:24.586866 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:24.586828    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b88b47-bf5a-423d-a3d9-cfe788fd4af6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3002a381-hmfss\" (UID: \"54b88b47-bf5a-423d-a3d9-cfe788fd4af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3002a381-hmfss"
Sep 10 21:14:24.587416 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:24.587397    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b88b47-bf5a-423d-a3d9-cfe788fd4af6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3002a381-hmfss\" (UID: \"54b88b47-bf5a-423d-a3d9-cfe788fd4af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3002a381-hmfss"
Sep 10 21:14:24.773258 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:24.773161    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3002a381-hmfss"
Sep 10 21:14:25.042732 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:25.042702    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3002a381-hmfss"]
Sep 10 21:14:25.044452 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:14:25.044393    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b88b47_bf5a_423d_a3d9_cfe788fd4af6.slice/crio-eb23f7eef35f1b7fd3e35df5d23f8166569fc2bce1df047342a0fa224affc570 WatchSource:0}: Error finding container eb23f7eef35f1b7fd3e35df5d23f8166569fc2bce1df047342a0fa224affc570: Status 404 returned error can't find the container with id eb23f7eef35f1b7fd3e35df5d23f8166569fc2bce1df047342a0fa224affc570
Sep 10 21:14:25.067221 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:25.067154    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3002a381-hmfss" event={"ID":"54b88b47-bf5a-423d-a3d9-cfe788fd4af6","Type":"ContainerStarted","Data":"eb23f7eef35f1b7fd3e35df5d23f8166569fc2bce1df047342a0fa224affc570"}
Sep 10 21:14:26.074050 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:26.073957    2584 generic.go:356] "Generic (PLEG): container finished" podID="54b88b47-bf5a-423d-a3d9-cfe788fd4af6" containerID="98ae987ca1341601e5b84925cbecfcc638e248d8416b6fb7dcb3b57e90738eae" exitCode=0
Sep 10 21:14:26.074050 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:26.074015    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3002a381-hmfss" event={"ID":"54b88b47-bf5a-423d-a3d9-cfe788fd4af6","Type":"ContainerDied","Data":"98ae987ca1341601e5b84925cbecfcc638e248d8416b6fb7dcb3b57e90738eae"}
Sep 10 21:14:27.211648 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:27.211621    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3002a381-hmfss"
Sep 10 21:14:27.311361 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:27.311330    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b88b47-bf5a-423d-a3d9-cfe788fd4af6-maas-api-service-ca\") pod \"54b88b47-bf5a-423d-a3d9-cfe788fd4af6\" (UID: \"54b88b47-bf5a-423d-a3d9-cfe788fd4af6\") "
Sep 10 21:14:27.311732 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:27.311710    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b88b47-bf5a-423d-a3d9-cfe788fd4af6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "54b88b47-bf5a-423d-a3d9-cfe788fd4af6" (UID: "54b88b47-bf5a-423d-a3d9-cfe788fd4af6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:14:27.412758 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:27.412719    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b88b47-bf5a-423d-a3d9-cfe788fd4af6-maas-api-service-ca\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:14:28.086079 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:28.086042    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3002a381-hmfss" event={"ID":"54b88b47-bf5a-423d-a3d9-cfe788fd4af6","Type":"ContainerDied","Data":"eb23f7eef35f1b7fd3e35df5d23f8166569fc2bce1df047342a0fa224affc570"}
Sep 10 21:14:28.086079 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:28.086065    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3002a381-hmfss"
Sep 10 21:14:28.086326 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:28.086092    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb23f7eef35f1b7fd3e35df5d23f8166569fc2bce1df047342a0fa224affc570"
Sep 10 21:14:31.604407 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:31.604367    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3002a381-hmfss"]
Sep 10 21:14:31.606761 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:31.606736    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3002a381-hmfss"]
Sep 10 21:14:32.840143 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:32.840109    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54b88b47-bf5a-423d-a3d9-cfe788fd4af6" path="/var/lib/kubelet/pods/54b88b47-bf5a-423d-a3d9-cfe788fd4af6/volumes"
Sep 10 21:14:41.875762 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:41.875726    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"]
Sep 10 21:14:41.876367 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:41.876120    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54b88b47-bf5a-423d-a3d9-cfe788fd4af6" containerName="revoke-keys"
Sep 10 21:14:41.876367 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:41.876132    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="54b88b47-bf5a-423d-a3d9-cfe788fd4af6" containerName="revoke-keys"
Sep 10 21:14:41.876367 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:41.876210    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="54b88b47-bf5a-423d-a3d9-cfe788fd4af6" containerName="revoke-keys"
Sep 10 21:14:41.878219 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:41.878201    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:14:41.881288 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:41.881270    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-2bbc78\"/\"explicit-model-2bbc78-kserve-self-signed-certs\""
Sep 10 21:14:41.881388 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:41.881323    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-2bbc78\"/\"openshift-service-ca.crt\""
Sep 10 21:14:41.881463 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:41.881439    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-2bbc78\"/\"default-dockercfg-b7ct6\""
Sep 10 21:14:41.881557 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:41.881499    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-2bbc78\"/\"kube-root-ca.crt\""
Sep 10 21:14:41.890545 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:41.890524    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"]
Sep 10 21:14:41.952237 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:41.952202    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0544dd1-9220-4630-b88c-06e874a328bd-dshm\") pod \"explicit-model-2bbc78-kserve-7dcd8b8777-vck4h\" (UID: \"b0544dd1-9220-4630-b88c-06e874a328bd\") " pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:14:41.952237 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:41.952252    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0544dd1-9220-4630-b88c-06e874a328bd-tmp-dir\") pod \"explicit-model-2bbc78-kserve-7dcd8b8777-vck4h\" (UID: \"b0544dd1-9220-4630-b88c-06e874a328bd\") " pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:14:41.952477 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:41.952344    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0544dd1-9220-4630-b88c-06e874a328bd-model-cache\") pod \"explicit-model-2bbc78-kserve-7dcd8b8777-vck4h\" (UID: \"b0544dd1-9220-4630-b88c-06e874a328bd\") " pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:14:41.952477 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:41.952380    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0544dd1-9220-4630-b88c-06e874a328bd-home\") pod \"explicit-model-2bbc78-kserve-7dcd8b8777-vck4h\" (UID: \"b0544dd1-9220-4630-b88c-06e874a328bd\") " pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:14:41.952477 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:41.952465    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0544dd1-9220-4630-b88c-06e874a328bd-tls-certs\") pod \"explicit-model-2bbc78-kserve-7dcd8b8777-vck4h\" (UID: \"b0544dd1-9220-4630-b88c-06e874a328bd\") " pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:14:41.952578 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:41.952515    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmcmn\" (UniqueName: \"kubernetes.io/projected/b0544dd1-9220-4630-b88c-06e874a328bd-kube-api-access-gmcmn\") pod \"explicit-model-2bbc78-kserve-7dcd8b8777-vck4h\" (UID: \"b0544dd1-9220-4630-b88c-06e874a328bd\") " pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:14:42.053298 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:42.053257    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0544dd1-9220-4630-b88c-06e874a328bd-dshm\") pod \"explicit-model-2bbc78-kserve-7dcd8b8777-vck4h\" (UID: \"b0544dd1-9220-4630-b88c-06e874a328bd\") " pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:14:42.053488 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:42.053306    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0544dd1-9220-4630-b88c-06e874a328bd-tmp-dir\") pod \"explicit-model-2bbc78-kserve-7dcd8b8777-vck4h\" (UID: \"b0544dd1-9220-4630-b88c-06e874a328bd\") " pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:14:42.053488 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:42.053357    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0544dd1-9220-4630-b88c-06e874a328bd-model-cache\") pod \"explicit-model-2bbc78-kserve-7dcd8b8777-vck4h\" (UID: \"b0544dd1-9220-4630-b88c-06e874a328bd\") " pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:14:42.053488 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:42.053386    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0544dd1-9220-4630-b88c-06e874a328bd-home\") pod \"explicit-model-2bbc78-kserve-7dcd8b8777-vck4h\" (UID: \"b0544dd1-9220-4630-b88c-06e874a328bd\") " pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:14:42.053615 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:42.053488    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0544dd1-9220-4630-b88c-06e874a328bd-tls-certs\") pod \"explicit-model-2bbc78-kserve-7dcd8b8777-vck4h\" (UID: \"b0544dd1-9220-4630-b88c-06e874a328bd\") " pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:14:42.053615 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:42.053525    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmcmn\" (UniqueName: \"kubernetes.io/projected/b0544dd1-9220-4630-b88c-06e874a328bd-kube-api-access-gmcmn\") pod \"explicit-model-2bbc78-kserve-7dcd8b8777-vck4h\" (UID: \"b0544dd1-9220-4630-b88c-06e874a328bd\") " pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:14:42.053841 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:42.053803    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0544dd1-9220-4630-b88c-06e874a328bd-model-cache\") pod \"explicit-model-2bbc78-kserve-7dcd8b8777-vck4h\" (UID: \"b0544dd1-9220-4630-b88c-06e874a328bd\") " pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:14:42.053841 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:42.053834    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0544dd1-9220-4630-b88c-06e874a328bd-tmp-dir\") pod \"explicit-model-2bbc78-kserve-7dcd8b8777-vck4h\" (UID: \"b0544dd1-9220-4630-b88c-06e874a328bd\") " pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:14:42.054002 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:42.053851    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0544dd1-9220-4630-b88c-06e874a328bd-home\") pod \"explicit-model-2bbc78-kserve-7dcd8b8777-vck4h\" (UID: \"b0544dd1-9220-4630-b88c-06e874a328bd\") " pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:14:42.055601 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:42.055573    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0544dd1-9220-4630-b88c-06e874a328bd-dshm\") pod \"explicit-model-2bbc78-kserve-7dcd8b8777-vck4h\" (UID: \"b0544dd1-9220-4630-b88c-06e874a328bd\") " pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:14:42.055832 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:42.055816    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0544dd1-9220-4630-b88c-06e874a328bd-tls-certs\") pod \"explicit-model-2bbc78-kserve-7dcd8b8777-vck4h\" (UID: \"b0544dd1-9220-4630-b88c-06e874a328bd\") " pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:14:42.061704 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:42.061682    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmcmn\" (UniqueName: \"kubernetes.io/projected/b0544dd1-9220-4630-b88c-06e874a328bd-kube-api-access-gmcmn\") pod \"explicit-model-2bbc78-kserve-7dcd8b8777-vck4h\" (UID: \"b0544dd1-9220-4630-b88c-06e874a328bd\") " pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:14:42.188494 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:42.188402    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:14:42.464214 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:42.464189    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"]
Sep 10 21:14:42.465945 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:14:42.465918    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0544dd1_9220_4630_b88c_06e874a328bd.slice/crio-82c00a126924f73f46d5f826a2f8a81ba4f78d0237d5bc85356270f74c84496c WatchSource:0}: Error finding container 82c00a126924f73f46d5f826a2f8a81ba4f78d0237d5bc85356270f74c84496c: Status 404 returned error can't find the container with id 82c00a126924f73f46d5f826a2f8a81ba4f78d0237d5bc85356270f74c84496c
Sep 10 21:14:43.164053 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:43.164012    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h" event={"ID":"b0544dd1-9220-4630-b88c-06e874a328bd","Type":"ContainerStarted","Data":"b37fc209bffc6c6ab147a73064517de6f8499c63814e35bd2d0c7ab8337760a9"}
Sep 10 21:14:43.164053 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:43.164059    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h" event={"ID":"b0544dd1-9220-4630-b88c-06e874a328bd","Type":"ContainerStarted","Data":"82c00a126924f73f46d5f826a2f8a81ba4f78d0237d5bc85356270f74c84496c"}
Sep 10 21:14:43.181844 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:43.181791    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h" podStartSLOduration=2.181771551 podStartE2EDuration="2.181771551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:14:43.179744465 +0000 UTC m=+942.844769265" watchObservedRunningTime="2026-09-10 21:14:43.181771551 +0000 UTC m=+942.846796352"
Sep 10 21:14:52.188912 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:52.188869    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:14:52.189336 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:52.188943    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:14:52.201252 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:52.201225    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:14:52.220908 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:14:52.220879    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:15:00.138780 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:00.138744    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2bbc78-29817915-vz4nd"]
Sep 10 21:15:00.141646 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:00.141624    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2bbc78-29817915-vz4nd"
Sep 10 21:15:00.144012 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:00.143987    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4gs5\""
Sep 10 21:15:00.145419 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:00.145397    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-414d5f-29817915-p2cfg"]
Sep 10 21:15:00.148733 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:00.148712    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-414d5f-29817915-p2cfg"
Sep 10 21:15:00.151006 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:00.150975    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2bbc78-29817915-vz4nd"]
Sep 10 21:15:00.157363 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:00.157338    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-414d5f-29817915-p2cfg"]
Sep 10 21:15:00.158504 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:00.158487    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2bbc78-29817915-vz4nd"
Sep 10 21:15:00.165218 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:00.165192    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-414d5f-29817915-p2cfg"
Sep 10 21:15:00.459983 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:00.459921    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2bbc78-29817915-vz4nd"]
Sep 10 21:15:00.462212 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:15:00.462187    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec573b8_08fb_457a_8447_01a991c2bd42.slice/crio-93dddb9deb669933e4996d28434b92fba779ea23f749bb0afcac71942856d0b1 WatchSource:0}: Error finding container 93dddb9deb669933e4996d28434b92fba779ea23f749bb0afcac71942856d0b1: Status 404 returned error can't find the container with id 93dddb9deb669933e4996d28434b92fba779ea23f749bb0afcac71942856d0b1
Sep 10 21:15:00.462804 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:00.462780    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-414d5f-29817915-p2cfg"]
Sep 10 21:15:00.463100 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:15:00.463051    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513a4bee_f302_47eb_b242_f019625dfb6a.slice/crio-bf1f7b0cc8cd9f1895514d82b5d2fca3464f7d6974917f6b9347da5e86293064 WatchSource:0}: Error finding container bf1f7b0cc8cd9f1895514d82b5d2fca3464f7d6974917f6b9347da5e86293064: Status 404 returned error can't find the container with id bf1f7b0cc8cd9f1895514d82b5d2fca3464f7d6974917f6b9347da5e86293064
Sep 10 21:15:01.260791 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:01.260750    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2bbc78-29817915-vz4nd" event={"ID":"3ec573b8-08fb-457a-8447-01a991c2bd42","Type":"ContainerStarted","Data":"cf15f95e025adebd9cfebc3128c829fba9446aa812d0f6559fcbab656de3657c"}
Sep 10 21:15:01.261270 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:01.260818    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2bbc78-29817915-vz4nd" event={"ID":"3ec573b8-08fb-457a-8447-01a991c2bd42","Type":"ContainerStarted","Data":"93dddb9deb669933e4996d28434b92fba779ea23f749bb0afcac71942856d0b1"}
Sep 10 21:15:01.268861 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:01.268831    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-414d5f-29817915-p2cfg" event={"ID":"513a4bee-f302-47eb-b242-f019625dfb6a","Type":"ContainerStarted","Data":"65b613f27e3a7e117d04678e5ad1f69c29b253e9a78b840545b6576c2393394b"}
Sep 10 21:15:01.268994 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:01.268912    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-414d5f-29817915-p2cfg" event={"ID":"513a4bee-f302-47eb-b242-f019625dfb6a","Type":"ContainerStarted","Data":"bf1f7b0cc8cd9f1895514d82b5d2fca3464f7d6974917f6b9347da5e86293064"}
Sep 10 21:15:01.279328 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:01.279281    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2bbc78-29817915-vz4nd" podStartSLOduration=1.2792666019999999 podStartE2EDuration="1.279266602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:01.277459842 +0000 UTC m=+960.942484642" watchObservedRunningTime="2026-09-10 21:15:01.279266602 +0000 UTC m=+960.944291404"
Sep 10 21:15:01.291327 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:01.291286    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-414d5f-29817915-p2cfg" podStartSLOduration=1.291272638 podStartE2EDuration="1.291272638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:01.290905452 +0000 UTC m=+960.955930253" watchObservedRunningTime="2026-09-10 21:15:01.291272638 +0000 UTC m=+960.956297682"
Sep 10 21:15:02.275891 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:02.275860    2584 generic.go:356] "Generic (PLEG): container finished" podID="513a4bee-f302-47eb-b242-f019625dfb6a" containerID="65b613f27e3a7e117d04678e5ad1f69c29b253e9a78b840545b6576c2393394b" exitCode=0
Sep 10 21:15:02.276341 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:02.275947    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-414d5f-29817915-p2cfg" event={"ID":"513a4bee-f302-47eb-b242-f019625dfb6a","Type":"ContainerDied","Data":"65b613f27e3a7e117d04678e5ad1f69c29b253e9a78b840545b6576c2393394b"}
Sep 10 21:15:02.277555 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:02.277534    2584 generic.go:356] "Generic (PLEG): container finished" podID="3ec573b8-08fb-457a-8447-01a991c2bd42" containerID="cf15f95e025adebd9cfebc3128c829fba9446aa812d0f6559fcbab656de3657c" exitCode=0
Sep 10 21:15:02.277680 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:02.277576    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2bbc78-29817915-vz4nd" event={"ID":"3ec573b8-08fb-457a-8447-01a991c2bd42","Type":"ContainerDied","Data":"cf15f95e025adebd9cfebc3128c829fba9446aa812d0f6559fcbab656de3657c"}
Sep 10 21:15:03.407573 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:03.407550    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-414d5f-29817915-p2cfg"
Sep 10 21:15:03.442536 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:03.442510    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2bbc78-29817915-vz4nd"
Sep 10 21:15:03.872778 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:03.872742    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"]
Sep 10 21:15:03.873125 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:03.873112    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ec573b8-08fb-457a-8447-01a991c2bd42" containerName="cleanup"
Sep 10 21:15:03.873125 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:03.873125    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ec573b8-08fb-457a-8447-01a991c2bd42" containerName="cleanup"
Sep 10 21:15:03.873249 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:03.873140    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="513a4bee-f302-47eb-b242-f019625dfb6a" containerName="cleanup"
Sep 10 21:15:03.873249 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:03.873145    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="513a4bee-f302-47eb-b242-f019625dfb6a" containerName="cleanup"
Sep 10 21:15:03.873249 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:03.873200    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="513a4bee-f302-47eb-b242-f019625dfb6a" containerName="cleanup"
Sep 10 21:15:03.873249 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:03.873209    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ec573b8-08fb-457a-8447-01a991c2bd42" containerName="cleanup"
Sep 10 21:15:03.875087 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:03.875071    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:15:03.877631 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:03.877606    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-414d5f\"/\"default-dockercfg-jhrpp\""
Sep 10 21:15:03.877751 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:03.877609    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-414d5f\"/\"openshift-service-ca.crt\""
Sep 10 21:15:03.877751 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:03.877610    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-414d5f\"/\"kube-root-ca.crt\""
Sep 10 21:15:03.877751 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:03.877660    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-414d5f\"/\"test-model-414d5f-kserve-self-signed-certs\""
Sep 10 21:15:03.885390 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:03.885370    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"]
Sep 10 21:15:03.974596 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:03.974560    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e75427ea-7c98-496b-8a05-af05b767e6e3-model-cache\") pod \"test-model-414d5f-kserve-7866545d97-dr6gc\" (UID: \"e75427ea-7c98-496b-8a05-af05b767e6e3\") " pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:15:03.974765 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:03.974600    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e75427ea-7c98-496b-8a05-af05b767e6e3-home\") pod \"test-model-414d5f-kserve-7866545d97-dr6gc\" (UID: \"e75427ea-7c98-496b-8a05-af05b767e6e3\") " pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:15:03.974765 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:03.974637    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e75427ea-7c98-496b-8a05-af05b767e6e3-tmp-dir\") pod \"test-model-414d5f-kserve-7866545d97-dr6gc\" (UID: \"e75427ea-7c98-496b-8a05-af05b767e6e3\") " pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:15:03.974765 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:03.974694    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e75427ea-7c98-496b-8a05-af05b767e6e3-tls-certs\") pod \"test-model-414d5f-kserve-7866545d97-dr6gc\" (UID: \"e75427ea-7c98-496b-8a05-af05b767e6e3\") " pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:15:03.974925 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:03.974775    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e75427ea-7c98-496b-8a05-af05b767e6e3-dshm\") pod \"test-model-414d5f-kserve-7866545d97-dr6gc\" (UID: \"e75427ea-7c98-496b-8a05-af05b767e6e3\") " pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:15:03.974925 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:03.974834    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc88z\" (UniqueName: \"kubernetes.io/projected/e75427ea-7c98-496b-8a05-af05b767e6e3-kube-api-access-qc88z\") pod \"test-model-414d5f-kserve-7866545d97-dr6gc\" (UID: \"e75427ea-7c98-496b-8a05-af05b767e6e3\") " pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:15:04.075516 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:04.075479    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e75427ea-7c98-496b-8a05-af05b767e6e3-model-cache\") pod \"test-model-414d5f-kserve-7866545d97-dr6gc\" (UID: \"e75427ea-7c98-496b-8a05-af05b767e6e3\") " pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:15:04.075516 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:04.075521    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e75427ea-7c98-496b-8a05-af05b767e6e3-home\") pod \"test-model-414d5f-kserve-7866545d97-dr6gc\" (UID: \"e75427ea-7c98-496b-8a05-af05b767e6e3\") " pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:15:04.075769 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:04.075550    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e75427ea-7c98-496b-8a05-af05b767e6e3-tmp-dir\") pod \"test-model-414d5f-kserve-7866545d97-dr6gc\" (UID: \"e75427ea-7c98-496b-8a05-af05b767e6e3\") " pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:15:04.075769 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:04.075599    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e75427ea-7c98-496b-8a05-af05b767e6e3-tls-certs\") pod \"test-model-414d5f-kserve-7866545d97-dr6gc\" (UID: \"e75427ea-7c98-496b-8a05-af05b767e6e3\") " pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:15:04.075769 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:04.075658    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e75427ea-7c98-496b-8a05-af05b767e6e3-dshm\") pod \"test-model-414d5f-kserve-7866545d97-dr6gc\" (UID: \"e75427ea-7c98-496b-8a05-af05b767e6e3\") " pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:15:04.075769 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:04.075701    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc88z\" (UniqueName: \"kubernetes.io/projected/e75427ea-7c98-496b-8a05-af05b767e6e3-kube-api-access-qc88z\") pod \"test-model-414d5f-kserve-7866545d97-dr6gc\" (UID: \"e75427ea-7c98-496b-8a05-af05b767e6e3\") " pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:15:04.076041 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:04.075967    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e75427ea-7c98-496b-8a05-af05b767e6e3-home\") pod \"test-model-414d5f-kserve-7866545d97-dr6gc\" (UID: \"e75427ea-7c98-496b-8a05-af05b767e6e3\") " pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:15:04.076041 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:04.075987    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e75427ea-7c98-496b-8a05-af05b767e6e3-tmp-dir\") pod \"test-model-414d5f-kserve-7866545d97-dr6gc\" (UID: \"e75427ea-7c98-496b-8a05-af05b767e6e3\") " pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:15:04.076124 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:04.076036    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e75427ea-7c98-496b-8a05-af05b767e6e3-model-cache\") pod \"test-model-414d5f-kserve-7866545d97-dr6gc\" (UID: \"e75427ea-7c98-496b-8a05-af05b767e6e3\") " pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:15:04.077924 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:04.077907    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e75427ea-7c98-496b-8a05-af05b767e6e3-dshm\") pod \"test-model-414d5f-kserve-7866545d97-dr6gc\" (UID: \"e75427ea-7c98-496b-8a05-af05b767e6e3\") " pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:15:04.078186 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:04.078170    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e75427ea-7c98-496b-8a05-af05b767e6e3-tls-certs\") pod \"test-model-414d5f-kserve-7866545d97-dr6gc\" (UID: \"e75427ea-7c98-496b-8a05-af05b767e6e3\") " pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:15:04.089120 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:04.089092    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc88z\" (UniqueName: \"kubernetes.io/projected/e75427ea-7c98-496b-8a05-af05b767e6e3-kube-api-access-qc88z\") pod \"test-model-414d5f-kserve-7866545d97-dr6gc\" (UID: \"e75427ea-7c98-496b-8a05-af05b767e6e3\") " pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:15:04.185660 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:04.185575    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:15:04.288218 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:04.288185    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2bbc78-29817915-vz4nd" event={"ID":"3ec573b8-08fb-457a-8447-01a991c2bd42","Type":"ContainerDied","Data":"93dddb9deb669933e4996d28434b92fba779ea23f749bb0afcac71942856d0b1"}
Sep 10 21:15:04.288389 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:04.288235    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93dddb9deb669933e4996d28434b92fba779ea23f749bb0afcac71942856d0b1"
Sep 10 21:15:04.288389 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:04.288198    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2bbc78-29817915-vz4nd"
Sep 10 21:15:04.289614 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:04.289590    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-414d5f-29817915-p2cfg" event={"ID":"513a4bee-f302-47eb-b242-f019625dfb6a","Type":"ContainerDied","Data":"bf1f7b0cc8cd9f1895514d82b5d2fca3464f7d6974917f6b9347da5e86293064"}
Sep 10 21:15:04.289614 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:04.289609    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-414d5f-29817915-p2cfg"
Sep 10 21:15:04.289777 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:04.289629    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf1f7b0cc8cd9f1895514d82b5d2fca3464f7d6974917f6b9347da5e86293064"
Sep 10 21:15:04.460469 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:04.460444    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"]
Sep 10 21:15:04.462356 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:15:04.462331    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode75427ea_7c98_496b_8a05_af05b767e6e3.slice/crio-312ddbbf308dbc3546d6031e7ce8bc4520e589b75693709ca1699dc82d0affe9 WatchSource:0}: Error finding container 312ddbbf308dbc3546d6031e7ce8bc4520e589b75693709ca1699dc82d0affe9: Status 404 returned error can't find the container with id 312ddbbf308dbc3546d6031e7ce8bc4520e589b75693709ca1699dc82d0affe9
Sep 10 21:15:05.296116 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:05.296077    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc" event={"ID":"e75427ea-7c98-496b-8a05-af05b767e6e3","Type":"ContainerStarted","Data":"6a5c349018403d7a8b9e8cd0e972c6a8cbb6777e65919b81525e1ef95ba6b4ee"}
Sep 10 21:15:05.296116 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:05.296120    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc" event={"ID":"e75427ea-7c98-496b-8a05-af05b767e6e3","Type":"ContainerStarted","Data":"312ddbbf308dbc3546d6031e7ce8bc4520e589b75693709ca1699dc82d0affe9"}
Sep 10 21:15:05.317562 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:05.317509    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc" podStartSLOduration=2.317491297 podStartE2EDuration="2.317491297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:05.31609473 +0000 UTC m=+964.981119527" watchObservedRunningTime="2026-09-10 21:15:05.317491297 +0000 UTC m=+964.982516097"
Sep 10 21:15:05.846193 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:05.846158    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"]
Sep 10 21:15:14.186747 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:14.186713    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:15:14.187242 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:14.186779    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:15:14.199560 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:14.199537    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:15:14.353031 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:14.352996    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:15:16.626341 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:16.626295    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2bbc78-29817915-vz4nd"]
Sep 10 21:15:16.628621 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:16.628597    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2bbc78-29817915-vz4nd"]
Sep 10 21:15:16.841554 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:16.841520    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ec573b8-08fb-457a-8447-01a991c2bd42" path="/var/lib/kubelet/pods/3ec573b8-08fb-457a-8447-01a991c2bd42/volumes"
Sep 10 21:15:20.865879 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:20.865824    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h" podUID="b0544dd1-9220-4630-b88c-06e874a328bd" containerName="main" containerID="cri-o://b37fc209bffc6c6ab147a73064517de6f8499c63814e35bd2d0c7ab8337760a9" gracePeriod=45
Sep 10 21:15:21.117843 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.117780    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:15:21.252477 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.252444    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0544dd1-9220-4630-b88c-06e874a328bd-tls-certs\") pod \"b0544dd1-9220-4630-b88c-06e874a328bd\" (UID: \"b0544dd1-9220-4630-b88c-06e874a328bd\") "
Sep 10 21:15:21.252644 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.252491    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmcmn\" (UniqueName: \"kubernetes.io/projected/b0544dd1-9220-4630-b88c-06e874a328bd-kube-api-access-gmcmn\") pod \"b0544dd1-9220-4630-b88c-06e874a328bd\" (UID: \"b0544dd1-9220-4630-b88c-06e874a328bd\") "
Sep 10 21:15:21.252644 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.252518    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0544dd1-9220-4630-b88c-06e874a328bd-home\") pod \"b0544dd1-9220-4630-b88c-06e874a328bd\" (UID: \"b0544dd1-9220-4630-b88c-06e874a328bd\") "
Sep 10 21:15:21.252644 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.252581    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0544dd1-9220-4630-b88c-06e874a328bd-tmp-dir\") pod \"b0544dd1-9220-4630-b88c-06e874a328bd\" (UID: \"b0544dd1-9220-4630-b88c-06e874a328bd\") "
Sep 10 21:15:21.252644 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.252638    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0544dd1-9220-4630-b88c-06e874a328bd-model-cache\") pod \"b0544dd1-9220-4630-b88c-06e874a328bd\" (UID: \"b0544dd1-9220-4630-b88c-06e874a328bd\") "
Sep 10 21:15:21.252855 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.252674    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0544dd1-9220-4630-b88c-06e874a328bd-dshm\") pod \"b0544dd1-9220-4630-b88c-06e874a328bd\" (UID: \"b0544dd1-9220-4630-b88c-06e874a328bd\") "
Sep 10 21:15:21.252855 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.252790    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0544dd1-9220-4630-b88c-06e874a328bd-home" (OuterVolumeSpecName: "home") pod "b0544dd1-9220-4630-b88c-06e874a328bd" (UID: "b0544dd1-9220-4630-b88c-06e874a328bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:21.252970 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.252911    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0544dd1-9220-4630-b88c-06e874a328bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b0544dd1-9220-4630-b88c-06e874a328bd" (UID: "b0544dd1-9220-4630-b88c-06e874a328bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:21.252970 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.252931    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0544dd1-9220-4630-b88c-06e874a328bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "b0544dd1-9220-4630-b88c-06e874a328bd" (UID: "b0544dd1-9220-4630-b88c-06e874a328bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:21.253096 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.253075    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0544dd1-9220-4630-b88c-06e874a328bd-home\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:15:21.253136 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.253104    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0544dd1-9220-4630-b88c-06e874a328bd-tmp-dir\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:15:21.253136 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.253119    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0544dd1-9220-4630-b88c-06e874a328bd-model-cache\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:15:21.254750 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.254729    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0544dd1-9220-4630-b88c-06e874a328bd-dshm" (OuterVolumeSpecName: "dshm") pod "b0544dd1-9220-4630-b88c-06e874a328bd" (UID: "b0544dd1-9220-4630-b88c-06e874a328bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:21.254833 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.254759    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0544dd1-9220-4630-b88c-06e874a328bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0544dd1-9220-4630-b88c-06e874a328bd" (UID: "b0544dd1-9220-4630-b88c-06e874a328bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:15:21.254867 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.254836    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0544dd1-9220-4630-b88c-06e874a328bd-kube-api-access-gmcmn" (OuterVolumeSpecName: "kube-api-access-gmcmn") pod "b0544dd1-9220-4630-b88c-06e874a328bd" (UID: "b0544dd1-9220-4630-b88c-06e874a328bd"). InnerVolumeSpecName "kube-api-access-gmcmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:15:21.353884 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.353840    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmcmn\" (UniqueName: \"kubernetes.io/projected/b0544dd1-9220-4630-b88c-06e874a328bd-kube-api-access-gmcmn\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:15:21.353884 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.353880    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0544dd1-9220-4630-b88c-06e874a328bd-dshm\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:15:21.353884 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.353892    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0544dd1-9220-4630-b88c-06e874a328bd-tls-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:15:21.380000 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.379968    2584 generic.go:356] "Generic (PLEG): container finished" podID="b0544dd1-9220-4630-b88c-06e874a328bd" containerID="b37fc209bffc6c6ab147a73064517de6f8499c63814e35bd2d0c7ab8337760a9" exitCode=0
Sep 10 21:15:21.380133 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.380015    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"
Sep 10 21:15:21.380133 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.380014    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h" event={"ID":"b0544dd1-9220-4630-b88c-06e874a328bd","Type":"ContainerDied","Data":"b37fc209bffc6c6ab147a73064517de6f8499c63814e35bd2d0c7ab8337760a9"}
Sep 10 21:15:21.380133 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.380053    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h" event={"ID":"b0544dd1-9220-4630-b88c-06e874a328bd","Type":"ContainerDied","Data":"82c00a126924f73f46d5f826a2f8a81ba4f78d0237d5bc85356270f74c84496c"}
Sep 10 21:15:21.380133 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.380076    2584 scope.go:117] "RemoveContainer" containerID="b37fc209bffc6c6ab147a73064517de6f8499c63814e35bd2d0c7ab8337760a9"
Sep 10 21:15:21.389657 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.389640    2584 scope.go:117] "RemoveContainer" containerID="b37fc209bffc6c6ab147a73064517de6f8499c63814e35bd2d0c7ab8337760a9"
Sep 10 21:15:21.389927 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:15:21.389911    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b37fc209bffc6c6ab147a73064517de6f8499c63814e35bd2d0c7ab8337760a9\": container with ID starting with b37fc209bffc6c6ab147a73064517de6f8499c63814e35bd2d0c7ab8337760a9 not found: ID does not exist" containerID="b37fc209bffc6c6ab147a73064517de6f8499c63814e35bd2d0c7ab8337760a9"
Sep 10 21:15:21.389979 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.389934    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b37fc209bffc6c6ab147a73064517de6f8499c63814e35bd2d0c7ab8337760a9"} err="failed to get container status \"b37fc209bffc6c6ab147a73064517de6f8499c63814e35bd2d0c7ab8337760a9\": rpc error: code = NotFound desc = could not find container \"b37fc209bffc6c6ab147a73064517de6f8499c63814e35bd2d0c7ab8337760a9\": container with ID starting with b37fc209bffc6c6ab147a73064517de6f8499c63814e35bd2d0c7ab8337760a9 not found: ID does not exist"
Sep 10 21:15:21.402165 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.402143    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"]
Sep 10 21:15:21.409080 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:21.409058    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-2bbc78/explicit-model-2bbc78-kserve-7dcd8b8777-vck4h"]
Sep 10 21:15:22.839984 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:22.839954    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0544dd1-9220-4630-b88c-06e874a328bd" path="/var/lib/kubelet/pods/b0544dd1-9220-4630-b88c-06e874a328bd/volumes"
Sep 10 21:15:23.804568 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:23.804535    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"]
Sep 10 21:15:23.804838 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:23.804793    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9" podUID="5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" containerName="istio-proxy" containerID="cri-o://79cb6da9562646ec21d6697b73c5dc7f4c8a959a5497dae76c4a460226c9c5f2" gracePeriod=30
Sep 10 21:15:29.064726 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.064698    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:15:29.233330 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.233295    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istiod-ca-cert\") pod \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") "
Sep 10 21:15:29.233555 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.233342    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-workload-socket\") pod \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") "
Sep 10 21:15:29.233555 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.233370    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5vlc\" (UniqueName: \"kubernetes.io/projected/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-kube-api-access-p5vlc\") pod \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") "
Sep 10 21:15:29.233555 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.233412    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istio-data\") pod \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") "
Sep 10 21:15:29.233729 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.233566    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istio-envoy\") pod \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") "
Sep 10 21:15:29.233729 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.233660    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istio-token\") pod \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") "
Sep 10 21:15:29.233729 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.233713    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-workload-certs\") pod \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") "
Sep 10 21:15:29.233889 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.233752    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istio-data" (OuterVolumeSpecName: "istio-data") pod "5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" (UID: "5387c1fa-e1d2-4704-87ad-d6f34f17f1a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:29.233889 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.233771    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" (UID: "5387c1fa-e1d2-4704-87ad-d6f34f17f1a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:15:29.233889 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.233774    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" (UID: "5387c1fa-e1d2-4704-87ad-d6f34f17f1a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:29.233889 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.233811    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-credential-socket\") pod \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") "
Sep 10 21:15:29.233889 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.233850    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istio-podinfo\") pod \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\" (UID: \"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1\") "
Sep 10 21:15:29.234161 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.234013    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" (UID: "5387c1fa-e1d2-4704-87ad-d6f34f17f1a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:29.234161 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.234106    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" (UID: "5387c1fa-e1d2-4704-87ad-d6f34f17f1a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:29.234273 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.234252    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-workload-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:15:29.234326 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.234277    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-credential-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:15:29.234326 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.234298    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istiod-ca-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:15:29.234326 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.234311    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-workload-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:15:29.234326 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.234324    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istio-data\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:15:29.235801 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.235773    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-kube-api-access-p5vlc" (OuterVolumeSpecName: "kube-api-access-p5vlc") pod "5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" (UID: "5387c1fa-e1d2-4704-87ad-d6f34f17f1a1"). InnerVolumeSpecName "kube-api-access-p5vlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:15:29.235912 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.235893    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" (UID: "5387c1fa-e1d2-4704-87ad-d6f34f17f1a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:15:29.236266 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.236246    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" (UID: "5387c1fa-e1d2-4704-87ad-d6f34f17f1a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:29.236266 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.236257    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istio-token" (OuterVolumeSpecName: "istio-token") pod "5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" (UID: "5387c1fa-e1d2-4704-87ad-d6f34f17f1a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:15:29.335682 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.335653    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istio-podinfo\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:15:29.335682 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.335679    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5vlc\" (UniqueName: \"kubernetes.io/projected/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-kube-api-access-p5vlc\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:15:29.335682 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.335689    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istio-envoy\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:15:29.335948 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.335697    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1-istio-token\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:15:29.426127 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.426095    2584 generic.go:356] "Generic (PLEG): container finished" podID="5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" containerID="79cb6da9562646ec21d6697b73c5dc7f4c8a959a5497dae76c4a460226c9c5f2" exitCode=0
Sep 10 21:15:29.426316 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.426153    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9" event={"ID":"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1","Type":"ContainerDied","Data":"79cb6da9562646ec21d6697b73c5dc7f4c8a959a5497dae76c4a460226c9c5f2"}
Sep 10 21:15:29.426316 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.426160    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"
Sep 10 21:15:29.426316 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.426196    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9" event={"ID":"5387c1fa-e1d2-4704-87ad-d6f34f17f1a1","Type":"ContainerDied","Data":"ed64756a4b858313396dad15a99b77233877fa1eaddd6aecb0b5680be55276c7"}
Sep 10 21:15:29.426316 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.426218    2584 scope.go:117] "RemoveContainer" containerID="79cb6da9562646ec21d6697b73c5dc7f4c8a959a5497dae76c4a460226c9c5f2"
Sep 10 21:15:29.435775 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.435754    2584 scope.go:117] "RemoveContainer" containerID="eb7651d83efb0c9fcd159807e2b5fe0fd04c62d582cae19862eada844ea5af61"
Sep 10 21:15:29.444153 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.444135    2584 scope.go:117] "RemoveContainer" containerID="79cb6da9562646ec21d6697b73c5dc7f4c8a959a5497dae76c4a460226c9c5f2"
Sep 10 21:15:29.444447 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:15:29.444414    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79cb6da9562646ec21d6697b73c5dc7f4c8a959a5497dae76c4a460226c9c5f2\": container with ID starting with 79cb6da9562646ec21d6697b73c5dc7f4c8a959a5497dae76c4a460226c9c5f2 not found: ID does not exist" containerID="79cb6da9562646ec21d6697b73c5dc7f4c8a959a5497dae76c4a460226c9c5f2"
Sep 10 21:15:29.444489 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.444457    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79cb6da9562646ec21d6697b73c5dc7f4c8a959a5497dae76c4a460226c9c5f2"} err="failed to get container status \"79cb6da9562646ec21d6697b73c5dc7f4c8a959a5497dae76c4a460226c9c5f2\": rpc error: code = NotFound desc = could not find container \"79cb6da9562646ec21d6697b73c5dc7f4c8a959a5497dae76c4a460226c9c5f2\": container with ID starting with 79cb6da9562646ec21d6697b73c5dc7f4c8a959a5497dae76c4a460226c9c5f2 not found: ID does not exist"
Sep 10 21:15:29.444489 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.444475    2584 scope.go:117] "RemoveContainer" containerID="eb7651d83efb0c9fcd159807e2b5fe0fd04c62d582cae19862eada844ea5af61"
Sep 10 21:15:29.444757 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:15:29.444738    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb7651d83efb0c9fcd159807e2b5fe0fd04c62d582cae19862eada844ea5af61\": container with ID starting with eb7651d83efb0c9fcd159807e2b5fe0fd04c62d582cae19862eada844ea5af61 not found: ID does not exist" containerID="eb7651d83efb0c9fcd159807e2b5fe0fd04c62d582cae19862eada844ea5af61"
Sep 10 21:15:29.444810 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.444759    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb7651d83efb0c9fcd159807e2b5fe0fd04c62d582cae19862eada844ea5af61"} err="failed to get container status \"eb7651d83efb0c9fcd159807e2b5fe0fd04c62d582cae19862eada844ea5af61\": rpc error: code = NotFound desc = could not find container \"eb7651d83efb0c9fcd159807e2b5fe0fd04c62d582cae19862eada844ea5af61\": container with ID starting with eb7651d83efb0c9fcd159807e2b5fe0fd04c62d582cae19862eada844ea5af61 not found: ID does not exist"
Sep 10 21:15:29.451531 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.451510    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"]
Sep 10 21:15:29.455331 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:29.455311    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-2bbc78-openshift-default-7479bc75b4-dmbc9"]
Sep 10 21:15:30.841020 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:15:30.840990    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" path="/var/lib/kubelet/pods/5387c1fa-e1d2-4704-87ad-d6f34f17f1a1/volumes"
Sep 10 21:16:08.775960 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:08.775918    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"]
Sep 10 21:16:09.798959 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:09.798925    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"]
Sep 10 21:16:23.794838 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:23.794769    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg" podUID="97fd7ec2-999c-460a-87e5-4dc53d15d224" containerName="main" containerID="cri-o://9c1d87b9ccbdb7600bd0e1cbf6f98bd8f49856a34cf2c6bc628dedc71bfe30e6" gracePeriod=45
Sep 10 21:16:24.055135 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.055076    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:16:24.159411 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.159377    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97fd7ec2-999c-460a-87e5-4dc53d15d224-dshm\") pod \"97fd7ec2-999c-460a-87e5-4dc53d15d224\" (UID: \"97fd7ec2-999c-460a-87e5-4dc53d15d224\") "
Sep 10 21:16:24.159411 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.159413    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97fd7ec2-999c-460a-87e5-4dc53d15d224-home\") pod \"97fd7ec2-999c-460a-87e5-4dc53d15d224\" (UID: \"97fd7ec2-999c-460a-87e5-4dc53d15d224\") "
Sep 10 21:16:24.159666 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.159479    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97fd7ec2-999c-460a-87e5-4dc53d15d224-tmp-dir\") pod \"97fd7ec2-999c-460a-87e5-4dc53d15d224\" (UID: \"97fd7ec2-999c-460a-87e5-4dc53d15d224\") "
Sep 10 21:16:24.159666 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.159513    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74rx8\" (UniqueName: \"kubernetes.io/projected/97fd7ec2-999c-460a-87e5-4dc53d15d224-kube-api-access-74rx8\") pod \"97fd7ec2-999c-460a-87e5-4dc53d15d224\" (UID: \"97fd7ec2-999c-460a-87e5-4dc53d15d224\") "
Sep 10 21:16:24.159666 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.159529    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97fd7ec2-999c-460a-87e5-4dc53d15d224-model-cache\") pod \"97fd7ec2-999c-460a-87e5-4dc53d15d224\" (UID: \"97fd7ec2-999c-460a-87e5-4dc53d15d224\") "
Sep 10 21:16:24.159666 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.159549    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97fd7ec2-999c-460a-87e5-4dc53d15d224-tls-certs\") pod \"97fd7ec2-999c-460a-87e5-4dc53d15d224\" (UID: \"97fd7ec2-999c-460a-87e5-4dc53d15d224\") "
Sep 10 21:16:24.159878 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.159797    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fd7ec2-999c-460a-87e5-4dc53d15d224-home" (OuterVolumeSpecName: "home") pod "97fd7ec2-999c-460a-87e5-4dc53d15d224" (UID: "97fd7ec2-999c-460a-87e5-4dc53d15d224"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:24.159878 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.159825    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fd7ec2-999c-460a-87e5-4dc53d15d224-model-cache" (OuterVolumeSpecName: "model-cache") pod "97fd7ec2-999c-460a-87e5-4dc53d15d224" (UID: "97fd7ec2-999c-460a-87e5-4dc53d15d224"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:24.159878 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.159804    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fd7ec2-999c-460a-87e5-4dc53d15d224-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "97fd7ec2-999c-460a-87e5-4dc53d15d224" (UID: "97fd7ec2-999c-460a-87e5-4dc53d15d224"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:24.160021 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.159910    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97fd7ec2-999c-460a-87e5-4dc53d15d224-model-cache\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:24.161690 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.161666    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fd7ec2-999c-460a-87e5-4dc53d15d224-dshm" (OuterVolumeSpecName: "dshm") pod "97fd7ec2-999c-460a-87e5-4dc53d15d224" (UID: "97fd7ec2-999c-460a-87e5-4dc53d15d224"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:24.161802 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.161710    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97fd7ec2-999c-460a-87e5-4dc53d15d224-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "97fd7ec2-999c-460a-87e5-4dc53d15d224" (UID: "97fd7ec2-999c-460a-87e5-4dc53d15d224"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:16:24.161866 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.161794    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97fd7ec2-999c-460a-87e5-4dc53d15d224-kube-api-access-74rx8" (OuterVolumeSpecName: "kube-api-access-74rx8") pod "97fd7ec2-999c-460a-87e5-4dc53d15d224" (UID: "97fd7ec2-999c-460a-87e5-4dc53d15d224"). InnerVolumeSpecName "kube-api-access-74rx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:16:24.261336 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.261302    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97fd7ec2-999c-460a-87e5-4dc53d15d224-dshm\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:24.261336 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.261330    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97fd7ec2-999c-460a-87e5-4dc53d15d224-home\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:24.261336 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.261339    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97fd7ec2-999c-460a-87e5-4dc53d15d224-tmp-dir\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:24.261582 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.261350    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74rx8\" (UniqueName: \"kubernetes.io/projected/97fd7ec2-999c-460a-87e5-4dc53d15d224-kube-api-access-74rx8\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:24.261582 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.261359    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97fd7ec2-999c-460a-87e5-4dc53d15d224-tls-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:24.718544 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.718507    2584 generic.go:356] "Generic (PLEG): container finished" podID="97fd7ec2-999c-460a-87e5-4dc53d15d224" containerID="9c1d87b9ccbdb7600bd0e1cbf6f98bd8f49856a34cf2c6bc628dedc71bfe30e6" exitCode=0
Sep 10 21:16:24.718725 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.718551    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"
Sep 10 21:16:24.718725 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.718557    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg" event={"ID":"97fd7ec2-999c-460a-87e5-4dc53d15d224","Type":"ContainerDied","Data":"9c1d87b9ccbdb7600bd0e1cbf6f98bd8f49856a34cf2c6bc628dedc71bfe30e6"}
Sep 10 21:16:24.718725 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.718592    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg" event={"ID":"97fd7ec2-999c-460a-87e5-4dc53d15d224","Type":"ContainerDied","Data":"22ee44e1721282b23974d7e69c874b1465765bbe7227df744b232be987094316"}
Sep 10 21:16:24.718725 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.718608    2584 scope.go:117] "RemoveContainer" containerID="9c1d87b9ccbdb7600bd0e1cbf6f98bd8f49856a34cf2c6bc628dedc71bfe30e6"
Sep 10 21:16:24.730731 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.730709    2584 scope.go:117] "RemoveContainer" containerID="9c1d87b9ccbdb7600bd0e1cbf6f98bd8f49856a34cf2c6bc628dedc71bfe30e6"
Sep 10 21:16:24.731043 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:16:24.731023    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c1d87b9ccbdb7600bd0e1cbf6f98bd8f49856a34cf2c6bc628dedc71bfe30e6\": container with ID starting with 9c1d87b9ccbdb7600bd0e1cbf6f98bd8f49856a34cf2c6bc628dedc71bfe30e6 not found: ID does not exist" containerID="9c1d87b9ccbdb7600bd0e1cbf6f98bd8f49856a34cf2c6bc628dedc71bfe30e6"
Sep 10 21:16:24.731101 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.731064    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c1d87b9ccbdb7600bd0e1cbf6f98bd8f49856a34cf2c6bc628dedc71bfe30e6"} err="failed to get container status \"9c1d87b9ccbdb7600bd0e1cbf6f98bd8f49856a34cf2c6bc628dedc71bfe30e6\": rpc error: code = NotFound desc = could not find container \"9c1d87b9ccbdb7600bd0e1cbf6f98bd8f49856a34cf2c6bc628dedc71bfe30e6\": container with ID starting with 9c1d87b9ccbdb7600bd0e1cbf6f98bd8f49856a34cf2c6bc628dedc71bfe30e6 not found: ID does not exist"
Sep 10 21:16:24.742597 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.742570    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"]
Sep 10 21:16:24.746458 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.746415    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b22a78/test-model-b22a78-kserve-78bb494cdf-49nkg"]
Sep 10 21:16:24.818294 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.818251    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc" podUID="e75427ea-7c98-496b-8a05-af05b767e6e3" containerName="main" containerID="cri-o://6a5c349018403d7a8b9e8cd0e972c6a8cbb6777e65919b81525e1ef95ba6b4ee" gracePeriod=45
Sep 10 21:16:24.840580 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:24.840551    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97fd7ec2-999c-460a-87e5-4dc53d15d224" path="/var/lib/kubelet/pods/97fd7ec2-999c-460a-87e5-4dc53d15d224/volumes"
Sep 10 21:16:25.076308 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.076283    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:16:25.171408 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.171378    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e75427ea-7c98-496b-8a05-af05b767e6e3-model-cache\") pod \"e75427ea-7c98-496b-8a05-af05b767e6e3\" (UID: \"e75427ea-7c98-496b-8a05-af05b767e6e3\") "
Sep 10 21:16:25.171586 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.171448    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qc88z\" (UniqueName: \"kubernetes.io/projected/e75427ea-7c98-496b-8a05-af05b767e6e3-kube-api-access-qc88z\") pod \"e75427ea-7c98-496b-8a05-af05b767e6e3\" (UID: \"e75427ea-7c98-496b-8a05-af05b767e6e3\") "
Sep 10 21:16:25.171586 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.171504    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e75427ea-7c98-496b-8a05-af05b767e6e3-home\") pod \"e75427ea-7c98-496b-8a05-af05b767e6e3\" (UID: \"e75427ea-7c98-496b-8a05-af05b767e6e3\") "
Sep 10 21:16:25.171586 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.171520    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e75427ea-7c98-496b-8a05-af05b767e6e3-tmp-dir\") pod \"e75427ea-7c98-496b-8a05-af05b767e6e3\" (UID: \"e75427ea-7c98-496b-8a05-af05b767e6e3\") "
Sep 10 21:16:25.171727 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.171643    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e75427ea-7c98-496b-8a05-af05b767e6e3-tls-certs\") pod \"e75427ea-7c98-496b-8a05-af05b767e6e3\" (UID: \"e75427ea-7c98-496b-8a05-af05b767e6e3\") "
Sep 10 21:16:25.171727 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.171653    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e75427ea-7c98-496b-8a05-af05b767e6e3-model-cache" (OuterVolumeSpecName: "model-cache") pod "e75427ea-7c98-496b-8a05-af05b767e6e3" (UID: "e75427ea-7c98-496b-8a05-af05b767e6e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:25.171727 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.171693    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e75427ea-7c98-496b-8a05-af05b767e6e3-dshm\") pod \"e75427ea-7c98-496b-8a05-af05b767e6e3\" (UID: \"e75427ea-7c98-496b-8a05-af05b767e6e3\") "
Sep 10 21:16:25.171912 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.171732    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e75427ea-7c98-496b-8a05-af05b767e6e3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e75427ea-7c98-496b-8a05-af05b767e6e3" (UID: "e75427ea-7c98-496b-8a05-af05b767e6e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:25.171912 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.171749    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e75427ea-7c98-496b-8a05-af05b767e6e3-home" (OuterVolumeSpecName: "home") pod "e75427ea-7c98-496b-8a05-af05b767e6e3" (UID: "e75427ea-7c98-496b-8a05-af05b767e6e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:25.172113 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.172088    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e75427ea-7c98-496b-8a05-af05b767e6e3-model-cache\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:25.172113 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.172108    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e75427ea-7c98-496b-8a05-af05b767e6e3-home\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:25.172113 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.172118    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e75427ea-7c98-496b-8a05-af05b767e6e3-tmp-dir\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:25.173838 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.173813    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e75427ea-7c98-496b-8a05-af05b767e6e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e75427ea-7c98-496b-8a05-af05b767e6e3" (UID: "e75427ea-7c98-496b-8a05-af05b767e6e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:16:25.173838 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.173818    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e75427ea-7c98-496b-8a05-af05b767e6e3-dshm" (OuterVolumeSpecName: "dshm") pod "e75427ea-7c98-496b-8a05-af05b767e6e3" (UID: "e75427ea-7c98-496b-8a05-af05b767e6e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:25.173838 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.173834    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e75427ea-7c98-496b-8a05-af05b767e6e3-kube-api-access-qc88z" (OuterVolumeSpecName: "kube-api-access-qc88z") pod "e75427ea-7c98-496b-8a05-af05b767e6e3" (UID: "e75427ea-7c98-496b-8a05-af05b767e6e3"). InnerVolumeSpecName "kube-api-access-qc88z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:16:25.273403 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.273326    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e75427ea-7c98-496b-8a05-af05b767e6e3-tls-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:25.273403 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.273352    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e75427ea-7c98-496b-8a05-af05b767e6e3-dshm\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:25.273403 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.273363    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc88z\" (UniqueName: \"kubernetes.io/projected/e75427ea-7c98-496b-8a05-af05b767e6e3-kube-api-access-qc88z\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:25.726471 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.726410    2584 generic.go:356] "Generic (PLEG): container finished" podID="e75427ea-7c98-496b-8a05-af05b767e6e3" containerID="6a5c349018403d7a8b9e8cd0e972c6a8cbb6777e65919b81525e1ef95ba6b4ee" exitCode=0
Sep 10 21:16:25.726471 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.726471    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"
Sep 10 21:16:25.726732 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.726495    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc" event={"ID":"e75427ea-7c98-496b-8a05-af05b767e6e3","Type":"ContainerDied","Data":"6a5c349018403d7a8b9e8cd0e972c6a8cbb6777e65919b81525e1ef95ba6b4ee"}
Sep 10 21:16:25.726732 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.726545    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc" event={"ID":"e75427ea-7c98-496b-8a05-af05b767e6e3","Type":"ContainerDied","Data":"312ddbbf308dbc3546d6031e7ce8bc4520e589b75693709ca1699dc82d0affe9"}
Sep 10 21:16:25.726732 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.726570    2584 scope.go:117] "RemoveContainer" containerID="6a5c349018403d7a8b9e8cd0e972c6a8cbb6777e65919b81525e1ef95ba6b4ee"
Sep 10 21:16:25.740182 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.740157    2584 scope.go:117] "RemoveContainer" containerID="6a5c349018403d7a8b9e8cd0e972c6a8cbb6777e65919b81525e1ef95ba6b4ee"
Sep 10 21:16:25.740632 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:16:25.740603    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a5c349018403d7a8b9e8cd0e972c6a8cbb6777e65919b81525e1ef95ba6b4ee\": container with ID starting with 6a5c349018403d7a8b9e8cd0e972c6a8cbb6777e65919b81525e1ef95ba6b4ee not found: ID does not exist" containerID="6a5c349018403d7a8b9e8cd0e972c6a8cbb6777e65919b81525e1ef95ba6b4ee"
Sep 10 21:16:25.740720 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.740635    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a5c349018403d7a8b9e8cd0e972c6a8cbb6777e65919b81525e1ef95ba6b4ee"} err="failed to get container status \"6a5c349018403d7a8b9e8cd0e972c6a8cbb6777e65919b81525e1ef95ba6b4ee\": rpc error: code = NotFound desc = could not find container \"6a5c349018403d7a8b9e8cd0e972c6a8cbb6777e65919b81525e1ef95ba6b4ee\": container with ID starting with 6a5c349018403d7a8b9e8cd0e972c6a8cbb6777e65919b81525e1ef95ba6b4ee not found: ID does not exist"
Sep 10 21:16:25.751177 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.751149    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"]
Sep 10 21:16:25.754876 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:25.754853    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-414d5f/test-model-414d5f-kserve-7866545d97-dr6gc"]
Sep 10 21:16:26.840185 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:26.840151    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e75427ea-7c98-496b-8a05-af05b767e6e3" path="/var/lib/kubelet/pods/e75427ea-7c98-496b-8a05-af05b767e6e3/volumes"
Sep 10 21:16:27.910330 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:27.910293    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"]
Sep 10 21:16:27.910820 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:27.910594    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l" podUID="2a7315ba-55f9-495a-bd7e-a917794e9997" containerName="istio-proxy" containerID="cri-o://a05f75ef41098a3e98f0fad32a660f7a335dc056743fc4d8b0855b1b5716d17e" gracePeriod=30
Sep 10 21:16:28.292114 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.292006    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-414d5f-jkt6g"]
Sep 10 21:16:28.292550 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.292529    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e75427ea-7c98-496b-8a05-af05b767e6e3" containerName="main"
Sep 10 21:16:28.292668 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.292552    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e75427ea-7c98-496b-8a05-af05b767e6e3" containerName="main"
Sep 10 21:16:28.292668 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.292576    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" containerName="istio-proxy"
Sep 10 21:16:28.292668 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.292586    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" containerName="istio-proxy"
Sep 10 21:16:28.292668 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.292600    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0544dd1-9220-4630-b88c-06e874a328bd" containerName="main"
Sep 10 21:16:28.292668 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.292608    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0544dd1-9220-4630-b88c-06e874a328bd" containerName="main"
Sep 10 21:16:28.292668 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.292627    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97fd7ec2-999c-460a-87e5-4dc53d15d224" containerName="main"
Sep 10 21:16:28.292668 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.292635    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="97fd7ec2-999c-460a-87e5-4dc53d15d224" containerName="main"
Sep 10 21:16:28.292668 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.292650    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" containerName="istio-proxy"
Sep 10 21:16:28.292668 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.292659    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" containerName="istio-proxy"
Sep 10 21:16:28.293103 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.292761    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="97fd7ec2-999c-460a-87e5-4dc53d15d224" containerName="main"
Sep 10 21:16:28.293103 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.292773    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" containerName="istio-proxy"
Sep 10 21:16:28.293103 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.292785    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5387c1fa-e1d2-4704-87ad-d6f34f17f1a1" containerName="istio-proxy"
Sep 10 21:16:28.293103 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.292794    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0544dd1-9220-4630-b88c-06e874a328bd" containerName="main"
Sep 10 21:16:28.293103 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.292804    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e75427ea-7c98-496b-8a05-af05b767e6e3" containerName="main"
Sep 10 21:16:28.295969 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.295944    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-414d5f-jkt6g"
Sep 10 21:16:28.298995 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.298974    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4gs5\""
Sep 10 21:16:28.299180 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.299155    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:16:28.303472 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.303370    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-414d5f-jkt6g"]
Sep 10 21:16:28.403622 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.403587    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/350992e2-bb6a-41b5-8546-2e7cabd9f9a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-414d5f-jkt6g\" (UID: \"350992e2-bb6a-41b5-8546-2e7cabd9f9a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-414d5f-jkt6g"
Sep 10 21:16:28.504878 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.504843    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/350992e2-bb6a-41b5-8546-2e7cabd9f9a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-414d5f-jkt6g\" (UID: \"350992e2-bb6a-41b5-8546-2e7cabd9f9a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-414d5f-jkt6g"
Sep 10 21:16:28.505565 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.505536    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/350992e2-bb6a-41b5-8546-2e7cabd9f9a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-414d5f-jkt6g\" (UID: \"350992e2-bb6a-41b5-8546-2e7cabd9f9a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-414d5f-jkt6g"
Sep 10 21:16:28.607964 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.607878    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-414d5f-jkt6g"
Sep 10 21:16:28.880416 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:28.880386    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-414d5f-jkt6g"]
Sep 10 21:16:28.882611 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:16:28.882585    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod350992e2_bb6a_41b5_8546_2e7cabd9f9a8.slice/crio-9b82537639aa1f0f4be6a970af5a72b94ad6cd6480861fc53d97e93e18fe6873 WatchSource:0}: Error finding container 9b82537639aa1f0f4be6a970af5a72b94ad6cd6480861fc53d97e93e18fe6873: Status 404 returned error can't find the container with id 9b82537639aa1f0f4be6a970af5a72b94ad6cd6480861fc53d97e93e18fe6873
Sep 10 21:16:29.750839 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:29.750811    2584 generic.go:356] "Generic (PLEG): container finished" podID="350992e2-bb6a-41b5-8546-2e7cabd9f9a8" containerID="c3eb7ede8cedb98a7f2c8d8902077eba7e33c38c4c222aeb5f6ec5fa7c500900" exitCode=0
Sep 10 21:16:29.751201 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:29.750846    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-414d5f-jkt6g" event={"ID":"350992e2-bb6a-41b5-8546-2e7cabd9f9a8","Type":"ContainerDied","Data":"c3eb7ede8cedb98a7f2c8d8902077eba7e33c38c4c222aeb5f6ec5fa7c500900"}
Sep 10 21:16:29.751201 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:29.750880    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-414d5f-jkt6g" event={"ID":"350992e2-bb6a-41b5-8546-2e7cabd9f9a8","Type":"ContainerStarted","Data":"9b82537639aa1f0f4be6a970af5a72b94ad6cd6480861fc53d97e93e18fe6873"}
Sep 10 21:16:30.885665 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:30.885640    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-414d5f-jkt6g"
Sep 10 21:16:31.030488 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:31.030405    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/350992e2-bb6a-41b5-8546-2e7cabd9f9a8-maas-api-service-ca\") pod \"350992e2-bb6a-41b5-8546-2e7cabd9f9a8\" (UID: \"350992e2-bb6a-41b5-8546-2e7cabd9f9a8\") "
Sep 10 21:16:31.030809 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:31.030784    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/350992e2-bb6a-41b5-8546-2e7cabd9f9a8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "350992e2-bb6a-41b5-8546-2e7cabd9f9a8" (UID: "350992e2-bb6a-41b5-8546-2e7cabd9f9a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:16:31.131841 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:31.131812    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/350992e2-bb6a-41b5-8546-2e7cabd9f9a8-maas-api-service-ca\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:31.763929 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:31.763900    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-414d5f-jkt6g"
Sep 10 21:16:31.764218 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:31.763935    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-414d5f-jkt6g" event={"ID":"350992e2-bb6a-41b5-8546-2e7cabd9f9a8","Type":"ContainerDied","Data":"9b82537639aa1f0f4be6a970af5a72b94ad6cd6480861fc53d97e93e18fe6873"}
Sep 10 21:16:31.764218 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:31.764010    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b82537639aa1f0f4be6a970af5a72b94ad6cd6480861fc53d97e93e18fe6873"
Sep 10 21:16:32.553890 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:32.553852    2584 patch_prober.go:28] interesting pod/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 21:16:32.554250 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:32.553915    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l" podUID="2a7315ba-55f9-495a-bd7e-a917794e9997" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 21:16:33.188692 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.188669    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:16:33.254219 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.254190    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-istio-data\") pod \"2a7315ba-55f9-495a-bd7e-a917794e9997\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") "
Sep 10 21:16:33.254367 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.254238    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7315ba-55f9-495a-bd7e-a917794e9997-istiod-ca-cert\") pod \"2a7315ba-55f9-495a-bd7e-a917794e9997\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") "
Sep 10 21:16:33.254367 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.254265    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-credential-socket\") pod \"2a7315ba-55f9-495a-bd7e-a917794e9997\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") "
Sep 10 21:16:33.254367 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.254316    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7315ba-55f9-495a-bd7e-a917794e9997-istio-token\") pod \"2a7315ba-55f9-495a-bd7e-a917794e9997\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") "
Sep 10 21:16:33.254367 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.254354    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk8fb\" (UniqueName: \"kubernetes.io/projected/2a7315ba-55f9-495a-bd7e-a917794e9997-kube-api-access-rk8fb\") pod \"2a7315ba-55f9-495a-bd7e-a917794e9997\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") "
Sep 10 21:16:33.254580 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.254382    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-workload-socket\") pod \"2a7315ba-55f9-495a-bd7e-a917794e9997\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") "
Sep 10 21:16:33.254580 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.254407    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-workload-certs\") pod \"2a7315ba-55f9-495a-bd7e-a917794e9997\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") "
Sep 10 21:16:33.254580 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.254480    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7315ba-55f9-495a-bd7e-a917794e9997-istio-podinfo\") pod \"2a7315ba-55f9-495a-bd7e-a917794e9997\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") "
Sep 10 21:16:33.254580 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.254514    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-istio-envoy\") pod \"2a7315ba-55f9-495a-bd7e-a917794e9997\" (UID: \"2a7315ba-55f9-495a-bd7e-a917794e9997\") "
Sep 10 21:16:33.254787 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.254510    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-istio-data" (OuterVolumeSpecName: "istio-data") pod "2a7315ba-55f9-495a-bd7e-a917794e9997" (UID: "2a7315ba-55f9-495a-bd7e-a917794e9997"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:33.254787 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.254607    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2a7315ba-55f9-495a-bd7e-a917794e9997" (UID: "2a7315ba-55f9-495a-bd7e-a917794e9997"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:33.254787 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.254740    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2a7315ba-55f9-495a-bd7e-a917794e9997" (UID: "2a7315ba-55f9-495a-bd7e-a917794e9997"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:33.254787 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.254682    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a7315ba-55f9-495a-bd7e-a917794e9997-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2a7315ba-55f9-495a-bd7e-a917794e9997" (UID: "2a7315ba-55f9-495a-bd7e-a917794e9997"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:16:33.254787 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.254722    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2a7315ba-55f9-495a-bd7e-a917794e9997" (UID: "2a7315ba-55f9-495a-bd7e-a917794e9997"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:33.255029 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.254957    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7315ba-55f9-495a-bd7e-a917794e9997-istiod-ca-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:33.255029 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.254976    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-credential-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:33.255029 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.254992    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-workload-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:33.255029 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.255006    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-workload-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:33.255029 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.255019    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-istio-data\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:33.256562 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.256535    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7315ba-55f9-495a-bd7e-a917794e9997-kube-api-access-rk8fb" (OuterVolumeSpecName: "kube-api-access-rk8fb") pod "2a7315ba-55f9-495a-bd7e-a917794e9997" (UID: "2a7315ba-55f9-495a-bd7e-a917794e9997"). InnerVolumeSpecName "kube-api-access-rk8fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:16:33.256684 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.256669    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2a7315ba-55f9-495a-bd7e-a917794e9997" (UID: "2a7315ba-55f9-495a-bd7e-a917794e9997"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:33.256951 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.256878    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a7315ba-55f9-495a-bd7e-a917794e9997-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2a7315ba-55f9-495a-bd7e-a917794e9997" (UID: "2a7315ba-55f9-495a-bd7e-a917794e9997"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:16:33.256951 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.256907    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7315ba-55f9-495a-bd7e-a917794e9997-istio-token" (OuterVolumeSpecName: "istio-token") pod "2a7315ba-55f9-495a-bd7e-a917794e9997" (UID: "2a7315ba-55f9-495a-bd7e-a917794e9997"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:16:33.355567 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.355532    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7315ba-55f9-495a-bd7e-a917794e9997-istio-token\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:33.355567 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.355565    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk8fb\" (UniqueName: \"kubernetes.io/projected/2a7315ba-55f9-495a-bd7e-a917794e9997-kube-api-access-rk8fb\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:33.355754 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.355580    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7315ba-55f9-495a-bd7e-a917794e9997-istio-podinfo\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:33.355754 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.355592    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7315ba-55f9-495a-bd7e-a917794e9997-istio-envoy\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:33.777117 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.777083    2584 generic.go:356] "Generic (PLEG): container finished" podID="2a7315ba-55f9-495a-bd7e-a917794e9997" containerID="a05f75ef41098a3e98f0fad32a660f7a335dc056743fc4d8b0855b1b5716d17e" exitCode=0
Sep 10 21:16:33.777547 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.777131    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"
Sep 10 21:16:33.777547 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.777141    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l" event={"ID":"2a7315ba-55f9-495a-bd7e-a917794e9997","Type":"ContainerDied","Data":"a05f75ef41098a3e98f0fad32a660f7a335dc056743fc4d8b0855b1b5716d17e"}
Sep 10 21:16:33.777547 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.777183    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l" event={"ID":"2a7315ba-55f9-495a-bd7e-a917794e9997","Type":"ContainerDied","Data":"0374c9fa7491e77aa7451cafd67cf6075adc89b60e46cb4b7f4da09ba405bce4"}
Sep 10 21:16:33.777547 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.777204    2584 scope.go:117] "RemoveContainer" containerID="a05f75ef41098a3e98f0fad32a660f7a335dc056743fc4d8b0855b1b5716d17e"
Sep 10 21:16:33.787244 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.787228    2584 scope.go:117] "RemoveContainer" containerID="362827d42b99805bb0307779666882da2153b8cdf8d8bedd3a42eb306824c89c"
Sep 10 21:16:33.798157 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.798120    2584 scope.go:117] "RemoveContainer" containerID="a05f75ef41098a3e98f0fad32a660f7a335dc056743fc4d8b0855b1b5716d17e"
Sep 10 21:16:33.798687 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:16:33.798666    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a05f75ef41098a3e98f0fad32a660f7a335dc056743fc4d8b0855b1b5716d17e\": container with ID starting with a05f75ef41098a3e98f0fad32a660f7a335dc056743fc4d8b0855b1b5716d17e not found: ID does not exist" containerID="a05f75ef41098a3e98f0fad32a660f7a335dc056743fc4d8b0855b1b5716d17e"
Sep 10 21:16:33.798798 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.798699    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a05f75ef41098a3e98f0fad32a660f7a335dc056743fc4d8b0855b1b5716d17e"} err="failed to get container status \"a05f75ef41098a3e98f0fad32a660f7a335dc056743fc4d8b0855b1b5716d17e\": rpc error: code = NotFound desc = could not find container \"a05f75ef41098a3e98f0fad32a660f7a335dc056743fc4d8b0855b1b5716d17e\": container with ID starting with a05f75ef41098a3e98f0fad32a660f7a335dc056743fc4d8b0855b1b5716d17e not found: ID does not exist"
Sep 10 21:16:33.798798 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.798723    2584 scope.go:117] "RemoveContainer" containerID="362827d42b99805bb0307779666882da2153b8cdf8d8bedd3a42eb306824c89c"
Sep 10 21:16:33.799025 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:16:33.799007    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"362827d42b99805bb0307779666882da2153b8cdf8d8bedd3a42eb306824c89c\": container with ID starting with 362827d42b99805bb0307779666882da2153b8cdf8d8bedd3a42eb306824c89c not found: ID does not exist" containerID="362827d42b99805bb0307779666882da2153b8cdf8d8bedd3a42eb306824c89c"
Sep 10 21:16:33.799068 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.799033    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"362827d42b99805bb0307779666882da2153b8cdf8d8bedd3a42eb306824c89c"} err="failed to get container status \"362827d42b99805bb0307779666882da2153b8cdf8d8bedd3a42eb306824c89c\": rpc error: code = NotFound desc = could not find container \"362827d42b99805bb0307779666882da2153b8cdf8d8bedd3a42eb306824c89c\": container with ID starting with 362827d42b99805bb0307779666882da2153b8cdf8d8bedd3a42eb306824c89c not found: ID does not exist"
Sep 10 21:16:33.800725 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.800704    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"]
Sep 10 21:16:33.802723 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:33.802703    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b22a78-openshift-default-58f877d5b6-j4v7l"]
Sep 10 21:16:34.839413 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:34.839380    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a7315ba-55f9-495a-bd7e-a917794e9997" path="/var/lib/kubelet/pods/2a7315ba-55f9-495a-bd7e-a917794e9997/volumes"
Sep 10 21:16:38.891813 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:38.891779    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-414d5f-29817915-p2cfg"]
Sep 10 21:16:38.897714 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:38.897688    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-414d5f-29817915-p2cfg"]
Sep 10 21:16:39.080087 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:39.080057    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-414d5f-jkt6g"]
Sep 10 21:16:39.084868 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:39.084842    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-414d5f-jkt6g"]
Sep 10 21:16:40.846289 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:40.846251    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="350992e2-bb6a-41b5-8546-2e7cabd9f9a8" path="/var/lib/kubelet/pods/350992e2-bb6a-41b5-8546-2e7cabd9f9a8/volumes"
Sep 10 21:16:40.846690 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:40.846565    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="513a4bee-f302-47eb-b242-f019625dfb6a" path="/var/lib/kubelet/pods/513a4bee-f302-47eb-b242-f019625dfb6a/volumes"
Sep 10 21:16:46.070570 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:46.070533    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"]
Sep 10 21:16:46.071005 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:46.070748    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt" podUID="c76d25ff-57e2-4366-a081-675f880e63b1" containerName="istio-proxy" containerID="cri-o://4a0a3839eeecfa03c61c5cbefa4e86de5c843ac4de173836700824846dcf38e1" gracePeriod=30
Sep 10 21:16:47.041506 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.041469    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"]
Sep 10 21:16:47.041833 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.041822    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a7315ba-55f9-495a-bd7e-a917794e9997" containerName="istio-proxy"
Sep 10 21:16:47.041879 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.041835    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a7315ba-55f9-495a-bd7e-a917794e9997" containerName="istio-proxy"
Sep 10 21:16:47.041879 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.041857    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="350992e2-bb6a-41b5-8546-2e7cabd9f9a8" containerName="revoke-keys"
Sep 10 21:16:47.041879 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.041862    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="350992e2-bb6a-41b5-8546-2e7cabd9f9a8" containerName="revoke-keys"
Sep 10 21:16:47.041972 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.041920    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="350992e2-bb6a-41b5-8546-2e7cabd9f9a8" containerName="revoke-keys"
Sep 10 21:16:47.041972 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.041930    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a7315ba-55f9-495a-bd7e-a917794e9997" containerName="istio-proxy"
Sep 10 21:16:47.041972 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.041936    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a7315ba-55f9-495a-bd7e-a917794e9997" containerName="istio-proxy"
Sep 10 21:16:47.046516 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.046497    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.049020 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.048995    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-0ac302-openshift-default-dockercfg-jbhjc\""
Sep 10 21:16:47.058675 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.058648    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"]
Sep 10 21:16:47.091441 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.091392    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62491f9c-f6c8-484a-9488-7b38a70cd163-istio-token\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.091882 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.091487    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-workload-socket\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.091882 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.091534    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-credential-socket\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.091882 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.091556    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tps55\" (UniqueName: \"kubernetes.io/projected/62491f9c-f6c8-484a-9488-7b38a70cd163-kube-api-access-tps55\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.091882 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.091590    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-istio-data\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.091882 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.091609    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-istio-envoy\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.091882 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.091662    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62491f9c-f6c8-484a-9488-7b38a70cd163-istio-podinfo\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.091882 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.091695    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-workload-certs\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.091882 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.091712    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62491f9c-f6c8-484a-9488-7b38a70cd163-istiod-ca-cert\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.193043 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.193006    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-workload-socket\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.193043 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.193050    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-credential-socket\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.193290 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.193072    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tps55\" (UniqueName: \"kubernetes.io/projected/62491f9c-f6c8-484a-9488-7b38a70cd163-kube-api-access-tps55\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.193290 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.193115    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-istio-data\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.193290 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.193130    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-istio-envoy\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.193290 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.193150    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62491f9c-f6c8-484a-9488-7b38a70cd163-istio-podinfo\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.193290 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.193189    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-workload-certs\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.193290 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.193215    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62491f9c-f6c8-484a-9488-7b38a70cd163-istiod-ca-cert\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.193290 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.193286    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62491f9c-f6c8-484a-9488-7b38a70cd163-istio-token\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.193682 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.193513    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-workload-socket\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.193682 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.193554    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-istio-data\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.193682 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.193565    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-credential-socket\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.193804 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.193770    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-workload-certs\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.194298 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.194276    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62491f9c-f6c8-484a-9488-7b38a70cd163-istiod-ca-cert\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.195576 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.195555    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-istio-envoy\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.195933 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.195892    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62491f9c-f6c8-484a-9488-7b38a70cd163-istio-podinfo\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.202046 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.202024    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tps55\" (UniqueName: \"kubernetes.io/projected/62491f9c-f6c8-484a-9488-7b38a70cd163-kube-api-access-tps55\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.202216 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.202198    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62491f9c-f6c8-484a-9488-7b38a70cd163-istio-token\") pod \"e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") " pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.356832 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.356736    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:47.633923 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.633897    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"]
Sep 10 21:16:47.636344 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:16:47.636313    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62491f9c_f6c8_484a_9488_7b38a70cd163.slice/crio-a39aaa679ec2d9a4629d8df9fe0344e67490462653513a12ff15d53e0d972ec3 WatchSource:0}: Error finding container a39aaa679ec2d9a4629d8df9fe0344e67490462653513a12ff15d53e0d972ec3: Status 404 returned error can't find the container with id a39aaa679ec2d9a4629d8df9fe0344e67490462653513a12ff15d53e0d972ec3
Sep 10 21:16:47.638709 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.638672    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:16:47.638810 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.638758    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:16:47.638874 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.638807    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:16:47.850986 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.850949    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq" event={"ID":"62491f9c-f6c8-484a-9488-7b38a70cd163","Type":"ContainerStarted","Data":"b3d151da81acfe8a0229f559d3f133d6c4c2d4a636cec4d445b4f5adf98fed96"}
Sep 10 21:16:47.851194 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.851011    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq" event={"ID":"62491f9c-f6c8-484a-9488-7b38a70cd163","Type":"ContainerStarted","Data":"a39aaa679ec2d9a4629d8df9fe0344e67490462653513a12ff15d53e0d972ec3"}
Sep 10 21:16:47.871572 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:47.871515    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq" podStartSLOduration=0.871499347 podStartE2EDuration="871.499347ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:16:47.869605865 +0000 UTC m=+1067.534630666" watchObservedRunningTime="2026-09-10 21:16:47.871499347 +0000 UTC m=+1067.536524146"
Sep 10 21:16:48.357927 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:48.357890    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:48.362528 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:48.362504    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:48.855478 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:48.855447    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:48.856522 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:48.856503    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:16:49.091143 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:49.091100    2584 patch_prober.go:28] interesting pod/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 21:16:49.091306 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:49.091169    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt" podUID="c76d25ff-57e2-4366-a081-675f880e63b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 21:16:51.321170 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.321139    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:16:51.332374 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.332310    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-workload-socket\") pod \"c76d25ff-57e2-4366-a081-675f880e63b1\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") "
Sep 10 21:16:51.332374 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.332357    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlxdn\" (UniqueName: \"kubernetes.io/projected/c76d25ff-57e2-4366-a081-675f880e63b1-kube-api-access-dlxdn\") pod \"c76d25ff-57e2-4366-a081-675f880e63b1\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") "
Sep 10 21:16:51.332585 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.332391    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76d25ff-57e2-4366-a081-675f880e63b1-istio-podinfo\") pod \"c76d25ff-57e2-4366-a081-675f880e63b1\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") "
Sep 10 21:16:51.332585 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.332467    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-istio-data\") pod \"c76d25ff-57e2-4366-a081-675f880e63b1\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") "
Sep 10 21:16:51.332585 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.332523    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-istio-envoy\") pod \"c76d25ff-57e2-4366-a081-675f880e63b1\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") "
Sep 10 21:16:51.332585 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.332550    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76d25ff-57e2-4366-a081-675f880e63b1-istio-token\") pod \"c76d25ff-57e2-4366-a081-675f880e63b1\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") "
Sep 10 21:16:51.332585 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.332583    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76d25ff-57e2-4366-a081-675f880e63b1-istiod-ca-cert\") pod \"c76d25ff-57e2-4366-a081-675f880e63b1\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") "
Sep 10 21:16:51.334674 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.332608    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-credential-socket\") pod \"c76d25ff-57e2-4366-a081-675f880e63b1\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") "
Sep 10 21:16:51.334674 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.332620    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c76d25ff-57e2-4366-a081-675f880e63b1" (UID: "c76d25ff-57e2-4366-a081-675f880e63b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:51.334674 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.332673    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-workload-certs\") pod \"c76d25ff-57e2-4366-a081-675f880e63b1\" (UID: \"c76d25ff-57e2-4366-a081-675f880e63b1\") "
Sep 10 21:16:51.334674 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.333029    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-workload-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:51.334674 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.333290    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76d25ff-57e2-4366-a081-675f880e63b1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c76d25ff-57e2-4366-a081-675f880e63b1" (UID: "c76d25ff-57e2-4366-a081-675f880e63b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:16:51.334674 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.333583    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c76d25ff-57e2-4366-a081-675f880e63b1" (UID: "c76d25ff-57e2-4366-a081-675f880e63b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:51.334674 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.334068    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-istio-data" (OuterVolumeSpecName: "istio-data") pod "c76d25ff-57e2-4366-a081-675f880e63b1" (UID: "c76d25ff-57e2-4366-a081-675f880e63b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:51.334674 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.334192    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c76d25ff-57e2-4366-a081-675f880e63b1" (UID: "c76d25ff-57e2-4366-a081-675f880e63b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:51.335135 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.334872    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c76d25ff-57e2-4366-a081-675f880e63b1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c76d25ff-57e2-4366-a081-675f880e63b1" (UID: "c76d25ff-57e2-4366-a081-675f880e63b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:16:51.335135 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.334989    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76d25ff-57e2-4366-a081-675f880e63b1-kube-api-access-dlxdn" (OuterVolumeSpecName: "kube-api-access-dlxdn") pod "c76d25ff-57e2-4366-a081-675f880e63b1" (UID: "c76d25ff-57e2-4366-a081-675f880e63b1"). InnerVolumeSpecName "kube-api-access-dlxdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:16:51.335373 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.335358    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c76d25ff-57e2-4366-a081-675f880e63b1" (UID: "c76d25ff-57e2-4366-a081-675f880e63b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:51.335536 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.335522    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76d25ff-57e2-4366-a081-675f880e63b1-istio-token" (OuterVolumeSpecName: "istio-token") pod "c76d25ff-57e2-4366-a081-675f880e63b1" (UID: "c76d25ff-57e2-4366-a081-675f880e63b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:16:51.433860 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.433829    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-istio-data\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:51.433860 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.433855    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-istio-envoy\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:51.433860 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.433863    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76d25ff-57e2-4366-a081-675f880e63b1-istio-token\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:51.434074 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.433871    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76d25ff-57e2-4366-a081-675f880e63b1-istiod-ca-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:51.434074 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.433879    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-credential-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:51.434074 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.433888    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76d25ff-57e2-4366-a081-675f880e63b1-workload-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:51.434074 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.433897    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlxdn\" (UniqueName: \"kubernetes.io/projected/c76d25ff-57e2-4366-a081-675f880e63b1-kube-api-access-dlxdn\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:51.434074 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.433906    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76d25ff-57e2-4366-a081-675f880e63b1-istio-podinfo\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:16:51.872299 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.872251    2584 generic.go:356] "Generic (PLEG): container finished" podID="c76d25ff-57e2-4366-a081-675f880e63b1" containerID="4a0a3839eeecfa03c61c5cbefa4e86de5c843ac4de173836700824846dcf38e1" exitCode=0
Sep 10 21:16:51.872488 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.872306    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"
Sep 10 21:16:51.872488 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.872337    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt" event={"ID":"c76d25ff-57e2-4366-a081-675f880e63b1","Type":"ContainerDied","Data":"4a0a3839eeecfa03c61c5cbefa4e86de5c843ac4de173836700824846dcf38e1"}
Sep 10 21:16:51.872488 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.872399    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt" event={"ID":"c76d25ff-57e2-4366-a081-675f880e63b1","Type":"ContainerDied","Data":"6e82b891d30255ee581bd7123fe3c00893c4669f3afae132653ad862144e3069"}
Sep 10 21:16:51.872488 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.872441    2584 scope.go:117] "RemoveContainer" containerID="4a0a3839eeecfa03c61c5cbefa4e86de5c843ac4de173836700824846dcf38e1"
Sep 10 21:16:51.881741 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.881720    2584 scope.go:117] "RemoveContainer" containerID="4a45d119bc0d88279c9c6ced47273ff6b2f4149e58221edf5c735933db89c86d"
Sep 10 21:16:51.889745 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.889730    2584 scope.go:117] "RemoveContainer" containerID="4a0a3839eeecfa03c61c5cbefa4e86de5c843ac4de173836700824846dcf38e1"
Sep 10 21:16:51.890006 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:16:51.889991    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a0a3839eeecfa03c61c5cbefa4e86de5c843ac4de173836700824846dcf38e1\": container with ID starting with 4a0a3839eeecfa03c61c5cbefa4e86de5c843ac4de173836700824846dcf38e1 not found: ID does not exist" containerID="4a0a3839eeecfa03c61c5cbefa4e86de5c843ac4de173836700824846dcf38e1"
Sep 10 21:16:51.890047 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.890015    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a0a3839eeecfa03c61c5cbefa4e86de5c843ac4de173836700824846dcf38e1"} err="failed to get container status \"4a0a3839eeecfa03c61c5cbefa4e86de5c843ac4de173836700824846dcf38e1\": rpc error: code = NotFound desc = could not find container \"4a0a3839eeecfa03c61c5cbefa4e86de5c843ac4de173836700824846dcf38e1\": container with ID starting with 4a0a3839eeecfa03c61c5cbefa4e86de5c843ac4de173836700824846dcf38e1 not found: ID does not exist"
Sep 10 21:16:51.890047 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.890030    2584 scope.go:117] "RemoveContainer" containerID="4a45d119bc0d88279c9c6ced47273ff6b2f4149e58221edf5c735933db89c86d"
Sep 10 21:16:51.890241 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:16:51.890226    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a45d119bc0d88279c9c6ced47273ff6b2f4149e58221edf5c735933db89c86d\": container with ID starting with 4a45d119bc0d88279c9c6ced47273ff6b2f4149e58221edf5c735933db89c86d not found: ID does not exist" containerID="4a45d119bc0d88279c9c6ced47273ff6b2f4149e58221edf5c735933db89c86d"
Sep 10 21:16:51.890281 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.890246    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a45d119bc0d88279c9c6ced47273ff6b2f4149e58221edf5c735933db89c86d"} err="failed to get container status \"4a45d119bc0d88279c9c6ced47273ff6b2f4149e58221edf5c735933db89c86d\": rpc error: code = NotFound desc = could not find container \"4a45d119bc0d88279c9c6ced47273ff6b2f4149e58221edf5c735933db89c86d\": container with ID starting with 4a45d119bc0d88279c9c6ced47273ff6b2f4149e58221edf5c735933db89c86d not found: ID does not exist"
Sep 10 21:16:51.893985 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.893962    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"]
Sep 10 21:16:51.897239 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:51.897218    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-414d5f-openshift-default-5c6c6bb7f6-dfbqt"]
Sep 10 21:16:52.840232 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:16:52.840199    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c76d25ff-57e2-4366-a081-675f880e63b1" path="/var/lib/kubelet/pods/c76d25ff-57e2-4366-a081-675f880e63b1/volumes"
Sep 10 21:17:03.856876 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:03.856843    2584 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 10 21:17:03.857239 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:03.856901    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq" podUID="62491f9c-f6c8-484a-9488-7b38a70cd163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 10 21:17:03.950309 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:03.950239    2584 generic.go:356] "Generic (PLEG): container finished" podID="62491f9c-f6c8-484a-9488-7b38a70cd163" containerID="b3d151da81acfe8a0229f559d3f133d6c4c2d4a636cec4d445b4f5adf98fed96" exitCode=0
Sep 10 21:17:03.950474 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:03.950315    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq" event={"ID":"62491f9c-f6c8-484a-9488-7b38a70cd163","Type":"ContainerDied","Data":"b3d151da81acfe8a0229f559d3f133d6c4c2d4a636cec4d445b4f5adf98fed96"}
Sep 10 21:17:03.950774 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:03.950759    2584 scope.go:117] "RemoveContainer" containerID="b3d151da81acfe8a0229f559d3f133d6c4c2d4a636cec4d445b4f5adf98fed96"
Sep 10 21:17:03.951800 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:03.951775    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:17:03.951875 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:03.951860    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:17:03.951911 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:03.951897    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:17:04.958700 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:04.958656    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq" event={"ID":"62491f9c-f6c8-484a-9488-7b38a70cd163","Type":"ContainerStarted","Data":"5798c08c7c62b3c3943755f9ec172086915bcead5b4274c3fc982c15b479cb0a"}
Sep 10 21:17:05.357246 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:05.357152    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:17:05.357411 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:05.357381    2584 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 10 21:17:05.357513 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:05.357461    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq" podUID="62491f9c-f6c8-484a-9488-7b38a70cd163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 10 21:17:06.357414 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:06.357377    2584 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 10 21:17:06.357839 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:06.357466    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq" podUID="62491f9c-f6c8-484a-9488-7b38a70cd163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 10 21:17:07.357776 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:07.357735    2584 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 10 21:17:07.358175 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:07.357813    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq" podUID="62491f9c-f6c8-484a-9488-7b38a70cd163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 10 21:17:08.357128 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:08.357092    2584 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 10 21:17:08.357304 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:08.357160    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq" podUID="62491f9c-f6c8-484a-9488-7b38a70cd163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 10 21:17:09.357447 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:09.357386    2584 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 10 21:17:09.357933 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:09.357480    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq" podUID="62491f9c-f6c8-484a-9488-7b38a70cd163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 10 21:17:10.362004 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:10.361970    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:17:10.362459 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:10.362295    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:17:10.363585 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:10.363563    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:17:48.095252 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.095215    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"]
Sep 10 21:17:48.095751 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.095733    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c76d25ff-57e2-4366-a081-675f880e63b1" containerName="istio-proxy"
Sep 10 21:17:48.095751 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.095752    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c76d25ff-57e2-4366-a081-675f880e63b1" containerName="istio-proxy"
Sep 10 21:17:48.095895 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.095776    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c76d25ff-57e2-4366-a081-675f880e63b1" containerName="istio-proxy"
Sep 10 21:17:48.095895 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.095784    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c76d25ff-57e2-4366-a081-675f880e63b1" containerName="istio-proxy"
Sep 10 21:17:48.095895 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.095818    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a7315ba-55f9-495a-bd7e-a917794e9997" containerName="istio-proxy"
Sep 10 21:17:48.095895 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.095825    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a7315ba-55f9-495a-bd7e-a917794e9997" containerName="istio-proxy"
Sep 10 21:17:48.095895 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.095881    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c76d25ff-57e2-4366-a081-675f880e63b1" containerName="istio-proxy"
Sep 10 21:17:48.099156 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.099137    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.101522 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.101500    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-4ad710-openshift-default-dockercfg-rsnjk\""
Sep 10 21:17:48.112590 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.112563    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"]
Sep 10 21:17:48.196241 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.196202    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64121917-eb0e-4b8a-91f3-69655dac1b55-istiod-ca-cert\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.196439 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.196250    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64121917-eb0e-4b8a-91f3-69655dac1b55-istio-token\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.196439 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.196358    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-workload-socket\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.196439 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.196403    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-istio-envoy\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.196588 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.196472    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q5vx\" (UniqueName: \"kubernetes.io/projected/64121917-eb0e-4b8a-91f3-69655dac1b55-kube-api-access-5q5vx\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.196588 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.196540    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-istio-data\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.196588 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.196579    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64121917-eb0e-4b8a-91f3-69655dac1b55-istio-podinfo\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.196703 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.196689    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-workload-certs\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.196750 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.196740    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-credential-socket\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.298270 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.298224    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-istio-data\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.298515 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.298285    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64121917-eb0e-4b8a-91f3-69655dac1b55-istio-podinfo\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.298515 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.298331    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-workload-certs\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.298515 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.298371    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-credential-socket\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.298515 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.298400    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64121917-eb0e-4b8a-91f3-69655dac1b55-istiod-ca-cert\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.298515 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.298418    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64121917-eb0e-4b8a-91f3-69655dac1b55-istio-token\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.298515 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.298484    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-workload-socket\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.298515 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.298506    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-istio-envoy\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.298897 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.298536    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q5vx\" (UniqueName: \"kubernetes.io/projected/64121917-eb0e-4b8a-91f3-69655dac1b55-kube-api-access-5q5vx\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.298897 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.298765    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-istio-data\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.298897 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.298795    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-credential-socket\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.298897 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.298887    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-workload-certs\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.299067 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.299036    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-workload-socket\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.299272 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.299251    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64121917-eb0e-4b8a-91f3-69655dac1b55-istiod-ca-cert\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.300853 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.300836    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-istio-envoy\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.300952 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.300934    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64121917-eb0e-4b8a-91f3-69655dac1b55-istio-podinfo\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.312609 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.312572    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q5vx\" (UniqueName: \"kubernetes.io/projected/64121917-eb0e-4b8a-91f3-69655dac1b55-kube-api-access-5q5vx\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.313108 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.313090    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64121917-eb0e-4b8a-91f3-69655dac1b55-istio-token\") pod \"e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") " pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.411047 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.410994    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:48.689348 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.689320    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"]
Sep 10 21:17:48.691197 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:17:48.691169    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64121917_eb0e_4b8a_91f3_69655dac1b55.slice/crio-504dcb6ea4d41c54f07e92b47828436d72591a77cea9f71119d1eef746e7c4cb WatchSource:0}: Error finding container 504dcb6ea4d41c54f07e92b47828436d72591a77cea9f71119d1eef746e7c4cb: Status 404 returned error can't find the container with id 504dcb6ea4d41c54f07e92b47828436d72591a77cea9f71119d1eef746e7c4cb
Sep 10 21:17:48.693440 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.693389    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:17:48.693547 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.693493    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:17:48.693547 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:48.693541    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:17:49.174879 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:49.174838    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh" event={"ID":"64121917-eb0e-4b8a-91f3-69655dac1b55","Type":"ContainerStarted","Data":"0d8ecbae1257923e00a925863def93a27750c4d687429915a64e52e24e238aff"}
Sep 10 21:17:49.175268 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:49.174902    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh" event={"ID":"64121917-eb0e-4b8a-91f3-69655dac1b55","Type":"ContainerStarted","Data":"504dcb6ea4d41c54f07e92b47828436d72591a77cea9f71119d1eef746e7c4cb"}
Sep 10 21:17:49.195026 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:49.194981    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh" podStartSLOduration=1.194966981 podStartE2EDuration="1.194966981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:17:49.19337819 +0000 UTC m=+1128.858402989" watchObservedRunningTime="2026-09-10 21:17:49.194966981 +0000 UTC m=+1128.859991780"
Sep 10 21:17:49.411166 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:49.411134    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:49.415893 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:49.415864    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:50.055931 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.055901    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"]
Sep 10 21:17:50.056357 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.056343    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c76d25ff-57e2-4366-a081-675f880e63b1" containerName="istio-proxy"
Sep 10 21:17:50.059338 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.059322    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:17:50.062358 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.062330    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0ac302\"/\"auth-test-model-0ac302-kserve-self-signed-certs\""
Sep 10 21:17:50.062358 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.062342    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-0ac302\"/\"openshift-service-ca.crt\""
Sep 10 21:17:50.062358 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.062357    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-0ac302\"/\"kube-root-ca.crt\""
Sep 10 21:17:50.062601 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.062363    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0ac302\"/\"default-dockercfg-q2m6v\""
Sep 10 21:17:50.069265 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.069243    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"]
Sep 10 21:17:50.180396 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.180371    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:50.181337 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.181312    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:17:50.218733 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.218700    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f3d505e-09b3-4167-a9a8-d12f03b59077-tmp-dir\") pod \"auth-test-model-0ac302-kserve-648f5db887-pp4nf\" (UID: \"5f3d505e-09b3-4167-a9a8-d12f03b59077\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:17:50.218884 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.218758    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmt7s\" (UniqueName: \"kubernetes.io/projected/5f3d505e-09b3-4167-a9a8-d12f03b59077-kube-api-access-mmt7s\") pod \"auth-test-model-0ac302-kserve-648f5db887-pp4nf\" (UID: \"5f3d505e-09b3-4167-a9a8-d12f03b59077\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:17:50.218884 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.218826    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f3d505e-09b3-4167-a9a8-d12f03b59077-dshm\") pod \"auth-test-model-0ac302-kserve-648f5db887-pp4nf\" (UID: \"5f3d505e-09b3-4167-a9a8-d12f03b59077\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:17:50.218982 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.218882    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f3d505e-09b3-4167-a9a8-d12f03b59077-model-cache\") pod \"auth-test-model-0ac302-kserve-648f5db887-pp4nf\" (UID: \"5f3d505e-09b3-4167-a9a8-d12f03b59077\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:17:50.218982 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.218956    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f3d505e-09b3-4167-a9a8-d12f03b59077-tls-certs\") pod \"auth-test-model-0ac302-kserve-648f5db887-pp4nf\" (UID: \"5f3d505e-09b3-4167-a9a8-d12f03b59077\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:17:50.219062 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.218995    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f3d505e-09b3-4167-a9a8-d12f03b59077-home\") pod \"auth-test-model-0ac302-kserve-648f5db887-pp4nf\" (UID: \"5f3d505e-09b3-4167-a9a8-d12f03b59077\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:17:50.320078 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.319979    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f3d505e-09b3-4167-a9a8-d12f03b59077-tls-certs\") pod \"auth-test-model-0ac302-kserve-648f5db887-pp4nf\" (UID: \"5f3d505e-09b3-4167-a9a8-d12f03b59077\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:17:50.320343 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.320317    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f3d505e-09b3-4167-a9a8-d12f03b59077-home\") pod \"auth-test-model-0ac302-kserve-648f5db887-pp4nf\" (UID: \"5f3d505e-09b3-4167-a9a8-d12f03b59077\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:17:50.320618 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.320590    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f3d505e-09b3-4167-a9a8-d12f03b59077-tmp-dir\") pod \"auth-test-model-0ac302-kserve-648f5db887-pp4nf\" (UID: \"5f3d505e-09b3-4167-a9a8-d12f03b59077\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:17:50.320819 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.320799    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmt7s\" (UniqueName: \"kubernetes.io/projected/5f3d505e-09b3-4167-a9a8-d12f03b59077-kube-api-access-mmt7s\") pod \"auth-test-model-0ac302-kserve-648f5db887-pp4nf\" (UID: \"5f3d505e-09b3-4167-a9a8-d12f03b59077\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:17:50.320978 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.320963    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f3d505e-09b3-4167-a9a8-d12f03b59077-dshm\") pod \"auth-test-model-0ac302-kserve-648f5db887-pp4nf\" (UID: \"5f3d505e-09b3-4167-a9a8-d12f03b59077\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:17:50.321087 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.321073    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f3d505e-09b3-4167-a9a8-d12f03b59077-model-cache\") pod \"auth-test-model-0ac302-kserve-648f5db887-pp4nf\" (UID: \"5f3d505e-09b3-4167-a9a8-d12f03b59077\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:17:50.321212 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.320863    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f3d505e-09b3-4167-a9a8-d12f03b59077-home\") pod \"auth-test-model-0ac302-kserve-648f5db887-pp4nf\" (UID: \"5f3d505e-09b3-4167-a9a8-d12f03b59077\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:17:50.321338 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.321313    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f3d505e-09b3-4167-a9a8-d12f03b59077-tmp-dir\") pod \"auth-test-model-0ac302-kserve-648f5db887-pp4nf\" (UID: \"5f3d505e-09b3-4167-a9a8-d12f03b59077\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:17:50.322042 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.322021    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f3d505e-09b3-4167-a9a8-d12f03b59077-model-cache\") pod \"auth-test-model-0ac302-kserve-648f5db887-pp4nf\" (UID: \"5f3d505e-09b3-4167-a9a8-d12f03b59077\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:17:50.325216 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.324777    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f3d505e-09b3-4167-a9a8-d12f03b59077-dshm\") pod \"auth-test-model-0ac302-kserve-648f5db887-pp4nf\" (UID: \"5f3d505e-09b3-4167-a9a8-d12f03b59077\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:17:50.325216 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.325175    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f3d505e-09b3-4167-a9a8-d12f03b59077-tls-certs\") pod \"auth-test-model-0ac302-kserve-648f5db887-pp4nf\" (UID: \"5f3d505e-09b3-4167-a9a8-d12f03b59077\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:17:50.329823 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.329795    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmt7s\" (UniqueName: \"kubernetes.io/projected/5f3d505e-09b3-4167-a9a8-d12f03b59077-kube-api-access-mmt7s\") pod \"auth-test-model-0ac302-kserve-648f5db887-pp4nf\" (UID: \"5f3d505e-09b3-4167-a9a8-d12f03b59077\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:17:50.369868 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.369829    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:17:50.639513 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:50.639481    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"]
Sep 10 21:17:50.643158 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:17:50.643129    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f3d505e_09b3_4167_a9a8_d12f03b59077.slice/crio-ae58ec67ab9f10799f972827c06ce951fc8d1e4e2416e0173b47b2fca0687fed WatchSource:0}: Error finding container ae58ec67ab9f10799f972827c06ce951fc8d1e4e2416e0173b47b2fca0687fed: Status 404 returned error can't find the container with id ae58ec67ab9f10799f972827c06ce951fc8d1e4e2416e0173b47b2fca0687fed
Sep 10 21:17:51.187510 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:51.187396    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf" event={"ID":"5f3d505e-09b3-4167-a9a8-d12f03b59077","Type":"ContainerStarted","Data":"4a17809c66d1e2a87617fd84f4c2111b8744d9c2bb30934304670ed95d8ca9fe"}
Sep 10 21:17:51.187999 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:51.187531    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf" event={"ID":"5f3d505e-09b3-4167-a9a8-d12f03b59077","Type":"ContainerStarted","Data":"ae58ec67ab9f10799f972827c06ce951fc8d1e4e2416e0173b47b2fca0687fed"}
Sep 10 21:17:51.209818 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:17:51.209746    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf" podStartSLOduration=1.209724005 podStartE2EDuration="1.209724005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:17:51.208375119 +0000 UTC m=+1130.873399923" watchObservedRunningTime="2026-09-10 21:17:51.209724005 +0000 UTC m=+1130.874748807"
Sep 10 21:18:00.370670 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:00.370633    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:18:00.371061 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:00.370698    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:18:00.383351 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:00.383321    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:18:01.250399 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:01.250365    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:18:11.291876 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:11.291842    2584 generic.go:356] "Generic (PLEG): container finished" podID="64121917-eb0e-4b8a-91f3-69655dac1b55" containerID="0d8ecbae1257923e00a925863def93a27750c4d687429915a64e52e24e238aff" exitCode=0
Sep 10 21:18:11.292318 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:11.291917    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh" event={"ID":"64121917-eb0e-4b8a-91f3-69655dac1b55","Type":"ContainerDied","Data":"0d8ecbae1257923e00a925863def93a27750c4d687429915a64e52e24e238aff"}
Sep 10 21:18:11.292366 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:11.292334    2584 scope.go:117] "RemoveContainer" containerID="0d8ecbae1257923e00a925863def93a27750c4d687429915a64e52e24e238aff"
Sep 10 21:18:11.293376 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:11.293348    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:18:11.293509 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:11.293447    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:18:11.293509 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:11.293485    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:18:12.299261 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:12.299229    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh" event={"ID":"64121917-eb0e-4b8a-91f3-69655dac1b55","Type":"ContainerStarted","Data":"62f50709b3e7b7cff78f1e2920eb10b5fd85490de90d8dcaf17471852e3d3469"}
Sep 10 21:18:12.411500 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:12.411465    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:18:12.411838 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:12.411816    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 10 21:18:12.411987 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:12.411959    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh" podUID="64121917-eb0e-4b8a-91f3-69655dac1b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 10 21:18:13.411925 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:13.411880    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 10 21:18:13.412460 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:13.411978    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh" podUID="64121917-eb0e-4b8a-91f3-69655dac1b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 10 21:18:14.416937 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:14.416895    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 10 21:18:14.417579 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:14.417543    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh" podUID="64121917-eb0e-4b8a-91f3-69655dac1b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 10 21:18:15.412534 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:15.412490    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 10 21:18:15.412795 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:15.412562    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh" podUID="64121917-eb0e-4b8a-91f3-69655dac1b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 10 21:18:16.411445 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:16.411381    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 10 21:18:16.412057 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:16.411485    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh" podUID="64121917-eb0e-4b8a-91f3-69655dac1b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 10 21:18:16.980763 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:16.980726    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8122-b1a68d"]
Sep 10 21:18:16.984798 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:16.984772    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8122-b1a68d"
Sep 10 21:18:16.987014 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:16.986990    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cc2nx\""
Sep 10 21:18:16.991019 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:16.990997    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8122-b1a68d"]
Sep 10 21:18:17.077122 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:17.077078    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfr7f\" (UniqueName: \"kubernetes.io/projected/061f21b2-6086-41f9-b5b3-598d779acf60-kube-api-access-kfr7f\") pod \"test-curl-8122-b1a68d\" (UID: \"061f21b2-6086-41f9-b5b3-598d779acf60\") " pod="opendatahub/test-curl-8122-b1a68d"
Sep 10 21:18:17.178519 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:17.178484    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfr7f\" (UniqueName: \"kubernetes.io/projected/061f21b2-6086-41f9-b5b3-598d779acf60-kube-api-access-kfr7f\") pod \"test-curl-8122-b1a68d\" (UID: \"061f21b2-6086-41f9-b5b3-598d779acf60\") " pod="opendatahub/test-curl-8122-b1a68d"
Sep 10 21:18:17.189382 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:17.189359    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfr7f\" (UniqueName: \"kubernetes.io/projected/061f21b2-6086-41f9-b5b3-598d779acf60-kube-api-access-kfr7f\") pod \"test-curl-8122-b1a68d\" (UID: \"061f21b2-6086-41f9-b5b3-598d779acf60\") " pod="opendatahub/test-curl-8122-b1a68d"
Sep 10 21:18:17.294994 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:17.294900    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8122-b1a68d"
Sep 10 21:18:17.544643 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:17.544609    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:18:17.545139 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:17.544942    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:18:17.545795 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:17.545776    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:18:17.579029 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:17.579001    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8122-b1a68d"]
Sep 10 21:18:17.581181 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:18:17.581151    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod061f21b2_6086_41f9_b5b3_598d779acf60.slice/crio-dbc44aa389b224946747a137ceed03c576a713ce8bd28278dde958ff28558061 WatchSource:0}: Error finding container dbc44aa389b224946747a137ceed03c576a713ce8bd28278dde958ff28558061: Status 404 returned error can't find the container with id dbc44aa389b224946747a137ceed03c576a713ce8bd28278dde958ff28558061
Sep 10 21:18:17.583314 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:17.583297    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:18:18.355231 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:18.355205    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8122-b1a68d" event={"ID":"061f21b2-6086-41f9-b5b3-598d779acf60","Type":"ContainerStarted","Data":"dbc44aa389b224946747a137ceed03c576a713ce8bd28278dde958ff28558061"}
Sep 10 21:18:19.362163 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:19.362127    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8122-b1a68d" event={"ID":"061f21b2-6086-41f9-b5b3-598d779acf60","Type":"ContainerStarted","Data":"002a88f40f95aecd5fe279f9c7e450fe9501c036d2208bec0e1f4d45606e687c"}
Sep 10 21:18:19.379066 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:19.379014    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8122-b1a68d" podStartSLOduration=2.66512773 podStartE2EDuration="3.378997691s" totalImagesPullingTime="713.869961ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:18:19.376791939 +0000 UTC m=+1159.041816741" watchObservedRunningTime="2026-09-10 21:18:19.378997691 +0000 UTC m=+1159.044022505"
Sep 10 21:18:19.985622 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:19.985585    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8122-b1a68d"]
Sep 10 21:18:19.990504 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:19.990474    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8122-b1a68d"]
Sep 10 21:18:20.114007 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:20.113928    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8122-e939a2"]
Sep 10 21:18:20.114328 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:20.114315    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="061f21b2-6086-41f9-b5b3-598d779acf60" containerName="test-curl-8122-b1a68d"
Sep 10 21:18:20.114372 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:20.114330    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="061f21b2-6086-41f9-b5b3-598d779acf60" containerName="test-curl-8122-b1a68d"
Sep 10 21:18:20.114407 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:20.114392    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="061f21b2-6086-41f9-b5b3-598d779acf60" containerName="test-curl-8122-b1a68d"
Sep 10 21:18:20.117415 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:20.117399    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8122-e939a2"
Sep 10 21:18:20.124960 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:20.124937    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8122-e939a2"]
Sep 10 21:18:20.209070 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:20.209035    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcs4t\" (UniqueName: \"kubernetes.io/projected/a76bf2d6-49cb-4a56-9f09-7225b6583869-kube-api-access-lcs4t\") pod \"test-curl-8122-e939a2\" (UID: \"a76bf2d6-49cb-4a56-9f09-7225b6583869\") " pod="opendatahub/test-curl-8122-e939a2"
Sep 10 21:18:20.310536 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:20.310505    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcs4t\" (UniqueName: \"kubernetes.io/projected/a76bf2d6-49cb-4a56-9f09-7225b6583869-kube-api-access-lcs4t\") pod \"test-curl-8122-e939a2\" (UID: \"a76bf2d6-49cb-4a56-9f09-7225b6583869\") " pod="opendatahub/test-curl-8122-e939a2"
Sep 10 21:18:20.319769 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:20.319745    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcs4t\" (UniqueName: \"kubernetes.io/projected/a76bf2d6-49cb-4a56-9f09-7225b6583869-kube-api-access-lcs4t\") pod \"test-curl-8122-e939a2\" (UID: \"a76bf2d6-49cb-4a56-9f09-7225b6583869\") " pod="opendatahub/test-curl-8122-e939a2"
Sep 10 21:18:20.367197 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:18:20.367139    2584 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8122-b1a68d" container="test-curl-8122-b1a68d"
Sep 10 21:18:20.427538 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:20.427506    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8122-e939a2"
Sep 10 21:18:20.700857 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:20.700830    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8122-e939a2"]
Sep 10 21:18:20.703174 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:18:20.703140    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda76bf2d6_49cb_4a56_9f09_7225b6583869.slice/crio-7971ce73b9eee2c66aced045c9c8b853aae231ee2a62488e73eb41ffe9e72556 WatchSource:0}: Error finding container 7971ce73b9eee2c66aced045c9c8b853aae231ee2a62488e73eb41ffe9e72556: Status 404 returned error can't find the container with id 7971ce73b9eee2c66aced045c9c8b853aae231ee2a62488e73eb41ffe9e72556
Sep 10 21:18:21.373832 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:21.373789    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8122-e939a2" event={"ID":"a76bf2d6-49cb-4a56-9f09-7225b6583869","Type":"ContainerStarted","Data":"a03d18436be572a0f6244e448a6fb3aa3040b2db8ebad6187f78bcd870aec96f"}
Sep 10 21:18:21.374261 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:21.373839    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8122-e939a2" event={"ID":"a76bf2d6-49cb-4a56-9f09-7225b6583869","Type":"ContainerStarted","Data":"7971ce73b9eee2c66aced045c9c8b853aae231ee2a62488e73eb41ffe9e72556"}
Sep 10 21:18:21.374261 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:21.373958    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8122-b1a68d" podUID="061f21b2-6086-41f9-b5b3-598d779acf60" containerName="test-curl-8122-b1a68d" containerID="cri-o://002a88f40f95aecd5fe279f9c7e450fe9501c036d2208bec0e1f4d45606e687c" gracePeriod=2
Sep 10 21:18:21.390534 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:21.390489    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8122-e939a2" podStartSLOduration=1.390473907 podStartE2EDuration="1.390473907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:18:21.388868154 +0000 UTC m=+1161.053892956" watchObservedRunningTime="2026-09-10 21:18:21.390473907 +0000 UTC m=+1161.055498707"
Sep 10 21:18:22.007067 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:22.007036    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8122-e939a2"]
Sep 10 21:18:22.010973 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:22.010948    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8122-e939a2"]
Sep 10 21:18:22.132399 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:22.132366    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8122-bf6159"]
Sep 10 21:18:22.132785 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:22.132770    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a76bf2d6-49cb-4a56-9f09-7225b6583869" containerName="test-curl-8122-e939a2"
Sep 10 21:18:22.132831 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:22.132787    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a76bf2d6-49cb-4a56-9f09-7225b6583869" containerName="test-curl-8122-e939a2"
Sep 10 21:18:22.139158 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:22.139123    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a76bf2d6-49cb-4a56-9f09-7225b6583869" containerName="test-curl-8122-e939a2"
Sep 10 21:18:22.143839 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:22.143815    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8122-bf6159"]
Sep 10 21:18:22.143992 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:22.143970    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8122-bf6159"
Sep 10 21:18:22.230638 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:22.230599    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2l86\" (UniqueName: \"kubernetes.io/projected/b51cd936-9123-426f-bc62-0ddd7d07ee19-kube-api-access-g2l86\") pod \"test-curl-8122-bf6159\" (UID: \"b51cd936-9123-426f-bc62-0ddd7d07ee19\") " pod="opendatahub/test-curl-8122-bf6159"
Sep 10 21:18:22.331736 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:22.331702    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2l86\" (UniqueName: \"kubernetes.io/projected/b51cd936-9123-426f-bc62-0ddd7d07ee19-kube-api-access-g2l86\") pod \"test-curl-8122-bf6159\" (UID: \"b51cd936-9123-426f-bc62-0ddd7d07ee19\") " pod="opendatahub/test-curl-8122-bf6159"
Sep 10 21:18:22.340882 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:22.340854    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2l86\" (UniqueName: \"kubernetes.io/projected/b51cd936-9123-426f-bc62-0ddd7d07ee19-kube-api-access-g2l86\") pod \"test-curl-8122-bf6159\" (UID: \"b51cd936-9123-426f-bc62-0ddd7d07ee19\") " pod="opendatahub/test-curl-8122-bf6159"
Sep 10 21:18:22.379299 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:18:22.379274    2584 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8122-e939a2" container="test-curl-8122-e939a2"
Sep 10 21:18:22.453563 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:22.453494    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8122-bf6159"
Sep 10 21:18:22.724132 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:22.724110    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8122-bf6159"]
Sep 10 21:18:22.726703 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:18:22.726672    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb51cd936_9123_426f_bc62_0ddd7d07ee19.slice/crio-659449282f59a9c7bcb140866d19530b0fcab1cdec4b84706147fa2cc14de695 WatchSource:0}: Error finding container 659449282f59a9c7bcb140866d19530b0fcab1cdec4b84706147fa2cc14de695: Status 404 returned error can't find the container with id 659449282f59a9c7bcb140866d19530b0fcab1cdec4b84706147fa2cc14de695
Sep 10 21:18:23.385990 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:23.385958    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8122-bf6159" event={"ID":"b51cd936-9123-426f-bc62-0ddd7d07ee19","Type":"ContainerStarted","Data":"736345f783f36dbdbf2f72ae5aadbd679f151ccbd2c1472e77710a2b5ceaf420"}
Sep 10 21:18:23.386315 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:23.386009    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8122-bf6159" event={"ID":"b51cd936-9123-426f-bc62-0ddd7d07ee19","Type":"ContainerStarted","Data":"659449282f59a9c7bcb140866d19530b0fcab1cdec4b84706147fa2cc14de695"}
Sep 10 21:18:23.386315 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:23.386113    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8122-e939a2" podUID="a76bf2d6-49cb-4a56-9f09-7225b6583869" containerName="test-curl-8122-e939a2" containerID="cri-o://a03d18436be572a0f6244e448a6fb3aa3040b2db8ebad6187f78bcd870aec96f" gracePeriod=2
Sep 10 21:18:23.404222 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:23.404162    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8122-bf6159" podStartSLOduration=1.4041397199999999 podStartE2EDuration="1.40413972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:18:23.401533204 +0000 UTC m=+1163.066558005" watchObservedRunningTime="2026-09-10 21:18:23.40413972 +0000 UTC m=+1163.069164522"
Sep 10 21:18:23.515072 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:23.515049    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8122-b1a68d"
Sep 10 21:18:23.644563 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:23.644483    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfr7f\" (UniqueName: \"kubernetes.io/projected/061f21b2-6086-41f9-b5b3-598d779acf60-kube-api-access-kfr7f\") pod \"061f21b2-6086-41f9-b5b3-598d779acf60\" (UID: \"061f21b2-6086-41f9-b5b3-598d779acf60\") "
Sep 10 21:18:23.646621 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:23.646590    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061f21b2-6086-41f9-b5b3-598d779acf60-kube-api-access-kfr7f" (OuterVolumeSpecName: "kube-api-access-kfr7f") pod "061f21b2-6086-41f9-b5b3-598d779acf60" (UID: "061f21b2-6086-41f9-b5b3-598d779acf60"). InnerVolumeSpecName "kube-api-access-kfr7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:18:23.745970 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:23.745937    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfr7f\" (UniqueName: \"kubernetes.io/projected/061f21b2-6086-41f9-b5b3-598d779acf60-kube-api-access-kfr7f\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:18:23.998167 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:23.998066    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8122-bf6159"]
Sep 10 21:18:24.004212 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:24.004180    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8122-bf6159"]
Sep 10 21:18:24.123104 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:24.123062    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8122-d17c76"]
Sep 10 21:18:24.123575 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:24.123556    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b51cd936-9123-426f-bc62-0ddd7d07ee19" containerName="test-curl-8122-bf6159"
Sep 10 21:18:24.123575 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:24.123577    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b51cd936-9123-426f-bc62-0ddd7d07ee19" containerName="test-curl-8122-bf6159"
Sep 10 21:18:24.123729 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:24.123662    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b51cd936-9123-426f-bc62-0ddd7d07ee19" containerName="test-curl-8122-bf6159"
Sep 10 21:18:24.126628 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:24.126608    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8122-d17c76"
Sep 10 21:18:24.131415 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:24.131386    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8122-d17c76"]
Sep 10 21:18:24.250640 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:24.250549    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp7z5\" (UniqueName: \"kubernetes.io/projected/da3d2c00-4c73-45b1-8138-9cc296fce1e7-kube-api-access-jp7z5\") pod \"test-curl-8122-d17c76\" (UID: \"da3d2c00-4c73-45b1-8138-9cc296fce1e7\") " pod="opendatahub/test-curl-8122-d17c76"
Sep 10 21:18:24.351261 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:24.351225    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp7z5\" (UniqueName: \"kubernetes.io/projected/da3d2c00-4c73-45b1-8138-9cc296fce1e7-kube-api-access-jp7z5\") pod \"test-curl-8122-d17c76\" (UID: \"da3d2c00-4c73-45b1-8138-9cc296fce1e7\") " pod="opendatahub/test-curl-8122-d17c76"
Sep 10 21:18:24.364688 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:24.364663    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp7z5\" (UniqueName: \"kubernetes.io/projected/da3d2c00-4c73-45b1-8138-9cc296fce1e7-kube-api-access-jp7z5\") pod \"test-curl-8122-d17c76\" (UID: \"da3d2c00-4c73-45b1-8138-9cc296fce1e7\") " pod="opendatahub/test-curl-8122-d17c76"
Sep 10 21:18:24.392614 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:24.392584    2584 generic.go:356] "Generic (PLEG): container finished" podID="061f21b2-6086-41f9-b5b3-598d779acf60" containerID="002a88f40f95aecd5fe279f9c7e450fe9501c036d2208bec0e1f4d45606e687c" exitCode=137
Sep 10 21:18:24.393049 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:24.392622    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8122-b1a68d"
Sep 10 21:18:24.393049 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:24.392683    2584 scope.go:117] "RemoveContainer" containerID="002a88f40f95aecd5fe279f9c7e450fe9501c036d2208bec0e1f4d45606e687c"
Sep 10 21:18:24.393175 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:18:24.393052    2584 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8122-bf6159" container="test-curl-8122-bf6159"
Sep 10 21:18:24.401366 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:24.401347    2584 scope.go:117] "RemoveContainer" containerID="002a88f40f95aecd5fe279f9c7e450fe9501c036d2208bec0e1f4d45606e687c"
Sep 10 21:18:24.401653 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:18:24.401635    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"002a88f40f95aecd5fe279f9c7e450fe9501c036d2208bec0e1f4d45606e687c\": container with ID starting with 002a88f40f95aecd5fe279f9c7e450fe9501c036d2208bec0e1f4d45606e687c not found: ID does not exist" containerID="002a88f40f95aecd5fe279f9c7e450fe9501c036d2208bec0e1f4d45606e687c"
Sep 10 21:18:24.401703 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:24.401660    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002a88f40f95aecd5fe279f9c7e450fe9501c036d2208bec0e1f4d45606e687c"} err="failed to get container status \"002a88f40f95aecd5fe279f9c7e450fe9501c036d2208bec0e1f4d45606e687c\": rpc error: code = NotFound desc = could not find container \"002a88f40f95aecd5fe279f9c7e450fe9501c036d2208bec0e1f4d45606e687c\": container with ID starting with 002a88f40f95aecd5fe279f9c7e450fe9501c036d2208bec0e1f4d45606e687c not found: ID does not exist"
Sep 10 21:18:24.403251 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:18:24.403232    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8122-b1a68d\" is forbidden: User \"system:node:ip-10-0-135-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-205.ec2.internal' and this object" podUID="061f21b2-6086-41f9-b5b3-598d779acf60" pod="opendatahub/test-curl-8122-b1a68d"
Sep 10 21:18:24.437711 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:24.437665    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8122-d17c76"
Sep 10 21:18:24.712341 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:24.712309    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8122-d17c76"]
Sep 10 21:18:24.714653 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:18:24.714623    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda3d2c00_4c73_45b1_8138_9cc296fce1e7.slice/crio-a9a33f901e40e60da8bc9b9e52caa203f03e0c6d156ba3d60ffbb1597e99f903 WatchSource:0}: Error finding container a9a33f901e40e60da8bc9b9e52caa203f03e0c6d156ba3d60ffbb1597e99f903: Status 404 returned error can't find the container with id a9a33f901e40e60da8bc9b9e52caa203f03e0c6d156ba3d60ffbb1597e99f903
Sep 10 21:18:24.842205 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:24.842168    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="061f21b2-6086-41f9-b5b3-598d779acf60" path="/var/lib/kubelet/pods/061f21b2-6086-41f9-b5b3-598d779acf60/volumes"
Sep 10 21:18:25.401189 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:25.401156    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8122-d17c76" event={"ID":"da3d2c00-4c73-45b1-8138-9cc296fce1e7","Type":"ContainerStarted","Data":"235a24ee4770feeb153e87f42241683b325b9eb3577b1164b3ba3f4719c5b579"}
Sep 10 21:18:25.401567 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:25.401211    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8122-d17c76" event={"ID":"da3d2c00-4c73-45b1-8138-9cc296fce1e7","Type":"ContainerStarted","Data":"a9a33f901e40e60da8bc9b9e52caa203f03e0c6d156ba3d60ffbb1597e99f903"}
Sep 10 21:18:25.402791 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:25.402762    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8122-bf6159" podUID="b51cd936-9123-426f-bc62-0ddd7d07ee19" containerName="test-curl-8122-bf6159" containerID="cri-o://736345f783f36dbdbf2f72ae5aadbd679f151ccbd2c1472e77710a2b5ceaf420" gracePeriod=2
Sep 10 21:18:25.420208 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:25.420144    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8122-d17c76" podStartSLOduration=1.4201229610000001 podStartE2EDuration="1.420122961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:18:25.418000921 +0000 UTC m=+1165.083025722" watchObservedRunningTime="2026-09-10 21:18:25.420122961 +0000 UTC m=+1165.085147765"
Sep 10 21:18:25.583597 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:25.583562    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8122-e939a2"
Sep 10 21:18:25.665719 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:25.665618    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcs4t\" (UniqueName: \"kubernetes.io/projected/a76bf2d6-49cb-4a56-9f09-7225b6583869-kube-api-access-lcs4t\") pod \"a76bf2d6-49cb-4a56-9f09-7225b6583869\" (UID: \"a76bf2d6-49cb-4a56-9f09-7225b6583869\") "
Sep 10 21:18:25.667869 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:25.667840    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76bf2d6-49cb-4a56-9f09-7225b6583869-kube-api-access-lcs4t" (OuterVolumeSpecName: "kube-api-access-lcs4t") pod "a76bf2d6-49cb-4a56-9f09-7225b6583869" (UID: "a76bf2d6-49cb-4a56-9f09-7225b6583869"). InnerVolumeSpecName "kube-api-access-lcs4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:18:25.766699 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:25.766666    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcs4t\" (UniqueName: \"kubernetes.io/projected/a76bf2d6-49cb-4a56-9f09-7225b6583869-kube-api-access-lcs4t\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:18:26.043385 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:26.043302    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8122-d17c76"]
Sep 10 21:18:26.047600 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:26.047564    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8122-d17c76"]
Sep 10 21:18:26.409876 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:26.409844    2584 generic.go:356] "Generic (PLEG): container finished" podID="a76bf2d6-49cb-4a56-9f09-7225b6583869" containerID="a03d18436be572a0f6244e448a6fb3aa3040b2db8ebad6187f78bcd870aec96f" exitCode=137
Sep 10 21:18:26.410334 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:26.409904    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8122-e939a2"
Sep 10 21:18:26.410334 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:26.409961    2584 scope.go:117] "RemoveContainer" containerID="a03d18436be572a0f6244e448a6fb3aa3040b2db8ebad6187f78bcd870aec96f"
Sep 10 21:18:26.410498 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:18:26.410338    2584 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8122-d17c76" container="test-curl-8122-d17c76"
Sep 10 21:18:26.420167 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:26.420147    2584 scope.go:117] "RemoveContainer" containerID="a03d18436be572a0f6244e448a6fb3aa3040b2db8ebad6187f78bcd870aec96f"
Sep 10 21:18:26.420495 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:18:26.420472    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a03d18436be572a0f6244e448a6fb3aa3040b2db8ebad6187f78bcd870aec96f\": container with ID starting with a03d18436be572a0f6244e448a6fb3aa3040b2db8ebad6187f78bcd870aec96f not found: ID does not exist" containerID="a03d18436be572a0f6244e448a6fb3aa3040b2db8ebad6187f78bcd870aec96f"
Sep 10 21:18:26.420572 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:26.420506    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a03d18436be572a0f6244e448a6fb3aa3040b2db8ebad6187f78bcd870aec96f"} err="failed to get container status \"a03d18436be572a0f6244e448a6fb3aa3040b2db8ebad6187f78bcd870aec96f\": rpc error: code = NotFound desc = could not find container \"a03d18436be572a0f6244e448a6fb3aa3040b2db8ebad6187f78bcd870aec96f\": container with ID starting with a03d18436be572a0f6244e448a6fb3aa3040b2db8ebad6187f78bcd870aec96f not found: ID does not exist"
Sep 10 21:18:26.421154 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:18:26.421136    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8122-e939a2\" is forbidden: User \"system:node:ip-10-0-135-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-205.ec2.internal' and this object" podUID="a76bf2d6-49cb-4a56-9f09-7225b6583869" pod="opendatahub/test-curl-8122-e939a2"
Sep 10 21:18:26.842358 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:26.842281    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a76bf2d6-49cb-4a56-9f09-7225b6583869" path="/var/lib/kubelet/pods/a76bf2d6-49cb-4a56-9f09-7225b6583869/volumes"
Sep 10 21:18:26.971230 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:26.971198    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"]
Sep 10 21:18:26.971805 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:26.971782    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3d2c00-4c73-45b1-8138-9cc296fce1e7" containerName="test-curl-8122-d17c76"
Sep 10 21:18:26.971805 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:26.971807    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3d2c00-4c73-45b1-8138-9cc296fce1e7" containerName="test-curl-8122-d17c76"
Sep 10 21:18:26.971959 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:26.971933    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="da3d2c00-4c73-45b1-8138-9cc296fce1e7" containerName="test-curl-8122-d17c76"
Sep 10 21:18:26.981385 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:26.981359    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:26.984062 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:26.984044    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7f376d02-openshift-default-dockercfg-lq9bn\""
Sep 10 21:18:26.997076 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:26.997039    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"]
Sep 10 21:18:27.080112 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.080060    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-workload-socket\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.080300 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.080121    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istio-envoy\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.080399 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.080309    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istiod-ca-cert\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.080474 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.080398    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-credential-socket\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.080474 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.080463    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istio-data\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.080590 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.080527    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mb7l\" (UniqueName: \"kubernetes.io/projected/0d17bff8-357c-450d-a7b4-6a133b7e0a31-kube-api-access-9mb7l\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.080686 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.080658    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istio-podinfo\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.080686 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.080690    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-workload-certs\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.080880 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.080817    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istio-token\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.181721 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.181671    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istiod-ca-cert\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.181924 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.181742    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-credential-socket\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.181924 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.181767    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istio-data\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.181924 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.181798    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mb7l\" (UniqueName: \"kubernetes.io/projected/0d17bff8-357c-450d-a7b4-6a133b7e0a31-kube-api-access-9mb7l\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.181924 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.181861    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istio-podinfo\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.181924 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.181889    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-workload-certs\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.182258 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.181956    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istio-token\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.182258 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.182021    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-workload-socket\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.182258 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.182056    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istio-envoy\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.182970 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.182939    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-credential-socket\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.183308 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.183260    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istio-data\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.183517 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.183489    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-workload-socket\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.183707 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.183681    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-workload-certs\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.183707 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.183697    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istiod-ca-cert\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.189233 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.189204    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istio-podinfo\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.189349 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.189331    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istio-envoy\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.191878 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.191857    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istio-token\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.191975 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.191953    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mb7l\" (UniqueName: \"kubernetes.io/projected/0d17bff8-357c-450d-a7b4-6a133b7e0a31-kube-api-access-9mb7l\") pod \"e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") " pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.306096 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.305676    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:27.439311 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.438764    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8122-d17c76" podUID="da3d2c00-4c73-45b1-8138-9cc296fce1e7" containerName="test-curl-8122-d17c76" containerID="cri-o://235a24ee4770feeb153e87f42241683b325b9eb3577b1164b3ba3f4719c5b579" gracePeriod=2
Sep 10 21:18:27.587834 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.587808    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8122-bf6159"
Sep 10 21:18:27.608280 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.608253    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"]
Sep 10 21:18:27.611023 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:18:27.610983    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d17bff8_357c_450d_a7b4_6a133b7e0a31.slice/crio-bf42c49abfc60dbaff28be29f5b41a62ddb9a737fe6d517d57215461385f46ae WatchSource:0}: Error finding container bf42c49abfc60dbaff28be29f5b41a62ddb9a737fe6d517d57215461385f46ae: Status 404 returned error can't find the container with id bf42c49abfc60dbaff28be29f5b41a62ddb9a737fe6d517d57215461385f46ae
Sep 10 21:18:27.613703 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.613672    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:18:27.613773 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.613737    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:18:27.613773 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.613766    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:18:27.689222 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.689184    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2l86\" (UniqueName: \"kubernetes.io/projected/b51cd936-9123-426f-bc62-0ddd7d07ee19-kube-api-access-g2l86\") pod \"b51cd936-9123-426f-bc62-0ddd7d07ee19\" (UID: \"b51cd936-9123-426f-bc62-0ddd7d07ee19\") "
Sep 10 21:18:27.697757 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.695894    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51cd936-9123-426f-bc62-0ddd7d07ee19-kube-api-access-g2l86" (OuterVolumeSpecName: "kube-api-access-g2l86") pod "b51cd936-9123-426f-bc62-0ddd7d07ee19" (UID: "b51cd936-9123-426f-bc62-0ddd7d07ee19"). InnerVolumeSpecName "kube-api-access-g2l86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:18:27.790155 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:27.790121    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2l86\" (UniqueName: \"kubernetes.io/projected/b51cd936-9123-426f-bc62-0ddd7d07ee19-kube-api-access-g2l86\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:18:28.446007 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:28.445964    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr" event={"ID":"0d17bff8-357c-450d-a7b4-6a133b7e0a31","Type":"ContainerStarted","Data":"41fcb2825db249f1059bdeac96bcc080660d5d8455b49153062055907b3d5f59"}
Sep 10 21:18:28.446448 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:28.446022    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr" event={"ID":"0d17bff8-357c-450d-a7b4-6a133b7e0a31","Type":"ContainerStarted","Data":"bf42c49abfc60dbaff28be29f5b41a62ddb9a737fe6d517d57215461385f46ae"}
Sep 10 21:18:28.447550 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:28.447519    2584 generic.go:356] "Generic (PLEG): container finished" podID="b51cd936-9123-426f-bc62-0ddd7d07ee19" containerID="736345f783f36dbdbf2f72ae5aadbd679f151ccbd2c1472e77710a2b5ceaf420" exitCode=137
Sep 10 21:18:28.447653 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:28.447557    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8122-bf6159"
Sep 10 21:18:28.447653 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:28.447609    2584 scope.go:117] "RemoveContainer" containerID="736345f783f36dbdbf2f72ae5aadbd679f151ccbd2c1472e77710a2b5ceaf420"
Sep 10 21:18:28.457582 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:28.457552    2584 scope.go:117] "RemoveContainer" containerID="736345f783f36dbdbf2f72ae5aadbd679f151ccbd2c1472e77710a2b5ceaf420"
Sep 10 21:18:28.457868 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:18:28.457849    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"736345f783f36dbdbf2f72ae5aadbd679f151ccbd2c1472e77710a2b5ceaf420\": container with ID starting with 736345f783f36dbdbf2f72ae5aadbd679f151ccbd2c1472e77710a2b5ceaf420 not found: ID does not exist" containerID="736345f783f36dbdbf2f72ae5aadbd679f151ccbd2c1472e77710a2b5ceaf420"
Sep 10 21:18:28.457945 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:28.457873    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"736345f783f36dbdbf2f72ae5aadbd679f151ccbd2c1472e77710a2b5ceaf420"} err="failed to get container status \"736345f783f36dbdbf2f72ae5aadbd679f151ccbd2c1472e77710a2b5ceaf420\": rpc error: code = NotFound desc = could not find container \"736345f783f36dbdbf2f72ae5aadbd679f151ccbd2c1472e77710a2b5ceaf420\": container with ID starting with 736345f783f36dbdbf2f72ae5aadbd679f151ccbd2c1472e77710a2b5ceaf420 not found: ID does not exist"
Sep 10 21:18:28.470220 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:28.470173    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr" podStartSLOduration=2.4701578189999998 podStartE2EDuration="2.470157819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:18:28.469286138 +0000 UTC m=+1168.134310939" watchObservedRunningTime="2026-09-10 21:18:28.470157819 +0000 UTC m=+1168.135182619"
Sep 10 21:18:28.471381 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:18:28.471356    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8122-bf6159\" is forbidden: User \"system:node:ip-10-0-135-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-205.ec2.internal' and this object" podUID="b51cd936-9123-426f-bc62-0ddd7d07ee19" pod="opendatahub/test-curl-8122-bf6159"
Sep 10 21:18:28.842577 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:28.842497    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b51cd936-9123-426f-bc62-0ddd7d07ee19" path="/var/lib/kubelet/pods/b51cd936-9123-426f-bc62-0ddd7d07ee19/volumes"
Sep 10 21:18:29.307332 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:29.307295    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:29.313295 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:29.313270    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:29.459267 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:29.459222    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:29.460565 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:29.460544    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:29.660935 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:29.660854    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8122-d17c76"
Sep 10 21:18:29.816998 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:29.816964    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jp7z5\" (UniqueName: \"kubernetes.io/projected/da3d2c00-4c73-45b1-8138-9cc296fce1e7-kube-api-access-jp7z5\") pod \"da3d2c00-4c73-45b1-8138-9cc296fce1e7\" (UID: \"da3d2c00-4c73-45b1-8138-9cc296fce1e7\") "
Sep 10 21:18:29.819052 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:29.819016    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3d2c00-4c73-45b1-8138-9cc296fce1e7-kube-api-access-jp7z5" (OuterVolumeSpecName: "kube-api-access-jp7z5") pod "da3d2c00-4c73-45b1-8138-9cc296fce1e7" (UID: "da3d2c00-4c73-45b1-8138-9cc296fce1e7"). InnerVolumeSpecName "kube-api-access-jp7z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:18:29.919919 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:29.919827    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp7z5\" (UniqueName: \"kubernetes.io/projected/da3d2c00-4c73-45b1-8138-9cc296fce1e7-kube-api-access-jp7z5\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:18:30.465211 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:30.465175    2584 generic.go:356] "Generic (PLEG): container finished" podID="da3d2c00-4c73-45b1-8138-9cc296fce1e7" containerID="235a24ee4770feeb153e87f42241683b325b9eb3577b1164b3ba3f4719c5b579" exitCode=137
Sep 10 21:18:30.465211 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:30.465214    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8122-d17c76"
Sep 10 21:18:30.465748 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:30.465272    2584 scope.go:117] "RemoveContainer" containerID="235a24ee4770feeb153e87f42241683b325b9eb3577b1164b3ba3f4719c5b579"
Sep 10 21:18:30.473907 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:30.473888    2584 scope.go:117] "RemoveContainer" containerID="235a24ee4770feeb153e87f42241683b325b9eb3577b1164b3ba3f4719c5b579"
Sep 10 21:18:30.474180 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:18:30.474160    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"235a24ee4770feeb153e87f42241683b325b9eb3577b1164b3ba3f4719c5b579\": container with ID starting with 235a24ee4770feeb153e87f42241683b325b9eb3577b1164b3ba3f4719c5b579 not found: ID does not exist" containerID="235a24ee4770feeb153e87f42241683b325b9eb3577b1164b3ba3f4719c5b579"
Sep 10 21:18:30.474224 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:30.474187    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235a24ee4770feeb153e87f42241683b325b9eb3577b1164b3ba3f4719c5b579"} err="failed to get container status \"235a24ee4770feeb153e87f42241683b325b9eb3577b1164b3ba3f4719c5b579\": rpc error: code = NotFound desc = could not find container \"235a24ee4770feeb153e87f42241683b325b9eb3577b1164b3ba3f4719c5b579\": container with ID starting with 235a24ee4770feeb153e87f42241683b325b9eb3577b1164b3ba3f4719c5b579 not found: ID does not exist"
Sep 10 21:18:30.475520 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:18:30.475497    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8122-d17c76\" is forbidden: User \"system:node:ip-10-0-135-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-205.ec2.internal' and this object" podUID="da3d2c00-4c73-45b1-8138-9cc296fce1e7" pod="opendatahub/test-curl-8122-d17c76"
Sep 10 21:18:30.840017 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:18:30.839932    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8122-d17c76\" is forbidden: User \"system:node:ip-10-0-135-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-205.ec2.internal' and this object" podUID="da3d2c00-4c73-45b1-8138-9cc296fce1e7" pod="opendatahub/test-curl-8122-d17c76"
Sep 10 21:18:30.842504 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:30.842480    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da3d2c00-4c73-45b1-8138-9cc296fce1e7" path="/var/lib/kubelet/pods/da3d2c00-4c73-45b1-8138-9cc296fce1e7/volumes"
Sep 10 21:18:40.474725 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.474694    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"]
Sep 10 21:18:40.480456 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.480413    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:18:40.484897 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.484867    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-16bbec\"/\"kube-root-ca.crt\""
Sep 10 21:18:40.485030 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.484916    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-16bbec\"/\"auth-test-model-16bbec-kserve-self-signed-certs\""
Sep 10 21:18:40.485030 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.484953    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-16bbec\"/\"default-dockercfg-nwr5g\""
Sep 10 21:18:40.485030 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.485022    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-16bbec\"/\"openshift-service-ca.crt\""
Sep 10 21:18:40.488578 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.488557    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"]
Sep 10 21:18:40.530036 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.530002    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca940185-2e99-4f51-8ca7-8aa2b534d893-home\") pod \"auth-test-model-16bbec-kserve-557dc88ccf-jn7nh\" (UID: \"ca940185-2e99-4f51-8ca7-8aa2b534d893\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:18:40.530036 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.530038    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca940185-2e99-4f51-8ca7-8aa2b534d893-model-cache\") pod \"auth-test-model-16bbec-kserve-557dc88ccf-jn7nh\" (UID: \"ca940185-2e99-4f51-8ca7-8aa2b534d893\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:18:40.530303 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.530058    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca940185-2e99-4f51-8ca7-8aa2b534d893-tmp-dir\") pod \"auth-test-model-16bbec-kserve-557dc88ccf-jn7nh\" (UID: \"ca940185-2e99-4f51-8ca7-8aa2b534d893\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:18:40.530303 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.530141    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca940185-2e99-4f51-8ca7-8aa2b534d893-dshm\") pod \"auth-test-model-16bbec-kserve-557dc88ccf-jn7nh\" (UID: \"ca940185-2e99-4f51-8ca7-8aa2b534d893\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:18:40.530303 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.530206    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca940185-2e99-4f51-8ca7-8aa2b534d893-tls-certs\") pod \"auth-test-model-16bbec-kserve-557dc88ccf-jn7nh\" (UID: \"ca940185-2e99-4f51-8ca7-8aa2b534d893\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:18:40.530303 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.530251    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4xzr\" (UniqueName: \"kubernetes.io/projected/ca940185-2e99-4f51-8ca7-8aa2b534d893-kube-api-access-x4xzr\") pod \"auth-test-model-16bbec-kserve-557dc88ccf-jn7nh\" (UID: \"ca940185-2e99-4f51-8ca7-8aa2b534d893\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:18:40.631544 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.631512    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca940185-2e99-4f51-8ca7-8aa2b534d893-dshm\") pod \"auth-test-model-16bbec-kserve-557dc88ccf-jn7nh\" (UID: \"ca940185-2e99-4f51-8ca7-8aa2b534d893\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:18:40.631709 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.631556    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca940185-2e99-4f51-8ca7-8aa2b534d893-tls-certs\") pod \"auth-test-model-16bbec-kserve-557dc88ccf-jn7nh\" (UID: \"ca940185-2e99-4f51-8ca7-8aa2b534d893\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:18:40.631709 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.631596    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4xzr\" (UniqueName: \"kubernetes.io/projected/ca940185-2e99-4f51-8ca7-8aa2b534d893-kube-api-access-x4xzr\") pod \"auth-test-model-16bbec-kserve-557dc88ccf-jn7nh\" (UID: \"ca940185-2e99-4f51-8ca7-8aa2b534d893\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:18:40.631709 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.631687    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca940185-2e99-4f51-8ca7-8aa2b534d893-home\") pod \"auth-test-model-16bbec-kserve-557dc88ccf-jn7nh\" (UID: \"ca940185-2e99-4f51-8ca7-8aa2b534d893\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:18:40.631835 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.631713    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca940185-2e99-4f51-8ca7-8aa2b534d893-model-cache\") pod \"auth-test-model-16bbec-kserve-557dc88ccf-jn7nh\" (UID: \"ca940185-2e99-4f51-8ca7-8aa2b534d893\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:18:40.631835 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.631737    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca940185-2e99-4f51-8ca7-8aa2b534d893-tmp-dir\") pod \"auth-test-model-16bbec-kserve-557dc88ccf-jn7nh\" (UID: \"ca940185-2e99-4f51-8ca7-8aa2b534d893\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:18:40.632080 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.632053    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca940185-2e99-4f51-8ca7-8aa2b534d893-home\") pod \"auth-test-model-16bbec-kserve-557dc88ccf-jn7nh\" (UID: \"ca940185-2e99-4f51-8ca7-8aa2b534d893\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:18:40.632080 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.632073    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca940185-2e99-4f51-8ca7-8aa2b534d893-model-cache\") pod \"auth-test-model-16bbec-kserve-557dc88ccf-jn7nh\" (UID: \"ca940185-2e99-4f51-8ca7-8aa2b534d893\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:18:40.632240 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.632137    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca940185-2e99-4f51-8ca7-8aa2b534d893-tmp-dir\") pod \"auth-test-model-16bbec-kserve-557dc88ccf-jn7nh\" (UID: \"ca940185-2e99-4f51-8ca7-8aa2b534d893\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:18:40.633858 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.633839    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca940185-2e99-4f51-8ca7-8aa2b534d893-dshm\") pod \"auth-test-model-16bbec-kserve-557dc88ccf-jn7nh\" (UID: \"ca940185-2e99-4f51-8ca7-8aa2b534d893\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:18:40.634127 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.634106    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca940185-2e99-4f51-8ca7-8aa2b534d893-tls-certs\") pod \"auth-test-model-16bbec-kserve-557dc88ccf-jn7nh\" (UID: \"ca940185-2e99-4f51-8ca7-8aa2b534d893\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:18:40.641824 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.641798    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4xzr\" (UniqueName: \"kubernetes.io/projected/ca940185-2e99-4f51-8ca7-8aa2b534d893-kube-api-access-x4xzr\") pod \"auth-test-model-16bbec-kserve-557dc88ccf-jn7nh\" (UID: \"ca940185-2e99-4f51-8ca7-8aa2b534d893\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:18:40.791324 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:40.791236    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:18:41.069738 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:41.069679    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"]
Sep 10 21:18:41.072288 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:18:41.072259    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca940185_2e99_4f51_8ca7_8aa2b534d893.slice/crio-16f9233b181cb8c497e620d79b012be2448048027fa596b264e8cf5f22838578 WatchSource:0}: Error finding container 16f9233b181cb8c497e620d79b012be2448048027fa596b264e8cf5f22838578: Status 404 returned error can't find the container with id 16f9233b181cb8c497e620d79b012be2448048027fa596b264e8cf5f22838578
Sep 10 21:18:41.528824 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:41.528784    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh" event={"ID":"ca940185-2e99-4f51-8ca7-8aa2b534d893","Type":"ContainerStarted","Data":"16c58e627f22b240236e6963d8362090d60543c6125a0f61f128793bc72d310e"}
Sep 10 21:18:41.528824 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:41.528834    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh" event={"ID":"ca940185-2e99-4f51-8ca7-8aa2b534d893","Type":"ContainerStarted","Data":"16f9233b181cb8c497e620d79b012be2448048027fa596b264e8cf5f22838578"}
Sep 10 21:18:41.549616 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:41.549566    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh" podStartSLOduration=1.549546267 podStartE2EDuration="1.549546267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:18:41.547511265 +0000 UTC m=+1181.212536089" watchObservedRunningTime="2026-09-10 21:18:41.549546267 +0000 UTC m=+1181.214571072"
Sep 10 21:18:50.791703 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:50.791664    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:18:50.792101 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:50.791867    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:18:50.805023 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:50.804992    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:18:51.582816 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:51.582782    2584 generic.go:356] "Generic (PLEG): container finished" podID="0d17bff8-357c-450d-a7b4-6a133b7e0a31" containerID="41fcb2825db249f1059bdeac96bcc080660d5d8455b49153062055907b3d5f59" exitCode=0
Sep 10 21:18:51.583070 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:51.582857    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr" event={"ID":"0d17bff8-357c-450d-a7b4-6a133b7e0a31","Type":"ContainerDied","Data":"41fcb2825db249f1059bdeac96bcc080660d5d8455b49153062055907b3d5f59"}
Sep 10 21:18:51.583338 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:51.583322    2584 scope.go:117] "RemoveContainer" containerID="41fcb2825db249f1059bdeac96bcc080660d5d8455b49153062055907b3d5f59"
Sep 10 21:18:51.584521 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:51.584465    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:18:51.584612 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:51.584529    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:18:51.584612 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:51.584563    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:18:51.597448 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:51.597407    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:18:52.590472 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:52.590415    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr" event={"ID":"0d17bff8-357c-450d-a7b4-6a133b7e0a31","Type":"ContainerStarted","Data":"6731cafcb1679328d083096f1357ba2579b2e46d00ef7d817ba2752eaef82b27"}
Sep 10 21:18:53.307417 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:53.307374    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:53.307797 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:53.307771    2584 patch_prober.go:28] interesting pod/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 10 21:18:53.307959 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:53.307933    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr" podUID="0d17bff8-357c-450d-a7b4-6a133b7e0a31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 10 21:18:54.306667 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:54.306625    2584 patch_prober.go:28] interesting pod/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 10 21:18:54.307148 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:54.306693    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr" podUID="0d17bff8-357c-450d-a7b4-6a133b7e0a31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 10 21:18:55.306817 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:55.306770    2584 patch_prober.go:28] interesting pod/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 10 21:18:55.307320 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:55.306847    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr" podUID="0d17bff8-357c-450d-a7b4-6a133b7e0a31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 10 21:18:56.307382 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:56.307337    2584 patch_prober.go:28] interesting pod/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 10 21:18:56.307895 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:56.307404    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr" podUID="0d17bff8-357c-450d-a7b4-6a133b7e0a31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 10 21:18:57.307409 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:57.307366    2584 patch_prober.go:28] interesting pod/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 10 21:18:57.307923 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:57.307464    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr" podUID="0d17bff8-357c-450d-a7b4-6a133b7e0a31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 10 21:18:58.311304 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:58.311271    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:58.311772 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:58.311576    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:18:58.312257 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:18:58.312239    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:19:00.661954 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.661920    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"]
Sep 10 21:19:00.665458 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.665415    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:19:00.668628 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.668606    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7f376d02\"/\"kube-root-ca.crt\""
Sep 10 21:19:00.668739 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.668640    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7f376d02\"/\"e2e-lifecycle-model-7f376d02-kserve-self-signed-certs\""
Sep 10 21:19:00.668739 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.668652    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7f376d02\"/\"default-dockercfg-c9fqb\""
Sep 10 21:19:00.668739 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.668661    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7f376d02\"/\"openshift-service-ca.crt\""
Sep 10 21:19:00.674876 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.674846    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"]
Sep 10 21:19:00.748572 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.748541    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5798357d-5d13-4a88-acae-215eedfa75df-dshm\") pod \"e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h\" (UID: \"5798357d-5d13-4a88-acae-215eedfa75df\") " pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:19:00.748749 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.748580    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5798357d-5d13-4a88-acae-215eedfa75df-tmp-dir\") pod \"e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h\" (UID: \"5798357d-5d13-4a88-acae-215eedfa75df\") " pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:19:00.748749 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.748615    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5798357d-5d13-4a88-acae-215eedfa75df-tls-certs\") pod \"e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h\" (UID: \"5798357d-5d13-4a88-acae-215eedfa75df\") " pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:19:00.748749 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.748689    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5798357d-5d13-4a88-acae-215eedfa75df-home\") pod \"e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h\" (UID: \"5798357d-5d13-4a88-acae-215eedfa75df\") " pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:19:00.748749 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.748742    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzz9j\" (UniqueName: \"kubernetes.io/projected/5798357d-5d13-4a88-acae-215eedfa75df-kube-api-access-gzz9j\") pod \"e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h\" (UID: \"5798357d-5d13-4a88-acae-215eedfa75df\") " pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:19:00.748928 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.748772    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5798357d-5d13-4a88-acae-215eedfa75df-model-cache\") pod \"e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h\" (UID: \"5798357d-5d13-4a88-acae-215eedfa75df\") " pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:19:00.826538 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.826511    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-jkkfm_fd81843e-6283-43e3-a571-d5e47be26e38/console-operator/1.log"
Sep 10 21:19:00.830216 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.830197    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-jkkfm_fd81843e-6283-43e3-a571-d5e47be26e38/console-operator/1.log"
Sep 10 21:19:00.849727 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.849702    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5798357d-5d13-4a88-acae-215eedfa75df-home\") pod \"e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h\" (UID: \"5798357d-5d13-4a88-acae-215eedfa75df\") " pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:19:00.849851 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.849749    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzz9j\" (UniqueName: \"kubernetes.io/projected/5798357d-5d13-4a88-acae-215eedfa75df-kube-api-access-gzz9j\") pod \"e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h\" (UID: \"5798357d-5d13-4a88-acae-215eedfa75df\") " pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:19:00.849851 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.849777    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5798357d-5d13-4a88-acae-215eedfa75df-model-cache\") pod \"e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h\" (UID: \"5798357d-5d13-4a88-acae-215eedfa75df\") " pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:19:00.849851 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.849843    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5798357d-5d13-4a88-acae-215eedfa75df-dshm\") pod \"e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h\" (UID: \"5798357d-5d13-4a88-acae-215eedfa75df\") " pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:19:00.849981 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.849870    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5798357d-5d13-4a88-acae-215eedfa75df-tmp-dir\") pod \"e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h\" (UID: \"5798357d-5d13-4a88-acae-215eedfa75df\") " pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:19:00.849981 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.849907    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5798357d-5d13-4a88-acae-215eedfa75df-tls-certs\") pod \"e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h\" (UID: \"5798357d-5d13-4a88-acae-215eedfa75df\") " pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:19:00.850131 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.850107    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5798357d-5d13-4a88-acae-215eedfa75df-home\") pod \"e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h\" (UID: \"5798357d-5d13-4a88-acae-215eedfa75df\") " pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:19:00.850227 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.850152    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5798357d-5d13-4a88-acae-215eedfa75df-model-cache\") pod \"e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h\" (UID: \"5798357d-5d13-4a88-acae-215eedfa75df\") " pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:19:00.850289 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.850263    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5798357d-5d13-4a88-acae-215eedfa75df-tmp-dir\") pod \"e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h\" (UID: \"5798357d-5d13-4a88-acae-215eedfa75df\") " pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:19:00.859186 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.852064    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5798357d-5d13-4a88-acae-215eedfa75df-dshm\") pod \"e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h\" (UID: \"5798357d-5d13-4a88-acae-215eedfa75df\") " pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:19:00.859186 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.852206    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7f376d02\"/\"e2e-lifecycle-model-7f376d02-kserve-self-signed-certs\""
Sep 10 21:19:00.859186 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.858340    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7f376d02\"/\"kube-root-ca.crt\""
Sep 10 21:19:00.863147 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.863122    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5798357d-5d13-4a88-acae-215eedfa75df-tls-certs\") pod \"e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h\" (UID: \"5798357d-5d13-4a88-acae-215eedfa75df\") " pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:19:00.869131 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.869115    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7f376d02\"/\"openshift-service-ca.crt\""
Sep 10 21:19:00.879379 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.879348    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzz9j\" (UniqueName: \"kubernetes.io/projected/5798357d-5d13-4a88-acae-215eedfa75df-kube-api-access-gzz9j\") pod \"e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h\" (UID: \"5798357d-5d13-4a88-acae-215eedfa75df\") " pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:19:00.891548 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.891528    2584 scope.go:117] "RemoveContainer" containerID="73b1be22ed39fe3273a51c42c33054d62786f5c44c5027bb287bef47a9666145"
Sep 10 21:19:00.978136 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.978099    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7f376d02\"/\"default-dockercfg-c9fqb\""
Sep 10 21:19:00.986192 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:00.986066    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:19:01.260632 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:01.260461    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"]
Sep 10 21:19:01.262726 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:19:01.262701    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5798357d_5d13_4a88_acae_215eedfa75df.slice/crio-4e6428bec18c83d24ef8e15d2cd0a6cbc43559c99990a575d24282662bad0771 WatchSource:0}: Error finding container 4e6428bec18c83d24ef8e15d2cd0a6cbc43559c99990a575d24282662bad0771: Status 404 returned error can't find the container with id 4e6428bec18c83d24ef8e15d2cd0a6cbc43559c99990a575d24282662bad0771
Sep 10 21:19:01.650105 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:01.650070    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h" event={"ID":"5798357d-5d13-4a88-acae-215eedfa75df","Type":"ContainerStarted","Data":"5d703a1e5c4dad98e70611c1f7bfc70a59594a2531447c08320c264a2b3660dc"}
Sep 10 21:19:01.650304 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:01.650115    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h" event={"ID":"5798357d-5d13-4a88-acae-215eedfa75df","Type":"ContainerStarted","Data":"4e6428bec18c83d24ef8e15d2cd0a6cbc43559c99990a575d24282662bad0771"}
Sep 10 21:19:01.669636 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:01.669581    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h" podStartSLOduration=1.669557582 podStartE2EDuration="1.669557582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:19:01.667260611 +0000 UTC m=+1201.332285422" watchObservedRunningTime="2026-09-10 21:19:01.669557582 +0000 UTC m=+1201.334582580"
Sep 10 21:19:10.987273 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:10.987235    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:19:10.987273 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:10.987282    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:19:10.999751 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:10.999724    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:19:11.715467 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:11.715419    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:19:40.216120 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:40.216085    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f376d02-r6hwr"]
Sep 10 21:19:40.219646 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:40.219629    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f376d02-r6hwr"
Sep 10 21:19:40.221691 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:40.221671    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:19:40.221691 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:40.221670    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4gs5\""
Sep 10 21:19:40.229347 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:40.229321    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f376d02-r6hwr"]
Sep 10 21:19:40.361679 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:40.361644    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbebff3-e15b-43e5-acfa-57e5c025f71e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7f376d02-r6hwr\" (UID: \"dcbebff3-e15b-43e5-acfa-57e5c025f71e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f376d02-r6hwr"
Sep 10 21:19:40.462782 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:40.462749    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbebff3-e15b-43e5-acfa-57e5c025f71e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7f376d02-r6hwr\" (UID: \"dcbebff3-e15b-43e5-acfa-57e5c025f71e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f376d02-r6hwr"
Sep 10 21:19:40.463376 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:40.463352    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbebff3-e15b-43e5-acfa-57e5c025f71e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7f376d02-r6hwr\" (UID: \"dcbebff3-e15b-43e5-acfa-57e5c025f71e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f376d02-r6hwr"
Sep 10 21:19:40.531654 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:40.531550    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f376d02-r6hwr"
Sep 10 21:19:40.801691 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:40.801537    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f376d02-r6hwr"]
Sep 10 21:19:40.804463 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:19:40.804404    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbebff3_e15b_43e5_acfa_57e5c025f71e.slice/crio-6739db56d15d4f8ed9151c3d5096dcdce3e994c083f3dfda5bb1e6369f36094e WatchSource:0}: Error finding container 6739db56d15d4f8ed9151c3d5096dcdce3e994c083f3dfda5bb1e6369f36094e: Status 404 returned error can't find the container with id 6739db56d15d4f8ed9151c3d5096dcdce3e994c083f3dfda5bb1e6369f36094e
Sep 10 21:19:40.866003 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:40.865975    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f376d02-r6hwr" event={"ID":"dcbebff3-e15b-43e5-acfa-57e5c025f71e","Type":"ContainerStarted","Data":"6739db56d15d4f8ed9151c3d5096dcdce3e994c083f3dfda5bb1e6369f36094e"}
Sep 10 21:19:41.873173 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:41.873134    2584 generic.go:356] "Generic (PLEG): container finished" podID="dcbebff3-e15b-43e5-acfa-57e5c025f71e" containerID="64d208ffc46188badbb174e62609dbf6d3f7e78be38fd535409db3983fb01486" exitCode=0
Sep 10 21:19:41.873600 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:41.873178    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f376d02-r6hwr" event={"ID":"dcbebff3-e15b-43e5-acfa-57e5c025f71e","Type":"ContainerDied","Data":"64d208ffc46188badbb174e62609dbf6d3f7e78be38fd535409db3983fb01486"}
Sep 10 21:19:43.007693 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:43.007667    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f376d02-r6hwr"
Sep 10 21:19:43.090153 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:43.090124    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbebff3-e15b-43e5-acfa-57e5c025f71e-maas-api-service-ca\") pod \"dcbebff3-e15b-43e5-acfa-57e5c025f71e\" (UID: \"dcbebff3-e15b-43e5-acfa-57e5c025f71e\") "
Sep 10 21:19:43.090526 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:43.090500    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbebff3-e15b-43e5-acfa-57e5c025f71e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dcbebff3-e15b-43e5-acfa-57e5c025f71e" (UID: "dcbebff3-e15b-43e5-acfa-57e5c025f71e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:19:43.191385 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:43.191315    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbebff3-e15b-43e5-acfa-57e5c025f71e-maas-api-service-ca\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:19:43.884878 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:43.884847    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f376d02-r6hwr"
Sep 10 21:19:43.885055 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:43.884882    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f376d02-r6hwr" event={"ID":"dcbebff3-e15b-43e5-acfa-57e5c025f71e","Type":"ContainerDied","Data":"6739db56d15d4f8ed9151c3d5096dcdce3e994c083f3dfda5bb1e6369f36094e"}
Sep 10 21:19:43.885055 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:43.884929    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6739db56d15d4f8ed9151c3d5096dcdce3e994c083f3dfda5bb1e6369f36094e"
Sep 10 21:19:56.049219 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:56.049143    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f376d02-r6hwr"]
Sep 10 21:19:56.057757 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:56.057730    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f376d02-r6hwr"]
Sep 10 21:19:56.840724 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:19:56.840687    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcbebff3-e15b-43e5-acfa-57e5c025f71e" path="/var/lib/kubelet/pods/dcbebff3-e15b-43e5-acfa-57e5c025f71e/volumes"
Sep 10 21:20:00.924686 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:00.924655    2584 scope.go:117] "RemoveContainer" containerID="f830eaead0d3926b5a46d19257174e18a2b28a8510de8a231e295c47572b6cef"
Sep 10 21:20:00.932945 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:00.932930    2584 scope.go:117] "RemoveContainer" containerID="0aee2177c5a7fc19e43d43139a924999fbbf4798cd68da6484fed4a31274308c"
Sep 10 21:20:00.940750 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:00.940734    2584 scope.go:117] "RemoveContainer" containerID="ce55d03a794a334ef8569aa200dc63cc0eb6353cd263ac084bfa2efc758b2cac"
Sep 10 21:20:04.300043 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:04.300006    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"]
Sep 10 21:20:19.319515 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:19.319455    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h" podUID="5798357d-5d13-4a88-acae-215eedfa75df" containerName="main" containerID="cri-o://5d703a1e5c4dad98e70611c1f7bfc70a59594a2531447c08320c264a2b3660dc" gracePeriod=45
Sep 10 21:20:19.573171 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:19.573109    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:20:19.665557 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:19.665523    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5798357d-5d13-4a88-acae-215eedfa75df-home\") pod \"5798357d-5d13-4a88-acae-215eedfa75df\" (UID: \"5798357d-5d13-4a88-acae-215eedfa75df\") "
Sep 10 21:20:19.665743 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:19.665586    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzz9j\" (UniqueName: \"kubernetes.io/projected/5798357d-5d13-4a88-acae-215eedfa75df-kube-api-access-gzz9j\") pod \"5798357d-5d13-4a88-acae-215eedfa75df\" (UID: \"5798357d-5d13-4a88-acae-215eedfa75df\") "
Sep 10 21:20:19.665743 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:19.665713    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5798357d-5d13-4a88-acae-215eedfa75df-tls-certs\") pod \"5798357d-5d13-4a88-acae-215eedfa75df\" (UID: \"5798357d-5d13-4a88-acae-215eedfa75df\") "
Sep 10 21:20:19.665863 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:19.665741    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5798357d-5d13-4a88-acae-215eedfa75df-model-cache\") pod \"5798357d-5d13-4a88-acae-215eedfa75df\" (UID: \"5798357d-5d13-4a88-acae-215eedfa75df\") "
Sep 10 21:20:19.665863 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:19.665770    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5798357d-5d13-4a88-acae-215eedfa75df-dshm\") pod \"5798357d-5d13-4a88-acae-215eedfa75df\" (UID: \"5798357d-5d13-4a88-acae-215eedfa75df\") "
Sep 10 21:20:19.665863 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:19.665821    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5798357d-5d13-4a88-acae-215eedfa75df-home" (OuterVolumeSpecName: "home") pod "5798357d-5d13-4a88-acae-215eedfa75df" (UID: "5798357d-5d13-4a88-acae-215eedfa75df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:20:19.665863 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:19.665834    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5798357d-5d13-4a88-acae-215eedfa75df-tmp-dir\") pod \"5798357d-5d13-4a88-acae-215eedfa75df\" (UID: \"5798357d-5d13-4a88-acae-215eedfa75df\") "
Sep 10 21:20:19.666079 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:19.666060    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5798357d-5d13-4a88-acae-215eedfa75df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5798357d-5d13-4a88-acae-215eedfa75df" (UID: "5798357d-5d13-4a88-acae-215eedfa75df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:20:19.666137 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:19.666072    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5798357d-5d13-4a88-acae-215eedfa75df-model-cache" (OuterVolumeSpecName: "model-cache") pod "5798357d-5d13-4a88-acae-215eedfa75df" (UID: "5798357d-5d13-4a88-acae-215eedfa75df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:20:19.666382 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:19.666359    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5798357d-5d13-4a88-acae-215eedfa75df-home\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:20:19.666459 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:19.666387    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5798357d-5d13-4a88-acae-215eedfa75df-model-cache\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:20:19.666459 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:19.666402    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5798357d-5d13-4a88-acae-215eedfa75df-tmp-dir\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:20:19.667872 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:19.667850    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5798357d-5d13-4a88-acae-215eedfa75df-kube-api-access-gzz9j" (OuterVolumeSpecName: "kube-api-access-gzz9j") pod "5798357d-5d13-4a88-acae-215eedfa75df" (UID: "5798357d-5d13-4a88-acae-215eedfa75df"). InnerVolumeSpecName "kube-api-access-gzz9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:20:19.667925 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:19.667860    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5798357d-5d13-4a88-acae-215eedfa75df-dshm" (OuterVolumeSpecName: "dshm") pod "5798357d-5d13-4a88-acae-215eedfa75df" (UID: "5798357d-5d13-4a88-acae-215eedfa75df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:20:19.667925 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:19.667894    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5798357d-5d13-4a88-acae-215eedfa75df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5798357d-5d13-4a88-acae-215eedfa75df" (UID: "5798357d-5d13-4a88-acae-215eedfa75df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:20:19.767416 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:19.767380    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzz9j\" (UniqueName: \"kubernetes.io/projected/5798357d-5d13-4a88-acae-215eedfa75df-kube-api-access-gzz9j\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:20:19.767416 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:19.767412    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5798357d-5d13-4a88-acae-215eedfa75df-tls-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:20:19.767416 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:19.767451    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5798357d-5d13-4a88-acae-215eedfa75df-dshm\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:20:20.071058 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:20.071018    2584 generic.go:356] "Generic (PLEG): container finished" podID="5798357d-5d13-4a88-acae-215eedfa75df" containerID="5d703a1e5c4dad98e70611c1f7bfc70a59594a2531447c08320c264a2b3660dc" exitCode=0
Sep 10 21:20:20.071217 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:20.071067    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"
Sep 10 21:20:20.071217 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:20.071065    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h" event={"ID":"5798357d-5d13-4a88-acae-215eedfa75df","Type":"ContainerDied","Data":"5d703a1e5c4dad98e70611c1f7bfc70a59594a2531447c08320c264a2b3660dc"}
Sep 10 21:20:20.071217 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:20.071164    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h" event={"ID":"5798357d-5d13-4a88-acae-215eedfa75df","Type":"ContainerDied","Data":"4e6428bec18c83d24ef8e15d2cd0a6cbc43559c99990a575d24282662bad0771"}
Sep 10 21:20:20.071217 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:20.071186    2584 scope.go:117] "RemoveContainer" containerID="5d703a1e5c4dad98e70611c1f7bfc70a59594a2531447c08320c264a2b3660dc"
Sep 10 21:20:20.082735 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:20.082715    2584 scope.go:117] "RemoveContainer" containerID="5d703a1e5c4dad98e70611c1f7bfc70a59594a2531447c08320c264a2b3660dc"
Sep 10 21:20:20.083006 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:20:20.082991    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d703a1e5c4dad98e70611c1f7bfc70a59594a2531447c08320c264a2b3660dc\": container with ID starting with 5d703a1e5c4dad98e70611c1f7bfc70a59594a2531447c08320c264a2b3660dc not found: ID does not exist" containerID="5d703a1e5c4dad98e70611c1f7bfc70a59594a2531447c08320c264a2b3660dc"
Sep 10 21:20:20.083049 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:20.083015    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d703a1e5c4dad98e70611c1f7bfc70a59594a2531447c08320c264a2b3660dc"} err="failed to get container status \"5d703a1e5c4dad98e70611c1f7bfc70a59594a2531447c08320c264a2b3660dc\": rpc error: code = NotFound desc = could not find container \"5d703a1e5c4dad98e70611c1f7bfc70a59594a2531447c08320c264a2b3660dc\": container with ID starting with 5d703a1e5c4dad98e70611c1f7bfc70a59594a2531447c08320c264a2b3660dc not found: ID does not exist"
Sep 10 21:20:20.096824 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:20.096794    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"]
Sep 10 21:20:20.101673 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:20.101651    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-7f376d02/e2e-lifecycle-model-7f376d02-kserve-7b8bffc9f-28v6h"]
Sep 10 21:20:20.840201 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:20.840153    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5798357d-5d13-4a88-acae-215eedfa75df" path="/var/lib/kubelet/pods/5798357d-5d13-4a88-acae-215eedfa75df/volumes"
Sep 10 21:20:46.672518 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:46.672482    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8125-f7c7de"]
Sep 10 21:20:46.672895 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:46.672838    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcbebff3-e15b-43e5-acfa-57e5c025f71e" containerName="revoke-keys"
Sep 10 21:20:46.672895 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:46.672850    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcbebff3-e15b-43e5-acfa-57e5c025f71e" containerName="revoke-keys"
Sep 10 21:20:46.672895 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:46.672876    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5798357d-5d13-4a88-acae-215eedfa75df" containerName="main"
Sep 10 21:20:46.672895 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:46.672882    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5798357d-5d13-4a88-acae-215eedfa75df" containerName="main"
Sep 10 21:20:46.673041 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:46.672939    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5798357d-5d13-4a88-acae-215eedfa75df" containerName="main"
Sep 10 21:20:46.673041 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:46.672948    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcbebff3-e15b-43e5-acfa-57e5c025f71e" containerName="revoke-keys"
Sep 10 21:20:46.675589 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:46.675571    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8125-f7c7de"
Sep 10 21:20:46.677780 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:46.677761    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cc2nx\""
Sep 10 21:20:46.682334 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:46.682309    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8125-f7c7de"]
Sep 10 21:20:46.735655 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:46.735617    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq7ws\" (UniqueName: \"kubernetes.io/projected/4d197579-bb0f-4ad3-8e14-b29899111c3e-kube-api-access-sq7ws\") pod \"test-curl-8125-f7c7de\" (UID: \"4d197579-bb0f-4ad3-8e14-b29899111c3e\") " pod="opendatahub/test-curl-8125-f7c7de"
Sep 10 21:20:46.837204 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:46.837159    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq7ws\" (UniqueName: \"kubernetes.io/projected/4d197579-bb0f-4ad3-8e14-b29899111c3e-kube-api-access-sq7ws\") pod \"test-curl-8125-f7c7de\" (UID: \"4d197579-bb0f-4ad3-8e14-b29899111c3e\") " pod="opendatahub/test-curl-8125-f7c7de"
Sep 10 21:20:46.845108 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:46.845078    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq7ws\" (UniqueName: \"kubernetes.io/projected/4d197579-bb0f-4ad3-8e14-b29899111c3e-kube-api-access-sq7ws\") pod \"test-curl-8125-f7c7de\" (UID: \"4d197579-bb0f-4ad3-8e14-b29899111c3e\") " pod="opendatahub/test-curl-8125-f7c7de"
Sep 10 21:20:46.986026 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:46.985933    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8125-f7c7de"
Sep 10 21:20:47.256674 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:47.256582    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8125-f7c7de"]
Sep 10 21:20:47.259179 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:20:47.259157    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d197579_bb0f_4ad3_8e14_b29899111c3e.slice/crio-682a90b0548b6248ed8accc300f09548822ba5aca885776af50afb51651bdb5b WatchSource:0}: Error finding container 682a90b0548b6248ed8accc300f09548822ba5aca885776af50afb51651bdb5b: Status 404 returned error can't find the container with id 682a90b0548b6248ed8accc300f09548822ba5aca885776af50afb51651bdb5b
Sep 10 21:20:48.215455 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:48.215397    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"]
Sep 10 21:20:48.215864 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:48.215632    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr" podUID="0d17bff8-357c-450d-a7b4-6a133b7e0a31" containerName="istio-proxy" containerID="cri-o://6731cafcb1679328d083096f1357ba2579b2e46d00ef7d817ba2752eaef82b27" gracePeriod=30
Sep 10 21:20:48.224336 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:48.224304    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8125-f7c7de" event={"ID":"4d197579-bb0f-4ad3-8e14-b29899111c3e","Type":"ContainerStarted","Data":"902c9c8bfd4638423aa89565dc8a76a5d1ad332534108e761dcdad4fa3585ae3"}
Sep 10 21:20:48.224514 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:48.224348    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8125-f7c7de" event={"ID":"4d197579-bb0f-4ad3-8e14-b29899111c3e","Type":"ContainerStarted","Data":"682a90b0548b6248ed8accc300f09548822ba5aca885776af50afb51651bdb5b"}
Sep 10 21:20:48.255375 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:48.255308    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8125-f7c7de" podStartSLOduration=2.255293649 podStartE2EDuration="2.255293649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:48.253413114 +0000 UTC m=+1307.918437914" watchObservedRunningTime="2026-09-10 21:20:48.255293649 +0000 UTC m=+1307.920318448"
Sep 10 21:20:48.865099 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:48.865069    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8125-f7c7de"]
Sep 10 21:20:48.870500 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:48.870475    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8125-f7c7de"]
Sep 10 21:20:49.229081 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:20:49.229048    2584 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8125-f7c7de" container="test-curl-8125-f7c7de"
Sep 10 21:20:50.234250 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:50.234208    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8125-f7c7de" podUID="4d197579-bb0f-4ad3-8e14-b29899111c3e" containerName="test-curl-8125-f7c7de" containerID="cri-o://902c9c8bfd4638423aa89565dc8a76a5d1ad332534108e761dcdad4fa3585ae3" gracePeriod=2
Sep 10 21:20:50.236638 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:20:50.236610    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8125-f7c7de\" is forbidden: User \"system:node:ip-10-0-135-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-205.ec2.internal' and this object" podUID="4d197579-bb0f-4ad3-8e14-b29899111c3e" pod="opendatahub/test-curl-8125-f7c7de"
Sep 10 21:20:50.273879 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:50.273847    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"]
Sep 10 21:20:50.294986 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:20:50.294950    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8125-f7c7de\" is forbidden: User \"system:node:ip-10-0-135-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-205.ec2.internal' and this object" podUID="4d197579-bb0f-4ad3-8e14-b29899111c3e" pod="opendatahub/test-curl-8125-f7c7de"
Sep 10 21:20:50.838571 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:20:50.838532    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8125-f7c7de\" is forbidden: User \"system:node:ip-10-0-135-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-205.ec2.internal' and this object" podUID="4d197579-bb0f-4ad3-8e14-b29899111c3e" pod="opendatahub/test-curl-8125-f7c7de"
Sep 10 21:20:51.350835 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:51.350803    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"]
Sep 10 21:20:52.401777 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:52.401753    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8125-f7c7de"
Sep 10 21:20:52.495709 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:52.495614    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sq7ws\" (UniqueName: \"kubernetes.io/projected/4d197579-bb0f-4ad3-8e14-b29899111c3e-kube-api-access-sq7ws\") pod \"4d197579-bb0f-4ad3-8e14-b29899111c3e\" (UID: \"4d197579-bb0f-4ad3-8e14-b29899111c3e\") "
Sep 10 21:20:52.497754 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:52.497726    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d197579-bb0f-4ad3-8e14-b29899111c3e-kube-api-access-sq7ws" (OuterVolumeSpecName: "kube-api-access-sq7ws") pod "4d197579-bb0f-4ad3-8e14-b29899111c3e" (UID: "4d197579-bb0f-4ad3-8e14-b29899111c3e"). InnerVolumeSpecName "kube-api-access-sq7ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:20:52.596720 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:52.596687    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq7ws\" (UniqueName: \"kubernetes.io/projected/4d197579-bb0f-4ad3-8e14-b29899111c3e-kube-api-access-sq7ws\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:20:52.841702 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:52.841619    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d197579-bb0f-4ad3-8e14-b29899111c3e" path="/var/lib/kubelet/pods/4d197579-bb0f-4ad3-8e14-b29899111c3e/volumes"
Sep 10 21:20:53.251631 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.251598    2584 generic.go:356] "Generic (PLEG): container finished" podID="4d197579-bb0f-4ad3-8e14-b29899111c3e" containerID="902c9c8bfd4638423aa89565dc8a76a5d1ad332534108e761dcdad4fa3585ae3" exitCode=137
Sep 10 21:20:53.251752 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.251635    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8125-f7c7de"
Sep 10 21:20:53.251752 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.251672    2584 scope.go:117] "RemoveContainer" containerID="902c9c8bfd4638423aa89565dc8a76a5d1ad332534108e761dcdad4fa3585ae3"
Sep 10 21:20:53.327086 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.327059    2584 scope.go:117] "RemoveContainer" containerID="902c9c8bfd4638423aa89565dc8a76a5d1ad332534108e761dcdad4fa3585ae3"
Sep 10 21:20:53.327461 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:20:53.327440    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"902c9c8bfd4638423aa89565dc8a76a5d1ad332534108e761dcdad4fa3585ae3\": container with ID starting with 902c9c8bfd4638423aa89565dc8a76a5d1ad332534108e761dcdad4fa3585ae3 not found: ID does not exist" containerID="902c9c8bfd4638423aa89565dc8a76a5d1ad332534108e761dcdad4fa3585ae3"
Sep 10 21:20:53.327537 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.327472    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902c9c8bfd4638423aa89565dc8a76a5d1ad332534108e761dcdad4fa3585ae3"} err="failed to get container status \"902c9c8bfd4638423aa89565dc8a76a5d1ad332534108e761dcdad4fa3585ae3\": rpc error: code = NotFound desc = could not find container \"902c9c8bfd4638423aa89565dc8a76a5d1ad332534108e761dcdad4fa3585ae3\": container with ID starting with 902c9c8bfd4638423aa89565dc8a76a5d1ad332534108e761dcdad4fa3585ae3 not found: ID does not exist"
Sep 10 21:20:53.467843 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.467813    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:20:53.505282 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.505218    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istio-data\") pod \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") "
Sep 10 21:20:53.505282 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.505259    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istio-envoy\") pod \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") "
Sep 10 21:20:53.505509 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.505329    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-workload-certs\") pod \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") "
Sep 10 21:20:53.505509 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.505367    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istiod-ca-cert\") pod \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") "
Sep 10 21:20:53.505509 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.505402    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istio-podinfo\") pod \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") "
Sep 10 21:20:53.505509 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.505434    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-credential-socket\") pod \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") "
Sep 10 21:20:53.505509 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.505462    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istio-token\") pod \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") "
Sep 10 21:20:53.505509 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.505502    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-workload-socket\") pod \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") "
Sep 10 21:20:53.505821 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.505540    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istio-data" (OuterVolumeSpecName: "istio-data") pod "0d17bff8-357c-450d-a7b4-6a133b7e0a31" (UID: "0d17bff8-357c-450d-a7b4-6a133b7e0a31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:20:53.505821 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.505592    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mb7l\" (UniqueName: \"kubernetes.io/projected/0d17bff8-357c-450d-a7b4-6a133b7e0a31-kube-api-access-9mb7l\") pod \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\" (UID: \"0d17bff8-357c-450d-a7b4-6a133b7e0a31\") "
Sep 10 21:20:53.505821 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.505725    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0d17bff8-357c-450d-a7b4-6a133b7e0a31" (UID: "0d17bff8-357c-450d-a7b4-6a133b7e0a31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:20:53.505974 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.505830    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0d17bff8-357c-450d-a7b4-6a133b7e0a31" (UID: "0d17bff8-357c-450d-a7b4-6a133b7e0a31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:20:53.506027 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.506012    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istio-data\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:20:53.506082 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.506030    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-workload-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:20:53.506082 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.506045    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-workload-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:20:53.506342 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.506320    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0d17bff8-357c-450d-a7b4-6a133b7e0a31" (UID: "0d17bff8-357c-450d-a7b4-6a133b7e0a31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:20:53.506547 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.506520    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0d17bff8-357c-450d-a7b4-6a133b7e0a31" (UID: "0d17bff8-357c-450d-a7b4-6a133b7e0a31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:20:53.507676 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.507652    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istio-token" (OuterVolumeSpecName: "istio-token") pod "0d17bff8-357c-450d-a7b4-6a133b7e0a31" (UID: "0d17bff8-357c-450d-a7b4-6a133b7e0a31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:20:53.507804 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.507787    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d17bff8-357c-450d-a7b4-6a133b7e0a31-kube-api-access-9mb7l" (OuterVolumeSpecName: "kube-api-access-9mb7l") pod "0d17bff8-357c-450d-a7b4-6a133b7e0a31" (UID: "0d17bff8-357c-450d-a7b4-6a133b7e0a31"). InnerVolumeSpecName "kube-api-access-9mb7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:20:53.507950 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.507935    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0d17bff8-357c-450d-a7b4-6a133b7e0a31" (UID: "0d17bff8-357c-450d-a7b4-6a133b7e0a31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:20:53.508156 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.508131    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0d17bff8-357c-450d-a7b4-6a133b7e0a31" (UID: "0d17bff8-357c-450d-a7b4-6a133b7e0a31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:20:53.607068 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.607036    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mb7l\" (UniqueName: \"kubernetes.io/projected/0d17bff8-357c-450d-a7b4-6a133b7e0a31-kube-api-access-9mb7l\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:20:53.607068 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.607065    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istio-envoy\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:20:53.607068 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.607075    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istiod-ca-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:20:53.607300 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.607083    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istio-podinfo\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:20:53.607300 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.607091    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d17bff8-357c-450d-a7b4-6a133b7e0a31-credential-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:20:53.607300 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:53.607099    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d17bff8-357c-450d-a7b4-6a133b7e0a31-istio-token\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:20:54.260409 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:54.260370    2584 generic.go:356] "Generic (PLEG): container finished" podID="0d17bff8-357c-450d-a7b4-6a133b7e0a31" containerID="6731cafcb1679328d083096f1357ba2579b2e46d00ef7d817ba2752eaef82b27" exitCode=0
Sep 10 21:20:54.260609 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:54.260457    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"
Sep 10 21:20:54.260609 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:54.260475    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr" event={"ID":"0d17bff8-357c-450d-a7b4-6a133b7e0a31","Type":"ContainerDied","Data":"6731cafcb1679328d083096f1357ba2579b2e46d00ef7d817ba2752eaef82b27"}
Sep 10 21:20:54.260609 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:54.260556    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr" event={"ID":"0d17bff8-357c-450d-a7b4-6a133b7e0a31","Type":"ContainerDied","Data":"bf42c49abfc60dbaff28be29f5b41a62ddb9a737fe6d517d57215461385f46ae"}
Sep 10 21:20:54.260609 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:54.260580    2584 scope.go:117] "RemoveContainer" containerID="6731cafcb1679328d083096f1357ba2579b2e46d00ef7d817ba2752eaef82b27"
Sep 10 21:20:54.270758 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:54.270738    2584 scope.go:117] "RemoveContainer" containerID="41fcb2825db249f1059bdeac96bcc080660d5d8455b49153062055907b3d5f59"
Sep 10 21:20:54.278781 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:54.278761    2584 scope.go:117] "RemoveContainer" containerID="6731cafcb1679328d083096f1357ba2579b2e46d00ef7d817ba2752eaef82b27"
Sep 10 21:20:54.279068 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:20:54.279048    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6731cafcb1679328d083096f1357ba2579b2e46d00ef7d817ba2752eaef82b27\": container with ID starting with 6731cafcb1679328d083096f1357ba2579b2e46d00ef7d817ba2752eaef82b27 not found: ID does not exist" containerID="6731cafcb1679328d083096f1357ba2579b2e46d00ef7d817ba2752eaef82b27"
Sep 10 21:20:54.279115 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:54.279078    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6731cafcb1679328d083096f1357ba2579b2e46d00ef7d817ba2752eaef82b27"} err="failed to get container status \"6731cafcb1679328d083096f1357ba2579b2e46d00ef7d817ba2752eaef82b27\": rpc error: code = NotFound desc = could not find container \"6731cafcb1679328d083096f1357ba2579b2e46d00ef7d817ba2752eaef82b27\": container with ID starting with 6731cafcb1679328d083096f1357ba2579b2e46d00ef7d817ba2752eaef82b27 not found: ID does not exist"
Sep 10 21:20:54.279115 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:54.279097    2584 scope.go:117] "RemoveContainer" containerID="41fcb2825db249f1059bdeac96bcc080660d5d8455b49153062055907b3d5f59"
Sep 10 21:20:54.279358 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:20:54.279341    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41fcb2825db249f1059bdeac96bcc080660d5d8455b49153062055907b3d5f59\": container with ID starting with 41fcb2825db249f1059bdeac96bcc080660d5d8455b49153062055907b3d5f59 not found: ID does not exist" containerID="41fcb2825db249f1059bdeac96bcc080660d5d8455b49153062055907b3d5f59"
Sep 10 21:20:54.279405 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:54.279365    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41fcb2825db249f1059bdeac96bcc080660d5d8455b49153062055907b3d5f59"} err="failed to get container status \"41fcb2825db249f1059bdeac96bcc080660d5d8455b49153062055907b3d5f59\": rpc error: code = NotFound desc = could not find container \"41fcb2825db249f1059bdeac96bcc080660d5d8455b49153062055907b3d5f59\": container with ID starting with 41fcb2825db249f1059bdeac96bcc080660d5d8455b49153062055907b3d5f59 not found: ID does not exist"
Sep 10 21:20:54.288026 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:54.287995    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"]
Sep 10 21:20:54.291213 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:54.291190    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7f376d02-openshift-default-bf5f57564-l52cr"]
Sep 10 21:20:54.840725 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:54.840692    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d17bff8-357c-450d-a7b4-6a133b7e0a31" path="/var/lib/kubelet/pods/0d17bff8-357c-450d-a7b4-6a133b7e0a31/volumes"
Sep 10 21:20:56.487636 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.487605    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"]
Sep 10 21:20:56.488056 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.487979    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d197579-bb0f-4ad3-8e14-b29899111c3e" containerName="test-curl-8125-f7c7de"
Sep 10 21:20:56.488056 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.487992    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d197579-bb0f-4ad3-8e14-b29899111c3e" containerName="test-curl-8125-f7c7de"
Sep 10 21:20:56.488056 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.488007    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d17bff8-357c-450d-a7b4-6a133b7e0a31" containerName="istio-proxy"
Sep 10 21:20:56.488056 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.488013    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d17bff8-357c-450d-a7b4-6a133b7e0a31" containerName="istio-proxy"
Sep 10 21:20:56.488056 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.488029    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d17bff8-357c-450d-a7b4-6a133b7e0a31" containerName="istio-proxy"
Sep 10 21:20:56.488056 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.488035    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d17bff8-357c-450d-a7b4-6a133b7e0a31" containerName="istio-proxy"
Sep 10 21:20:56.488374 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.488101    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d197579-bb0f-4ad3-8e14-b29899111c3e" containerName="test-curl-8125-f7c7de"
Sep 10 21:20:56.488374 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.488111    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d17bff8-357c-450d-a7b4-6a133b7e0a31" containerName="istio-proxy"
Sep 10 21:20:56.488374 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.488116    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d17bff8-357c-450d-a7b4-6a133b7e0a31" containerName="istio-proxy"
Sep 10 21:20:56.492272 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.492245    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:20:56.494736 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.494712    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0ac302\"/\"sub-test-model-0ac302-kserve-self-signed-certs\""
Sep 10 21:20:56.503206 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.503187    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"]
Sep 10 21:20:56.533103 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.533076    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv54p\" (UniqueName: \"kubernetes.io/projected/8e94e56c-d489-43ba-b44c-c656a25b5225-kube-api-access-jv54p\") pod \"sub-test-model-0ac302-kserve-58f59b7d5c-w94dn\" (UID: \"8e94e56c-d489-43ba-b44c-c656a25b5225\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:20:56.533223 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.533136    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e94e56c-d489-43ba-b44c-c656a25b5225-model-cache\") pod \"sub-test-model-0ac302-kserve-58f59b7d5c-w94dn\" (UID: \"8e94e56c-d489-43ba-b44c-c656a25b5225\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:20:56.533223 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.533171    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e94e56c-d489-43ba-b44c-c656a25b5225-home\") pod \"sub-test-model-0ac302-kserve-58f59b7d5c-w94dn\" (UID: \"8e94e56c-d489-43ba-b44c-c656a25b5225\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:20:56.533223 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.533200    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e94e56c-d489-43ba-b44c-c656a25b5225-dshm\") pod \"sub-test-model-0ac302-kserve-58f59b7d5c-w94dn\" (UID: \"8e94e56c-d489-43ba-b44c-c656a25b5225\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:20:56.533352 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.533236    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e94e56c-d489-43ba-b44c-c656a25b5225-tmp-dir\") pod \"sub-test-model-0ac302-kserve-58f59b7d5c-w94dn\" (UID: \"8e94e56c-d489-43ba-b44c-c656a25b5225\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:20:56.533352 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.533281    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e94e56c-d489-43ba-b44c-c656a25b5225-tls-certs\") pod \"sub-test-model-0ac302-kserve-58f59b7d5c-w94dn\" (UID: \"8e94e56c-d489-43ba-b44c-c656a25b5225\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:20:56.634719 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.634658    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e94e56c-d489-43ba-b44c-c656a25b5225-tmp-dir\") pod \"sub-test-model-0ac302-kserve-58f59b7d5c-w94dn\" (UID: \"8e94e56c-d489-43ba-b44c-c656a25b5225\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:20:56.634901 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.634753    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e94e56c-d489-43ba-b44c-c656a25b5225-tls-certs\") pod \"sub-test-model-0ac302-kserve-58f59b7d5c-w94dn\" (UID: \"8e94e56c-d489-43ba-b44c-c656a25b5225\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:20:56.634901 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.634861    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv54p\" (UniqueName: \"kubernetes.io/projected/8e94e56c-d489-43ba-b44c-c656a25b5225-kube-api-access-jv54p\") pod \"sub-test-model-0ac302-kserve-58f59b7d5c-w94dn\" (UID: \"8e94e56c-d489-43ba-b44c-c656a25b5225\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:20:56.635044 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.634928    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e94e56c-d489-43ba-b44c-c656a25b5225-model-cache\") pod \"sub-test-model-0ac302-kserve-58f59b7d5c-w94dn\" (UID: \"8e94e56c-d489-43ba-b44c-c656a25b5225\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:20:56.635044 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.634960    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e94e56c-d489-43ba-b44c-c656a25b5225-home\") pod \"sub-test-model-0ac302-kserve-58f59b7d5c-w94dn\" (UID: \"8e94e56c-d489-43ba-b44c-c656a25b5225\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:20:56.635044 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.635007    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e94e56c-d489-43ba-b44c-c656a25b5225-dshm\") pod \"sub-test-model-0ac302-kserve-58f59b7d5c-w94dn\" (UID: \"8e94e56c-d489-43ba-b44c-c656a25b5225\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:20:56.635288 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.635252    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e94e56c-d489-43ba-b44c-c656a25b5225-tmp-dir\") pod \"sub-test-model-0ac302-kserve-58f59b7d5c-w94dn\" (UID: \"8e94e56c-d489-43ba-b44c-c656a25b5225\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:20:56.635571 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.635542    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e94e56c-d489-43ba-b44c-c656a25b5225-model-cache\") pod \"sub-test-model-0ac302-kserve-58f59b7d5c-w94dn\" (UID: \"8e94e56c-d489-43ba-b44c-c656a25b5225\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:20:56.635664 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.635633    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e94e56c-d489-43ba-b44c-c656a25b5225-home\") pod \"sub-test-model-0ac302-kserve-58f59b7d5c-w94dn\" (UID: \"8e94e56c-d489-43ba-b44c-c656a25b5225\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:20:56.638317 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.638291    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e94e56c-d489-43ba-b44c-c656a25b5225-dshm\") pod \"sub-test-model-0ac302-kserve-58f59b7d5c-w94dn\" (UID: \"8e94e56c-d489-43ba-b44c-c656a25b5225\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:20:56.638850 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.638827    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e94e56c-d489-43ba-b44c-c656a25b5225-tls-certs\") pod \"sub-test-model-0ac302-kserve-58f59b7d5c-w94dn\" (UID: \"8e94e56c-d489-43ba-b44c-c656a25b5225\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:20:56.643093 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.643066    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv54p\" (UniqueName: \"kubernetes.io/projected/8e94e56c-d489-43ba-b44c-c656a25b5225-kube-api-access-jv54p\") pod \"sub-test-model-0ac302-kserve-58f59b7d5c-w94dn\" (UID: \"8e94e56c-d489-43ba-b44c-c656a25b5225\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:20:56.803546 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:56.803456    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:20:57.084617 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:57.084518    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"]
Sep 10 21:20:57.090582 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:20:57.090537    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e94e56c_d489_43ba_b44c_c656a25b5225.slice/crio-bb862cf820638b7715210b96f8617617f8eea7fe6742043b4222662cba273547 WatchSource:0}: Error finding container bb862cf820638b7715210b96f8617617f8eea7fe6742043b4222662cba273547: Status 404 returned error can't find the container with id bb862cf820638b7715210b96f8617617f8eea7fe6742043b4222662cba273547
Sep 10 21:20:57.281000 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:57.280961    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn" event={"ID":"8e94e56c-d489-43ba-b44c-c656a25b5225","Type":"ContainerStarted","Data":"3e2677f7f60a96955c200563b971a409728e5f8909310e9d62b1a3446d64c714"}
Sep 10 21:20:57.281167 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:57.281016    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn" event={"ID":"8e94e56c-d489-43ba-b44c-c656a25b5225","Type":"ContainerStarted","Data":"bb862cf820638b7715210b96f8617617f8eea7fe6742043b4222662cba273547"}
Sep 10 21:20:57.322831 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:20:57.322779    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn" podStartSLOduration=1.32276204 podStartE2EDuration="1.32276204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:57.320710085 +0000 UTC m=+1316.985734886" watchObservedRunningTime="2026-09-10 21:20:57.32276204 +0000 UTC m=+1316.987786839"
Sep 10 21:21:00.966962 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:00.966933    2584 scope.go:117] "RemoveContainer" containerID="cf15f95e025adebd9cfebc3128c829fba9446aa812d0f6559fcbab656de3657c"
Sep 10 21:21:00.975335 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:00.975314    2584 scope.go:117] "RemoveContainer" containerID="65b613f27e3a7e117d04678e5ad1f69c29b253e9a78b840545b6576c2393394b"
Sep 10 21:21:00.982680 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:00.982657    2584 scope.go:117] "RemoveContainer" containerID="98ae987ca1341601e5b84925cbecfcc638e248d8416b6fb7dcb3b57e90738eae"
Sep 10 21:21:05.292860 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:05.292813    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf" podUID="5f3d505e-09b3-4167-a9a8-d12f03b59077" containerName="main" containerID="cri-o://4a17809c66d1e2a87617fd84f4c2111b8744d9c2bb30934304670ed95d8ca9fe" gracePeriod=45
Sep 10 21:21:05.553295 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:05.553237    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:21:05.625996 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:05.625959    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f3d505e-09b3-4167-a9a8-d12f03b59077-model-cache\") pod \"5f3d505e-09b3-4167-a9a8-d12f03b59077\" (UID: \"5f3d505e-09b3-4167-a9a8-d12f03b59077\") "
Sep 10 21:21:05.626157 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:05.626060    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f3d505e-09b3-4167-a9a8-d12f03b59077-tls-certs\") pod \"5f3d505e-09b3-4167-a9a8-d12f03b59077\" (UID: \"5f3d505e-09b3-4167-a9a8-d12f03b59077\") "
Sep 10 21:21:05.626157 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:05.626151    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f3d505e-09b3-4167-a9a8-d12f03b59077-home\") pod \"5f3d505e-09b3-4167-a9a8-d12f03b59077\" (UID: \"5f3d505e-09b3-4167-a9a8-d12f03b59077\") "
Sep 10 21:21:05.626276 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:05.626190    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f3d505e-09b3-4167-a9a8-d12f03b59077-tmp-dir\") pod \"5f3d505e-09b3-4167-a9a8-d12f03b59077\" (UID: \"5f3d505e-09b3-4167-a9a8-d12f03b59077\") "
Sep 10 21:21:05.626276 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:05.626220    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f3d505e-09b3-4167-a9a8-d12f03b59077-dshm\") pod \"5f3d505e-09b3-4167-a9a8-d12f03b59077\" (UID: \"5f3d505e-09b3-4167-a9a8-d12f03b59077\") "
Sep 10 21:21:05.626276 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:05.626225    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3d505e-09b3-4167-a9a8-d12f03b59077-model-cache" (OuterVolumeSpecName: "model-cache") pod "5f3d505e-09b3-4167-a9a8-d12f03b59077" (UID: "5f3d505e-09b3-4167-a9a8-d12f03b59077"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:05.626276 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:05.626256    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmt7s\" (UniqueName: \"kubernetes.io/projected/5f3d505e-09b3-4167-a9a8-d12f03b59077-kube-api-access-mmt7s\") pod \"5f3d505e-09b3-4167-a9a8-d12f03b59077\" (UID: \"5f3d505e-09b3-4167-a9a8-d12f03b59077\") "
Sep 10 21:21:05.626553 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:05.626395    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3d505e-09b3-4167-a9a8-d12f03b59077-home" (OuterVolumeSpecName: "home") pod "5f3d505e-09b3-4167-a9a8-d12f03b59077" (UID: "5f3d505e-09b3-4167-a9a8-d12f03b59077"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:05.626553 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:05.626538    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3d505e-09b3-4167-a9a8-d12f03b59077-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5f3d505e-09b3-4167-a9a8-d12f03b59077" (UID: "5f3d505e-09b3-4167-a9a8-d12f03b59077"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:05.626744 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:05.626707    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f3d505e-09b3-4167-a9a8-d12f03b59077-tmp-dir\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:21:05.626744 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:05.626746    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f3d505e-09b3-4167-a9a8-d12f03b59077-model-cache\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:21:05.626890 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:05.626759    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f3d505e-09b3-4167-a9a8-d12f03b59077-home\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:21:05.628508 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:05.628487    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3d505e-09b3-4167-a9a8-d12f03b59077-dshm" (OuterVolumeSpecName: "dshm") pod "5f3d505e-09b3-4167-a9a8-d12f03b59077" (UID: "5f3d505e-09b3-4167-a9a8-d12f03b59077"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:05.628572 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:05.628552    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3d505e-09b3-4167-a9a8-d12f03b59077-kube-api-access-mmt7s" (OuterVolumeSpecName: "kube-api-access-mmt7s") pod "5f3d505e-09b3-4167-a9a8-d12f03b59077" (UID: "5f3d505e-09b3-4167-a9a8-d12f03b59077"). InnerVolumeSpecName "kube-api-access-mmt7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:05.628624 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:05.628554    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f3d505e-09b3-4167-a9a8-d12f03b59077-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f3d505e-09b3-4167-a9a8-d12f03b59077" (UID: "5f3d505e-09b3-4167-a9a8-d12f03b59077"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:21:05.728144 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:05.728105    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f3d505e-09b3-4167-a9a8-d12f03b59077-tls-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:21:05.728144 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:05.728135    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f3d505e-09b3-4167-a9a8-d12f03b59077-dshm\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:21:05.728144 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:05.728147    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmt7s\" (UniqueName: \"kubernetes.io/projected/5f3d505e-09b3-4167-a9a8-d12f03b59077-kube-api-access-mmt7s\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:21:06.329299 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.329257    2584 generic.go:356] "Generic (PLEG): container finished" podID="5f3d505e-09b3-4167-a9a8-d12f03b59077" containerID="4a17809c66d1e2a87617fd84f4c2111b8744d9c2bb30934304670ed95d8ca9fe" exitCode=0
Sep 10 21:21:06.329699 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.329310    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"
Sep 10 21:21:06.329699 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.329341    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf" event={"ID":"5f3d505e-09b3-4167-a9a8-d12f03b59077","Type":"ContainerDied","Data":"4a17809c66d1e2a87617fd84f4c2111b8744d9c2bb30934304670ed95d8ca9fe"}
Sep 10 21:21:06.329699 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.329379    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf" event={"ID":"5f3d505e-09b3-4167-a9a8-d12f03b59077","Type":"ContainerDied","Data":"ae58ec67ab9f10799f972827c06ce951fc8d1e4e2416e0173b47b2fca0687fed"}
Sep 10 21:21:06.329699 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.329401    2584 scope.go:117] "RemoveContainer" containerID="4a17809c66d1e2a87617fd84f4c2111b8744d9c2bb30934304670ed95d8ca9fe"
Sep 10 21:21:06.338747 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.338728    2584 scope.go:117] "RemoveContainer" containerID="4a17809c66d1e2a87617fd84f4c2111b8744d9c2bb30934304670ed95d8ca9fe"
Sep 10 21:21:06.339038 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:21:06.339018    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a17809c66d1e2a87617fd84f4c2111b8744d9c2bb30934304670ed95d8ca9fe\": container with ID starting with 4a17809c66d1e2a87617fd84f4c2111b8744d9c2bb30934304670ed95d8ca9fe not found: ID does not exist" containerID="4a17809c66d1e2a87617fd84f4c2111b8744d9c2bb30934304670ed95d8ca9fe"
Sep 10 21:21:06.339087 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.339047    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a17809c66d1e2a87617fd84f4c2111b8744d9c2bb30934304670ed95d8ca9fe"} err="failed to get container status \"4a17809c66d1e2a87617fd84f4c2111b8744d9c2bb30934304670ed95d8ca9fe\": rpc error: code = NotFound desc = could not find container \"4a17809c66d1e2a87617fd84f4c2111b8744d9c2bb30934304670ed95d8ca9fe\": container with ID starting with 4a17809c66d1e2a87617fd84f4c2111b8744d9c2bb30934304670ed95d8ca9fe not found: ID does not exist"
Sep 10 21:21:06.351640 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.351606    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"]
Sep 10 21:21:06.355527 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.355505    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0ac302/auth-test-model-0ac302-kserve-648f5db887-pp4nf"]
Sep 10 21:21:06.370252 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.370217    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh" podUID="ca940185-2e99-4f51-8ca7-8aa2b534d893" containerName="main" containerID="cri-o://16c58e627f22b240236e6963d8362090d60543c6125a0f61f128793bc72d310e" gracePeriod=45
Sep 10 21:21:06.625178 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.625154    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:21:06.736768 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.736741    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca940185-2e99-4f51-8ca7-8aa2b534d893-tls-certs\") pod \"ca940185-2e99-4f51-8ca7-8aa2b534d893\" (UID: \"ca940185-2e99-4f51-8ca7-8aa2b534d893\") "
Sep 10 21:21:06.736974 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.736874    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca940185-2e99-4f51-8ca7-8aa2b534d893-model-cache\") pod \"ca940185-2e99-4f51-8ca7-8aa2b534d893\" (UID: \"ca940185-2e99-4f51-8ca7-8aa2b534d893\") "
Sep 10 21:21:06.736974 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.736929    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4xzr\" (UniqueName: \"kubernetes.io/projected/ca940185-2e99-4f51-8ca7-8aa2b534d893-kube-api-access-x4xzr\") pod \"ca940185-2e99-4f51-8ca7-8aa2b534d893\" (UID: \"ca940185-2e99-4f51-8ca7-8aa2b534d893\") "
Sep 10 21:21:06.736974 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.736954    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca940185-2e99-4f51-8ca7-8aa2b534d893-home\") pod \"ca940185-2e99-4f51-8ca7-8aa2b534d893\" (UID: \"ca940185-2e99-4f51-8ca7-8aa2b534d893\") "
Sep 10 21:21:06.737152 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.736988    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca940185-2e99-4f51-8ca7-8aa2b534d893-dshm\") pod \"ca940185-2e99-4f51-8ca7-8aa2b534d893\" (UID: \"ca940185-2e99-4f51-8ca7-8aa2b534d893\") "
Sep 10 21:21:06.737152 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.737016    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca940185-2e99-4f51-8ca7-8aa2b534d893-tmp-dir\") pod \"ca940185-2e99-4f51-8ca7-8aa2b534d893\" (UID: \"ca940185-2e99-4f51-8ca7-8aa2b534d893\") "
Sep 10 21:21:06.737263 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.737143    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca940185-2e99-4f51-8ca7-8aa2b534d893-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca940185-2e99-4f51-8ca7-8aa2b534d893" (UID: "ca940185-2e99-4f51-8ca7-8aa2b534d893"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:06.737317 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.737241    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca940185-2e99-4f51-8ca7-8aa2b534d893-home" (OuterVolumeSpecName: "home") pod "ca940185-2e99-4f51-8ca7-8aa2b534d893" (UID: "ca940185-2e99-4f51-8ca7-8aa2b534d893"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:06.737369 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.737313    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca940185-2e99-4f51-8ca7-8aa2b534d893-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca940185-2e99-4f51-8ca7-8aa2b534d893" (UID: "ca940185-2e99-4f51-8ca7-8aa2b534d893"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:06.737484 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.737459    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca940185-2e99-4f51-8ca7-8aa2b534d893-tmp-dir\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:21:06.737484 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.737484    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca940185-2e99-4f51-8ca7-8aa2b534d893-model-cache\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:21:06.737677 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.737493    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca940185-2e99-4f51-8ca7-8aa2b534d893-home\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:21:06.739128 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.739104    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca940185-2e99-4f51-8ca7-8aa2b534d893-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca940185-2e99-4f51-8ca7-8aa2b534d893" (UID: "ca940185-2e99-4f51-8ca7-8aa2b534d893"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:21:06.739232 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.739167    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca940185-2e99-4f51-8ca7-8aa2b534d893-dshm" (OuterVolumeSpecName: "dshm") pod "ca940185-2e99-4f51-8ca7-8aa2b534d893" (UID: "ca940185-2e99-4f51-8ca7-8aa2b534d893"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:06.739232 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.739176    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca940185-2e99-4f51-8ca7-8aa2b534d893-kube-api-access-x4xzr" (OuterVolumeSpecName: "kube-api-access-x4xzr") pod "ca940185-2e99-4f51-8ca7-8aa2b534d893" (UID: "ca940185-2e99-4f51-8ca7-8aa2b534d893"). InnerVolumeSpecName "kube-api-access-x4xzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:06.803702 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.803676    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:21:06.803826 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.803723    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:21:06.820362 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.820330    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:21:06.838015 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.837987    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4xzr\" (UniqueName: \"kubernetes.io/projected/ca940185-2e99-4f51-8ca7-8aa2b534d893-kube-api-access-x4xzr\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:21:06.838015 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.838012    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca940185-2e99-4f51-8ca7-8aa2b534d893-dshm\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:21:06.838015 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.838022    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca940185-2e99-4f51-8ca7-8aa2b534d893-tls-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:21:06.841340 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:06.841308    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f3d505e-09b3-4167-a9a8-d12f03b59077" path="/var/lib/kubelet/pods/5f3d505e-09b3-4167-a9a8-d12f03b59077/volumes"
Sep 10 21:21:07.335627 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:07.335591    2584 generic.go:356] "Generic (PLEG): container finished" podID="ca940185-2e99-4f51-8ca7-8aa2b534d893" containerID="16c58e627f22b240236e6963d8362090d60543c6125a0f61f128793bc72d310e" exitCode=0
Sep 10 21:21:07.336136 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:07.335647    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"
Sep 10 21:21:07.336136 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:07.335690    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh" event={"ID":"ca940185-2e99-4f51-8ca7-8aa2b534d893","Type":"ContainerDied","Data":"16c58e627f22b240236e6963d8362090d60543c6125a0f61f128793bc72d310e"}
Sep 10 21:21:07.336136 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:07.335750    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh" event={"ID":"ca940185-2e99-4f51-8ca7-8aa2b534d893","Type":"ContainerDied","Data":"16f9233b181cb8c497e620d79b012be2448048027fa596b264e8cf5f22838578"}
Sep 10 21:21:07.336136 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:07.335780    2584 scope.go:117] "RemoveContainer" containerID="16c58e627f22b240236e6963d8362090d60543c6125a0f61f128793bc72d310e"
Sep 10 21:21:07.344468 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:07.344446    2584 scope.go:117] "RemoveContainer" containerID="16c58e627f22b240236e6963d8362090d60543c6125a0f61f128793bc72d310e"
Sep 10 21:21:07.344765 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:21:07.344742    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16c58e627f22b240236e6963d8362090d60543c6125a0f61f128793bc72d310e\": container with ID starting with 16c58e627f22b240236e6963d8362090d60543c6125a0f61f128793bc72d310e not found: ID does not exist" containerID="16c58e627f22b240236e6963d8362090d60543c6125a0f61f128793bc72d310e"
Sep 10 21:21:07.344812 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:07.344778    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16c58e627f22b240236e6963d8362090d60543c6125a0f61f128793bc72d310e"} err="failed to get container status \"16c58e627f22b240236e6963d8362090d60543c6125a0f61f128793bc72d310e\": rpc error: code = NotFound desc = could not find container \"16c58e627f22b240236e6963d8362090d60543c6125a0f61f128793bc72d310e\": container with ID starting with 16c58e627f22b240236e6963d8362090d60543c6125a0f61f128793bc72d310e not found: ID does not exist"
Sep 10 21:21:07.348869 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:07.348854    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:21:07.364391 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:07.364366    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"]
Sep 10 21:21:07.370123 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:07.370101    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-16bbec/auth-test-model-16bbec-kserve-557dc88ccf-jn7nh"]
Sep 10 21:21:08.840789 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:08.840757    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca940185-2e99-4f51-8ca7-8aa2b534d893" path="/var/lib/kubelet/pods/ca940185-2e99-4f51-8ca7-8aa2b534d893/volumes"
Sep 10 21:21:34.573701 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.573625    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"]
Sep 10 21:21:34.574131 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.573981    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3d505e-09b3-4167-a9a8-d12f03b59077" containerName="main"
Sep 10 21:21:34.574131 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.573991    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3d505e-09b3-4167-a9a8-d12f03b59077" containerName="main"
Sep 10 21:21:34.574131 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.574010    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca940185-2e99-4f51-8ca7-8aa2b534d893" containerName="main"
Sep 10 21:21:34.574131 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.574016    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca940185-2e99-4f51-8ca7-8aa2b534d893" containerName="main"
Sep 10 21:21:34.574131 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.574071    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3d505e-09b3-4167-a9a8-d12f03b59077" containerName="main"
Sep 10 21:21:34.574131 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.574082    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca940185-2e99-4f51-8ca7-8aa2b534d893" containerName="main"
Sep 10 21:21:34.577069 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.577046    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:21:34.580051 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.580029    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-16bbec\"/\"default-dockercfg-nwr5g\""
Sep 10 21:21:34.580190 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.580040    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-16bbec\"/\"kube-root-ca.crt\""
Sep 10 21:21:34.580190 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.580050    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-16bbec\"/\"sub-test-model-16bbec-kserve-self-signed-certs\""
Sep 10 21:21:34.580190 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.580079    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-16bbec\"/\"openshift-service-ca.crt\""
Sep 10 21:21:34.590123 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.590098    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"]
Sep 10 21:21:34.703585 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.703540    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9c40364-f7af-4050-906d-718f584a8cb9-home\") pod \"sub-test-model-16bbec-kserve-554fd8d7b9-sljs8\" (UID: \"c9c40364-f7af-4050-906d-718f584a8cb9\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:21:34.703749 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.703597    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9c40364-f7af-4050-906d-718f584a8cb9-tmp-dir\") pod \"sub-test-model-16bbec-kserve-554fd8d7b9-sljs8\" (UID: \"c9c40364-f7af-4050-906d-718f584a8cb9\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:21:34.703749 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.703653    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6fvg\" (UniqueName: \"kubernetes.io/projected/c9c40364-f7af-4050-906d-718f584a8cb9-kube-api-access-h6fvg\") pod \"sub-test-model-16bbec-kserve-554fd8d7b9-sljs8\" (UID: \"c9c40364-f7af-4050-906d-718f584a8cb9\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:21:34.703749 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.703696    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9c40364-f7af-4050-906d-718f584a8cb9-dshm\") pod \"sub-test-model-16bbec-kserve-554fd8d7b9-sljs8\" (UID: \"c9c40364-f7af-4050-906d-718f584a8cb9\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:21:34.703749 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.703738    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c40364-f7af-4050-906d-718f584a8cb9-model-cache\") pod \"sub-test-model-16bbec-kserve-554fd8d7b9-sljs8\" (UID: \"c9c40364-f7af-4050-906d-718f584a8cb9\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:21:34.703895 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.703795    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c40364-f7af-4050-906d-718f584a8cb9-tls-certs\") pod \"sub-test-model-16bbec-kserve-554fd8d7b9-sljs8\" (UID: \"c9c40364-f7af-4050-906d-718f584a8cb9\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:21:34.805324 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.805284    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9c40364-f7af-4050-906d-718f584a8cb9-home\") pod \"sub-test-model-16bbec-kserve-554fd8d7b9-sljs8\" (UID: \"c9c40364-f7af-4050-906d-718f584a8cb9\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:21:34.805324 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.805329    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9c40364-f7af-4050-906d-718f584a8cb9-tmp-dir\") pod \"sub-test-model-16bbec-kserve-554fd8d7b9-sljs8\" (UID: \"c9c40364-f7af-4050-906d-718f584a8cb9\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:21:34.805611 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.805486    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6fvg\" (UniqueName: \"kubernetes.io/projected/c9c40364-f7af-4050-906d-718f584a8cb9-kube-api-access-h6fvg\") pod \"sub-test-model-16bbec-kserve-554fd8d7b9-sljs8\" (UID: \"c9c40364-f7af-4050-906d-718f584a8cb9\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:21:34.805611 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.805594    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9c40364-f7af-4050-906d-718f584a8cb9-dshm\") pod \"sub-test-model-16bbec-kserve-554fd8d7b9-sljs8\" (UID: \"c9c40364-f7af-4050-906d-718f584a8cb9\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:21:34.805726 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.805637    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c40364-f7af-4050-906d-718f584a8cb9-model-cache\") pod \"sub-test-model-16bbec-kserve-554fd8d7b9-sljs8\" (UID: \"c9c40364-f7af-4050-906d-718f584a8cb9\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:21:34.805726 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.805674    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9c40364-f7af-4050-906d-718f584a8cb9-tmp-dir\") pod \"sub-test-model-16bbec-kserve-554fd8d7b9-sljs8\" (UID: \"c9c40364-f7af-4050-906d-718f584a8cb9\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:21:34.805726 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.805702    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c40364-f7af-4050-906d-718f584a8cb9-tls-certs\") pod \"sub-test-model-16bbec-kserve-554fd8d7b9-sljs8\" (UID: \"c9c40364-f7af-4050-906d-718f584a8cb9\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:21:34.805851 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.805729    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9c40364-f7af-4050-906d-718f584a8cb9-home\") pod \"sub-test-model-16bbec-kserve-554fd8d7b9-sljs8\" (UID: \"c9c40364-f7af-4050-906d-718f584a8cb9\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:21:34.805888 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.805852    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c40364-f7af-4050-906d-718f584a8cb9-model-cache\") pod \"sub-test-model-16bbec-kserve-554fd8d7b9-sljs8\" (UID: \"c9c40364-f7af-4050-906d-718f584a8cb9\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:21:34.807745 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.807715    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9c40364-f7af-4050-906d-718f584a8cb9-dshm\") pod \"sub-test-model-16bbec-kserve-554fd8d7b9-sljs8\" (UID: \"c9c40364-f7af-4050-906d-718f584a8cb9\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:21:34.807989 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.807972    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c40364-f7af-4050-906d-718f584a8cb9-tls-certs\") pod \"sub-test-model-16bbec-kserve-554fd8d7b9-sljs8\" (UID: \"c9c40364-f7af-4050-906d-718f584a8cb9\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:21:34.814464 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.814416    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6fvg\" (UniqueName: \"kubernetes.io/projected/c9c40364-f7af-4050-906d-718f584a8cb9-kube-api-access-h6fvg\") pod \"sub-test-model-16bbec-kserve-554fd8d7b9-sljs8\" (UID: \"c9c40364-f7af-4050-906d-718f584a8cb9\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:21:34.886551 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:34.886528    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:21:35.160443 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:35.160293    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"]
Sep 10 21:21:35.162871 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:21:35.162840    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9c40364_f7af_4050_906d_718f584a8cb9.slice/crio-98556a59b44cec2fa0d8d89586c6c1a59cdeeb05b22dd45aab6ae60bf8e6ed65 WatchSource:0}: Error finding container 98556a59b44cec2fa0d8d89586c6c1a59cdeeb05b22dd45aab6ae60bf8e6ed65: Status 404 returned error can't find the container with id 98556a59b44cec2fa0d8d89586c6c1a59cdeeb05b22dd45aab6ae60bf8e6ed65
Sep 10 21:21:35.483182 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:35.483090    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8" event={"ID":"c9c40364-f7af-4050-906d-718f584a8cb9","Type":"ContainerStarted","Data":"f90e6cf97a5717f0d9d801dc5c361dd1b1abb4acaa3c60e814ffdd78dd1f4b9c"}
Sep 10 21:21:35.483182 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:35.483131    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8" event={"ID":"c9c40364-f7af-4050-906d-718f584a8cb9","Type":"ContainerStarted","Data":"98556a59b44cec2fa0d8d89586c6c1a59cdeeb05b22dd45aab6ae60bf8e6ed65"}
Sep 10 21:21:35.502319 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:35.502273    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8" podStartSLOduration=1.502258442 podStartE2EDuration="1.502258442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:21:35.500151072 +0000 UTC m=+1355.165175872" watchObservedRunningTime="2026-09-10 21:21:35.502258442 +0000 UTC m=+1355.167283241"
Sep 10 21:21:44.887698 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:44.887668    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:21:44.888200 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:44.887950    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:21:44.899911 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:44.899887    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:21:45.543439 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.543405    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:21:45.671251 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.671220    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"]
Sep 10 21:21:45.674721 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.674695    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.676914 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.676897    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a249a2-openshift-default-dockercfg-8z9d5\""
Sep 10 21:21:45.687058 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.687036    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"]
Sep 10 21:21:45.813744 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.813658    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxnfc\" (UniqueName: \"kubernetes.io/projected/bd12952d-7efc-450a-a457-bc039f995340-kube-api-access-nxnfc\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.813744 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.813710    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-workload-socket\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.813932 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.813843    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-istio-data\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.813932 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.813879    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-credential-socket\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.813932 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.813900    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-workload-certs\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.813932 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.813915    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd12952d-7efc-450a-a457-bc039f995340-istio-token\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.814069 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.813960    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd12952d-7efc-450a-a457-bc039f995340-istiod-ca-cert\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.814069 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.814023    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd12952d-7efc-450a-a457-bc039f995340-istio-podinfo\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.814069 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.814057    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-istio-envoy\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.915263 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.915233    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd12952d-7efc-450a-a457-bc039f995340-istiod-ca-cert\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.915735 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.915300    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd12952d-7efc-450a-a457-bc039f995340-istio-podinfo\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.915735 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.915355    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-istio-envoy\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.915735 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.915389    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnfc\" (UniqueName: \"kubernetes.io/projected/bd12952d-7efc-450a-a457-bc039f995340-kube-api-access-nxnfc\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.915735 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.915590    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-workload-socket\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.915735 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.915724    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-istio-data\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.916024 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.915762    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-credential-socket\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.916024 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.915811    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-workload-certs\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.916024 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.915840    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd12952d-7efc-450a-a457-bc039f995340-istio-token\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.916186 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.916119    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-workload-socket\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.916186 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.916148    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd12952d-7efc-450a-a457-bc039f995340-istiod-ca-cert\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.916304 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.916186    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-workload-certs\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.916304 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.916232    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-istio-data\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.916392 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.916363    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-credential-socket\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.917990 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.917962    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-istio-envoy\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.918095 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.918024    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd12952d-7efc-450a-a457-bc039f995340-istio-podinfo\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.922975 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.922953    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd12952d-7efc-450a-a457-bc039f995340-istio-token\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.923185 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.923161    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxnfc\" (UniqueName: \"kubernetes.io/projected/bd12952d-7efc-450a-a457-bc039f995340-kube-api-access-nxnfc\") pod \"e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") " pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:45.987092 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:45.987065    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:46.264041 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:46.264014    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"]
Sep 10 21:21:46.266886 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:21:46.266857    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd12952d_7efc_450a_a457_bc039f995340.slice/crio-b7d3ce0909e5e59f39fdd8b791ad60af644046f9afbb40a819972010cd3e830b WatchSource:0}: Error finding container b7d3ce0909e5e59f39fdd8b791ad60af644046f9afbb40a819972010cd3e830b: Status 404 returned error can't find the container with id b7d3ce0909e5e59f39fdd8b791ad60af644046f9afbb40a819972010cd3e830b
Sep 10 21:21:46.269488 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:46.269452    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:21:46.269585 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:46.269518    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:21:46.269585 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:46.269548    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:21:46.539777 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:46.539687    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7" event={"ID":"bd12952d-7efc-450a-a457-bc039f995340","Type":"ContainerStarted","Data":"531773008d0886620082702789773fb3b4660d38ba031c5f09b3d3f6ea68c80c"}
Sep 10 21:21:46.539777 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:46.539744    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7" event={"ID":"bd12952d-7efc-450a-a457-bc039f995340","Type":"ContainerStarted","Data":"b7d3ce0909e5e59f39fdd8b791ad60af644046f9afbb40a819972010cd3e830b"}
Sep 10 21:21:46.561479 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:46.561407    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7" podStartSLOduration=1.5613861 podStartE2EDuration="1.5613861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:21:46.560611396 +0000 UTC m=+1366.225636205" watchObservedRunningTime="2026-09-10 21:21:46.5613861 +0000 UTC m=+1366.226410903"
Sep 10 21:21:46.987807 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:46.987773    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:47.993679 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:47.993646    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:48.550984 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:48.550951    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:21:48.552023 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:21:48.552003    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:22:06.653435 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:06.653330    2584 generic.go:356] "Generic (PLEG): container finished" podID="bd12952d-7efc-450a-a457-bc039f995340" containerID="531773008d0886620082702789773fb3b4660d38ba031c5f09b3d3f6ea68c80c" exitCode=0
Sep 10 21:22:06.653435 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:06.653395    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7" event={"ID":"bd12952d-7efc-450a-a457-bc039f995340","Type":"ContainerDied","Data":"531773008d0886620082702789773fb3b4660d38ba031c5f09b3d3f6ea68c80c"}
Sep 10 21:22:06.653908 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:06.653829    2584 scope.go:117] "RemoveContainer" containerID="531773008d0886620082702789773fb3b4660d38ba031c5f09b3d3f6ea68c80c"
Sep 10 21:22:06.655066 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:06.655026    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:22:06.655284 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:06.655098    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:22:06.655284 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:06.655139    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:22:07.660948 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:07.660913    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7" event={"ID":"bd12952d-7efc-450a-a457-bc039f995340","Type":"ContainerStarted","Data":"15a8131ac467fdf56459bd4f6bc623a92b08448b52ac399ad7973e98650325ee"}
Sep 10 21:22:07.987981 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:07.987878    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:22:07.988167 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:07.988145    2584 patch_prober.go:28] interesting pod/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 10 21:22:07.988221 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:07.988196    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7" podUID="bd12952d-7efc-450a-a457-bc039f995340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 10 21:22:08.988195 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:08.988156    2584 patch_prober.go:28] interesting pod/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 10 21:22:08.988644 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:08.988230    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7" podUID="bd12952d-7efc-450a-a457-bc039f995340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 10 21:22:09.987769 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:09.987737    2584 patch_prober.go:28] interesting pod/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 10 21:22:09.987938 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:09.987802    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7" podUID="bd12952d-7efc-450a-a457-bc039f995340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 10 21:22:10.988417 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:10.988382    2584 patch_prober.go:28] interesting pod/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 10 21:22:10.988810 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:10.988483    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7" podUID="bd12952d-7efc-450a-a457-bc039f995340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 10 21:22:11.988223 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:11.988187    2584 patch_prober.go:28] interesting pod/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 10 21:22:11.988404 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:11.988253    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7" podUID="bd12952d-7efc-450a-a457-bc039f995340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 10 21:22:12.991924 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:12.991891    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:22:12.992322 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:12.992198    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:22:12.992865 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:12.992847    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:22:14.369854 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.369821    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"]
Sep 10 21:22:14.373056 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.373033    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.375734 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.375714    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-cf7171-openshift-default-dockercfg-78k2n\""
Sep 10 21:22:14.387207 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.387183    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"]
Sep 10 21:22:14.513784 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.513752    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-istio-data\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.513935 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.513802    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-workload-certs\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.513935 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.513824    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d9d1849-4946-4693-910a-eecce8049eee-istio-token\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.513935 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.513915    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d9d1849-4946-4693-910a-eecce8049eee-istiod-ca-cert\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.514044 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.513943    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d9d1849-4946-4693-910a-eecce8049eee-istio-podinfo\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.514044 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.513979    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-workload-socket\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.514044 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.514004    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-istio-envoy\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.514044 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.514022    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j487p\" (UniqueName: \"kubernetes.io/projected/4d9d1849-4946-4693-910a-eecce8049eee-kube-api-access-j487p\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.514191 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.514058    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-credential-socket\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.615405 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.615371    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-istio-envoy\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.615405 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.615410    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j487p\" (UniqueName: \"kubernetes.io/projected/4d9d1849-4946-4693-910a-eecce8049eee-kube-api-access-j487p\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.615654 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.615462    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-credential-socket\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.615654 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.615548    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-istio-data\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.615654 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.615596    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-workload-certs\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.615654 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.615628    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d9d1849-4946-4693-910a-eecce8049eee-istio-token\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.615846 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.615709    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d9d1849-4946-4693-910a-eecce8049eee-istiod-ca-cert\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.615846 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.615736    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d9d1849-4946-4693-910a-eecce8049eee-istio-podinfo\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.615846 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.615789    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-workload-socket\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.615978 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.615906    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-credential-socket\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.615978 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.615969    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-istio-data\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.616197 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.616175    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-workload-certs\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.616197 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.616189    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-workload-socket\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.616395 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.616367    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d9d1849-4946-4693-910a-eecce8049eee-istiod-ca-cert\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.617755 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.617730    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-istio-envoy\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.618234 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.618217    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d9d1849-4946-4693-910a-eecce8049eee-istio-podinfo\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.623728 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.623675    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d9d1849-4946-4693-910a-eecce8049eee-istio-token\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.624296 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.624271    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j487p\" (UniqueName: \"kubernetes.io/projected/4d9d1849-4946-4693-910a-eecce8049eee-kube-api-access-j487p\") pod \"e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") " pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.684104 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.684085    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:14.967702 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.967626    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"]
Sep 10 21:22:14.970889 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:22:14.970861    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9d1849_4946_4693_910a_eecce8049eee.slice/crio-dbf4827386fac7cc08ac06a70f96c541a712f01a4858437f00abab104911a8ff WatchSource:0}: Error finding container dbf4827386fac7cc08ac06a70f96c541a712f01a4858437f00abab104911a8ff: Status 404 returned error can't find the container with id dbf4827386fac7cc08ac06a70f96c541a712f01a4858437f00abab104911a8ff
Sep 10 21:22:14.973204 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.973174    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:22:14.973281 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.973241    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:22:14.973281 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:14.973271    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:22:15.709807 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:15.709772    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2" event={"ID":"4d9d1849-4946-4693-910a-eecce8049eee","Type":"ContainerStarted","Data":"c22153a0eef611e0b66755fcc02040a47df67915b37e94b2235defd9cc74295d"}
Sep 10 21:22:15.710156 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:15.709826    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2" event={"ID":"4d9d1849-4946-4693-910a-eecce8049eee","Type":"ContainerStarted","Data":"dbf4827386fac7cc08ac06a70f96c541a712f01a4858437f00abab104911a8ff"}
Sep 10 21:22:15.732262 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:15.732202    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2" podStartSLOduration=1.7321881019999998 podStartE2EDuration="1.732188102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:22:15.729973564 +0000 UTC m=+1395.394998364" watchObservedRunningTime="2026-09-10 21:22:15.732188102 +0000 UTC m=+1395.397212901"
Sep 10 21:22:16.684741 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:16.684710    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:16.689645 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:16.689619    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:16.715027 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:16.715003    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:16.715739 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:16.715723    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:30.706147 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:30.706111    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8125-0d1197"]
Sep 10 21:22:30.711781 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:30.711758    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8125-0d1197"
Sep 10 21:22:30.714210 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:30.714181    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cc2nx\""
Sep 10 21:22:30.719854 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:30.719818    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8125-0d1197"]
Sep 10 21:22:30.779190 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:30.779158    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2zsm\" (UniqueName: \"kubernetes.io/projected/d0b04c20-0f31-4441-bbda-54645f39a8aa-kube-api-access-q2zsm\") pod \"test-curl-8125-0d1197\" (UID: \"d0b04c20-0f31-4441-bbda-54645f39a8aa\") " pod="opendatahub/test-curl-8125-0d1197"
Sep 10 21:22:30.880722 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:30.880689    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2zsm\" (UniqueName: \"kubernetes.io/projected/d0b04c20-0f31-4441-bbda-54645f39a8aa-kube-api-access-q2zsm\") pod \"test-curl-8125-0d1197\" (UID: \"d0b04c20-0f31-4441-bbda-54645f39a8aa\") " pod="opendatahub/test-curl-8125-0d1197"
Sep 10 21:22:30.888859 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:30.888828    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2zsm\" (UniqueName: \"kubernetes.io/projected/d0b04c20-0f31-4441-bbda-54645f39a8aa-kube-api-access-q2zsm\") pod \"test-curl-8125-0d1197\" (UID: \"d0b04c20-0f31-4441-bbda-54645f39a8aa\") " pod="opendatahub/test-curl-8125-0d1197"
Sep 10 21:22:31.030565 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:31.030468    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8125-0d1197"
Sep 10 21:22:31.307758 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:31.307627    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8125-0d1197"]
Sep 10 21:22:31.317565 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:22:31.317298    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0b04c20_0f31_4441_bbda_54645f39a8aa.slice/crio-3b6f1cb4d3815be5f307b166111c6a374a24d1715397498f49b9e8716e80cba6 WatchSource:0}: Error finding container 3b6f1cb4d3815be5f307b166111c6a374a24d1715397498f49b9e8716e80cba6: Status 404 returned error can't find the container with id 3b6f1cb4d3815be5f307b166111c6a374a24d1715397498f49b9e8716e80cba6
Sep 10 21:22:31.792356 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:31.792314    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8125-0d1197" event={"ID":"d0b04c20-0f31-4441-bbda-54645f39a8aa","Type":"ContainerStarted","Data":"b396b9aa8f70824de7ff08f806e5bde0f9008559e90cde11f3af0accd9c2e153"}
Sep 10 21:22:31.792356 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:31.792363    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8125-0d1197" event={"ID":"d0b04c20-0f31-4441-bbda-54645f39a8aa","Type":"ContainerStarted","Data":"3b6f1cb4d3815be5f307b166111c6a374a24d1715397498f49b9e8716e80cba6"}
Sep 10 21:22:32.466269 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:32.466211    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8125-0d1197" podStartSLOduration=2.46618965 podStartE2EDuration="2.46618965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:22:31.811990705 +0000 UTC m=+1411.477015507" watchObservedRunningTime="2026-09-10 21:22:32.46618965 +0000 UTC m=+1412.131214450"
Sep 10 21:22:32.466525 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:32.466454    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8125-0d1197"]
Sep 10 21:22:32.470938 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:32.470862    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8125-0d1197"]
Sep 10 21:22:32.592622 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:32.592589    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8125-d10426"]
Sep 10 21:22:32.593012 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:32.592999    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0b04c20-0f31-4441-bbda-54645f39a8aa" containerName="test-curl-8125-0d1197"
Sep 10 21:22:32.593069 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:32.593013    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0b04c20-0f31-4441-bbda-54645f39a8aa" containerName="test-curl-8125-0d1197"
Sep 10 21:22:32.593108 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:32.593085    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0b04c20-0f31-4441-bbda-54645f39a8aa" containerName="test-curl-8125-0d1197"
Sep 10 21:22:32.596008 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:32.595992    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8125-d10426"
Sep 10 21:22:32.601531 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:32.601504    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8125-d10426"]
Sep 10 21:22:32.698930 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:32.698893    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgh77\" (UniqueName: \"kubernetes.io/projected/f8e0c90c-0d58-4beb-b46b-c7b663c771e3-kube-api-access-qgh77\") pod \"test-curl-8125-d10426\" (UID: \"f8e0c90c-0d58-4beb-b46b-c7b663c771e3\") " pod="opendatahub/test-curl-8125-d10426"
Sep 10 21:22:32.797221 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:22:32.797140    2584 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8125-0d1197" container="test-curl-8125-0d1197"
Sep 10 21:22:32.800212 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:32.800191    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgh77\" (UniqueName: \"kubernetes.io/projected/f8e0c90c-0d58-4beb-b46b-c7b663c771e3-kube-api-access-qgh77\") pod \"test-curl-8125-d10426\" (UID: \"f8e0c90c-0d58-4beb-b46b-c7b663c771e3\") " pod="opendatahub/test-curl-8125-d10426"
Sep 10 21:22:32.810384 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:32.810347    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgh77\" (UniqueName: \"kubernetes.io/projected/f8e0c90c-0d58-4beb-b46b-c7b663c771e3-kube-api-access-qgh77\") pod \"test-curl-8125-d10426\" (UID: \"f8e0c90c-0d58-4beb-b46b-c7b663c771e3\") " pod="opendatahub/test-curl-8125-d10426"
Sep 10 21:22:32.909157 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:32.909107    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8125-d10426"
Sep 10 21:22:33.186957 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:33.186929    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8125-d10426"]
Sep 10 21:22:33.188643 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:22:33.188617    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8e0c90c_0d58_4beb_b46b_c7b663c771e3.slice/crio-65e77ea3298b2102cf9b22ac856d28404da020146a50599a7dfc21cafbaa9b77 WatchSource:0}: Error finding container 65e77ea3298b2102cf9b22ac856d28404da020146a50599a7dfc21cafbaa9b77: Status 404 returned error can't find the container with id 65e77ea3298b2102cf9b22ac856d28404da020146a50599a7dfc21cafbaa9b77
Sep 10 21:22:33.804623 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:33.804551    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8125-0d1197" podUID="d0b04c20-0f31-4441-bbda-54645f39a8aa" containerName="test-curl-8125-0d1197" containerID="cri-o://b396b9aa8f70824de7ff08f806e5bde0f9008559e90cde11f3af0accd9c2e153" gracePeriod=2
Sep 10 21:22:33.805249 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:33.805216    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8125-d10426" event={"ID":"f8e0c90c-0d58-4beb-b46b-c7b663c771e3","Type":"ContainerStarted","Data":"466f65fdb81298f6ba5adecef9aa051961453cac650faf887d04e61f62ccf8ca"}
Sep 10 21:22:33.805363 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:33.805268    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8125-d10426" event={"ID":"f8e0c90c-0d58-4beb-b46b-c7b663c771e3","Type":"ContainerStarted","Data":"65e77ea3298b2102cf9b22ac856d28404da020146a50599a7dfc21cafbaa9b77"}
Sep 10 21:22:33.821854 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:33.821790    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8125-d10426" podStartSLOduration=1.82176744 podStartE2EDuration="1.82176744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:22:33.820298157 +0000 UTC m=+1413.485322957" watchObservedRunningTime="2026-09-10 21:22:33.82176744 +0000 UTC m=+1413.486792242"
Sep 10 21:22:34.453894 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:34.453854    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8125-d10426"]
Sep 10 21:22:34.458777 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:34.458696    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8125-d10426"]
Sep 10 21:22:34.811800 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:22:34.811705    2584 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8125-d10426" container="test-curl-8125-d10426"
Sep 10 21:22:35.818106 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:35.818063    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8125-d10426" podUID="f8e0c90c-0d58-4beb-b46b-c7b663c771e3" containerName="test-curl-8125-d10426" containerID="cri-o://466f65fdb81298f6ba5adecef9aa051961453cac650faf887d04e61f62ccf8ca" gracePeriod=2
Sep 10 21:22:36.020800 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:36.020774    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8125-0d1197"
Sep 10 21:22:36.054172 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:36.054140    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2zsm\" (UniqueName: \"kubernetes.io/projected/d0b04c20-0f31-4441-bbda-54645f39a8aa-kube-api-access-q2zsm\") pod \"d0b04c20-0f31-4441-bbda-54645f39a8aa\" (UID: \"d0b04c20-0f31-4441-bbda-54645f39a8aa\") "
Sep 10 21:22:36.056260 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:36.056225    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0b04c20-0f31-4441-bbda-54645f39a8aa-kube-api-access-q2zsm" (OuterVolumeSpecName: "kube-api-access-q2zsm") pod "d0b04c20-0f31-4441-bbda-54645f39a8aa" (UID: "d0b04c20-0f31-4441-bbda-54645f39a8aa"). InnerVolumeSpecName "kube-api-access-q2zsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:22:36.155870 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:36.155827    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2zsm\" (UniqueName: \"kubernetes.io/projected/d0b04c20-0f31-4441-bbda-54645f39a8aa-kube-api-access-q2zsm\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:22:36.539002 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:36.538967    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"]
Sep 10 21:22:36.824584 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:36.824496    2584 generic.go:356] "Generic (PLEG): container finished" podID="d0b04c20-0f31-4441-bbda-54645f39a8aa" containerID="b396b9aa8f70824de7ff08f806e5bde0f9008559e90cde11f3af0accd9c2e153" exitCode=137
Sep 10 21:22:36.824584 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:36.824532    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8125-0d1197"
Sep 10 21:22:36.825046 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:36.824589    2584 scope.go:117] "RemoveContainer" containerID="b396b9aa8f70824de7ff08f806e5bde0f9008559e90cde11f3af0accd9c2e153"
Sep 10 21:22:36.833449 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:36.833403    2584 scope.go:117] "RemoveContainer" containerID="b396b9aa8f70824de7ff08f806e5bde0f9008559e90cde11f3af0accd9c2e153"
Sep 10 21:22:36.833865 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:22:36.833846    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b396b9aa8f70824de7ff08f806e5bde0f9008559e90cde11f3af0accd9c2e153\": container with ID starting with b396b9aa8f70824de7ff08f806e5bde0f9008559e90cde11f3af0accd9c2e153 not found: ID does not exist" containerID="b396b9aa8f70824de7ff08f806e5bde0f9008559e90cde11f3af0accd9c2e153"
Sep 10 21:22:36.833951 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:36.833872    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b396b9aa8f70824de7ff08f806e5bde0f9008559e90cde11f3af0accd9c2e153"} err="failed to get container status \"b396b9aa8f70824de7ff08f806e5bde0f9008559e90cde11f3af0accd9c2e153\": rpc error: code = NotFound desc = could not find container \"b396b9aa8f70824de7ff08f806e5bde0f9008559e90cde11f3af0accd9c2e153\": container with ID starting with b396b9aa8f70824de7ff08f806e5bde0f9008559e90cde11f3af0accd9c2e153 not found: ID does not exist"
Sep 10 21:22:36.835059 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:22:36.835031    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8125-0d1197\" is forbidden: User \"system:node:ip-10-0-135-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-205.ec2.internal' and this object" podUID="d0b04c20-0f31-4441-bbda-54645f39a8aa" pod="opendatahub/test-curl-8125-0d1197"
Sep 10 21:22:36.840694 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:36.840670    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0b04c20-0f31-4441-bbda-54645f39a8aa" path="/var/lib/kubelet/pods/d0b04c20-0f31-4441-bbda-54645f39a8aa/volumes"
Sep 10 21:22:37.256051 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:37.256011    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"]
Sep 10 21:22:38.006968 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:38.006940    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8125-d10426"
Sep 10 21:22:38.076563 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:38.076476    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgh77\" (UniqueName: \"kubernetes.io/projected/f8e0c90c-0d58-4beb-b46b-c7b663c771e3-kube-api-access-qgh77\") pod \"f8e0c90c-0d58-4beb-b46b-c7b663c771e3\" (UID: \"f8e0c90c-0d58-4beb-b46b-c7b663c771e3\") "
Sep 10 21:22:38.078573 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:38.078547    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e0c90c-0d58-4beb-b46b-c7b663c771e3-kube-api-access-qgh77" (OuterVolumeSpecName: "kube-api-access-qgh77") pod "f8e0c90c-0d58-4beb-b46b-c7b663c771e3" (UID: "f8e0c90c-0d58-4beb-b46b-c7b663c771e3"). InnerVolumeSpecName "kube-api-access-qgh77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:22:38.177984 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:38.177946    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgh77\" (UniqueName: \"kubernetes.io/projected/f8e0c90c-0d58-4beb-b46b-c7b663c771e3-kube-api-access-qgh77\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:22:38.839188 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:38.839091    2584 generic.go:356] "Generic (PLEG): container finished" podID="f8e0c90c-0d58-4beb-b46b-c7b663c771e3" containerID="466f65fdb81298f6ba5adecef9aa051961453cac650faf887d04e61f62ccf8ca" exitCode=137
Sep 10 21:22:38.839391 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:38.839190    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8125-d10426"
Sep 10 21:22:38.840960 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:38.840933    2584 generic.go:356] "Generic (PLEG): container finished" podID="4d9d1849-4946-4693-910a-eecce8049eee" containerID="c22153a0eef611e0b66755fcc02040a47df67915b37e94b2235defd9cc74295d" exitCode=0
Sep 10 21:22:38.841199 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:38.841182    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8e0c90c-0d58-4beb-b46b-c7b663c771e3" path="/var/lib/kubelet/pods/f8e0c90c-0d58-4beb-b46b-c7b663c771e3/volumes"
Sep 10 21:22:38.841576 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:38.841559    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2" event={"ID":"4d9d1849-4946-4693-910a-eecce8049eee","Type":"ContainerDied","Data":"c22153a0eef611e0b66755fcc02040a47df67915b37e94b2235defd9cc74295d"}
Sep 10 21:22:38.841643 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:38.841629    2584 scope.go:117] "RemoveContainer" containerID="466f65fdb81298f6ba5adecef9aa051961453cac650faf887d04e61f62ccf8ca"
Sep 10 21:22:38.841936 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:38.841918    2584 scope.go:117] "RemoveContainer" containerID="c22153a0eef611e0b66755fcc02040a47df67915b37e94b2235defd9cc74295d"
Sep 10 21:22:38.843252 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:38.843219    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:22:38.843357 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:38.843308    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:22:38.843416 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:38.843373    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:22:38.851110 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:38.851092    2584 scope.go:117] "RemoveContainer" containerID="466f65fdb81298f6ba5adecef9aa051961453cac650faf887d04e61f62ccf8ca"
Sep 10 21:22:38.851469 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:22:38.851394    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"466f65fdb81298f6ba5adecef9aa051961453cac650faf887d04e61f62ccf8ca\": container with ID starting with 466f65fdb81298f6ba5adecef9aa051961453cac650faf887d04e61f62ccf8ca not found: ID does not exist" containerID="466f65fdb81298f6ba5adecef9aa051961453cac650faf887d04e61f62ccf8ca"
Sep 10 21:22:38.851469 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:38.851462    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"466f65fdb81298f6ba5adecef9aa051961453cac650faf887d04e61f62ccf8ca"} err="failed to get container status \"466f65fdb81298f6ba5adecef9aa051961453cac650faf887d04e61f62ccf8ca\": rpc error: code = NotFound desc = could not find container \"466f65fdb81298f6ba5adecef9aa051961453cac650faf887d04e61f62ccf8ca\": container with ID starting with 466f65fdb81298f6ba5adecef9aa051961453cac650faf887d04e61f62ccf8ca not found: ID does not exist"
Sep 10 21:22:39.848693 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:39.848658    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2" event={"ID":"4d9d1849-4946-4693-910a-eecce8049eee","Type":"ContainerStarted","Data":"c45cf181d08d14bbe1cb49b2c290a1b029fa94c43e583f8713dea7a7008293dc"}
Sep 10 21:22:40.684745 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:40.684709    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:40.684931 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:40.684907    2584 patch_prober.go:28] interesting pod/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 10 21:22:40.684978 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:40.684952    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2" podUID="4d9d1849-4946-4693-910a-eecce8049eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 10 21:22:41.685505 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:41.685467    2584 patch_prober.go:28] interesting pod/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 10 21:22:41.685891 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:41.685527    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2" podUID="4d9d1849-4946-4693-910a-eecce8049eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 10 21:22:42.684761 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:42.684724    2584 patch_prober.go:28] interesting pod/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 10 21:22:42.684950 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:42.684785    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2" podUID="4d9d1849-4946-4693-910a-eecce8049eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 10 21:22:43.684565 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:43.684529    2584 patch_prober.go:28] interesting pod/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 10 21:22:43.684954 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:43.684582    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2" podUID="4d9d1849-4946-4693-910a-eecce8049eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 10 21:22:44.684713 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:44.684680    2584 patch_prober.go:28] interesting pod/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 10 21:22:44.685108 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:44.684734    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2" podUID="4d9d1849-4946-4693-910a-eecce8049eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 10 21:22:45.688865 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:45.688830    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:45.689263 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:45.689114    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:45.689749 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:45.689733    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:22:45.795104 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:45.795072    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a249a2-8bqqz"]
Sep 10 21:22:45.795528 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:45.795514    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e0c90c-0d58-4beb-b46b-c7b663c771e3" containerName="test-curl-8125-d10426"
Sep 10 21:22:45.795584 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:45.795529    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e0c90c-0d58-4beb-b46b-c7b663c771e3" containerName="test-curl-8125-d10426"
Sep 10 21:22:45.795625 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:45.795598    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e0c90c-0d58-4beb-b46b-c7b663c771e3" containerName="test-curl-8125-d10426"
Sep 10 21:22:45.797625 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:45.797608    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a249a2-8bqqz"
Sep 10 21:22:45.800276 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:45.800255    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:22:45.800416 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:45.800261    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4gs5\""
Sep 10 21:22:45.806966 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:45.806938    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a249a2-8bqqz"]
Sep 10 21:22:45.863490 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:45.863445    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa5e0906-ce06-45a1-bf08-4bd4005d4779-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a249a2-8bqqz\" (UID: \"fa5e0906-ce06-45a1-bf08-4bd4005d4779\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a249a2-8bqqz"
Sep 10 21:22:45.965019 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:45.964916    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa5e0906-ce06-45a1-bf08-4bd4005d4779-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a249a2-8bqqz\" (UID: \"fa5e0906-ce06-45a1-bf08-4bd4005d4779\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a249a2-8bqqz"
Sep 10 21:22:45.965727 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:45.965703    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa5e0906-ce06-45a1-bf08-4bd4005d4779-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a249a2-8bqqz\" (UID: \"fa5e0906-ce06-45a1-bf08-4bd4005d4779\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a249a2-8bqqz"
Sep 10 21:22:46.111008 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:46.110962    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a249a2-8bqqz"
Sep 10 21:22:46.404308 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:46.404272    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a249a2-8bqqz"]
Sep 10 21:22:46.405414 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:22:46.405389    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa5e0906_ce06_45a1_bf08_4bd4005d4779.slice/crio-f5b509c28dfd6e51309e56a797daa60041f4c359eaf8ce77a891fd92c8aed19f WatchSource:0}: Error finding container f5b509c28dfd6e51309e56a797daa60041f4c359eaf8ce77a891fd92c8aed19f: Status 404 returned error can't find the container with id f5b509c28dfd6e51309e56a797daa60041f4c359eaf8ce77a891fd92c8aed19f
Sep 10 21:22:46.885667 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:46.885632    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a249a2-8bqqz" event={"ID":"fa5e0906-ce06-45a1-bf08-4bd4005d4779","Type":"ContainerStarted","Data":"10feb7b749904cfa0d28520eef96a409ae4bb95bb27a314e3f01a36dd50dd7f0"}
Sep 10 21:22:46.885667 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:46.885681    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a249a2-8bqqz" event={"ID":"fa5e0906-ce06-45a1-bf08-4bd4005d4779","Type":"ContainerStarted","Data":"f5b509c28dfd6e51309e56a797daa60041f4c359eaf8ce77a891fd92c8aed19f"}
Sep 10 21:22:46.901970 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:46.901917    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a249a2-8bqqz" podStartSLOduration=1.901903825 podStartE2EDuration="1.901903825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:22:46.899536819 +0000 UTC m=+1426.564561619" watchObservedRunningTime="2026-09-10 21:22:46.901903825 +0000 UTC m=+1426.566928625"
Sep 10 21:22:46.972619 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:46.972546    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"]
Sep 10 21:22:46.975456 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:46.975435    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:22:46.977707 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:46.977686    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0ac302\"/\"rate-test-model-0ac302-kserve-self-signed-certs\""
Sep 10 21:22:46.984500 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:46.984480    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"]
Sep 10 21:22:47.074864 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.074831    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0256de55-42b3-4b85-a6ab-959a51325e8e-model-cache\") pod \"rate-test-model-0ac302-kserve-67bb4d8875-qznnz\" (UID: \"0256de55-42b3-4b85-a6ab-959a51325e8e\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:22:47.074864 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.074865    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0256de55-42b3-4b85-a6ab-959a51325e8e-dshm\") pod \"rate-test-model-0ac302-kserve-67bb4d8875-qznnz\" (UID: \"0256de55-42b3-4b85-a6ab-959a51325e8e\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:22:47.075105 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.074886    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hzx5\" (UniqueName: \"kubernetes.io/projected/0256de55-42b3-4b85-a6ab-959a51325e8e-kube-api-access-6hzx5\") pod \"rate-test-model-0ac302-kserve-67bb4d8875-qznnz\" (UID: \"0256de55-42b3-4b85-a6ab-959a51325e8e\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:22:47.075105 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.075005    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0256de55-42b3-4b85-a6ab-959a51325e8e-tmp-dir\") pod \"rate-test-model-0ac302-kserve-67bb4d8875-qznnz\" (UID: \"0256de55-42b3-4b85-a6ab-959a51325e8e\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:22:47.075105 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.075043    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0256de55-42b3-4b85-a6ab-959a51325e8e-home\") pod \"rate-test-model-0ac302-kserve-67bb4d8875-qznnz\" (UID: \"0256de55-42b3-4b85-a6ab-959a51325e8e\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:22:47.075105 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.075099    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0256de55-42b3-4b85-a6ab-959a51325e8e-tls-certs\") pod \"rate-test-model-0ac302-kserve-67bb4d8875-qznnz\" (UID: \"0256de55-42b3-4b85-a6ab-959a51325e8e\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:22:47.176290 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.176258    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0256de55-42b3-4b85-a6ab-959a51325e8e-model-cache\") pod \"rate-test-model-0ac302-kserve-67bb4d8875-qznnz\" (UID: \"0256de55-42b3-4b85-a6ab-959a51325e8e\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:22:47.176479 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.176299    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0256de55-42b3-4b85-a6ab-959a51325e8e-dshm\") pod \"rate-test-model-0ac302-kserve-67bb4d8875-qznnz\" (UID: \"0256de55-42b3-4b85-a6ab-959a51325e8e\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:22:47.176479 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.176437    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hzx5\" (UniqueName: \"kubernetes.io/projected/0256de55-42b3-4b85-a6ab-959a51325e8e-kube-api-access-6hzx5\") pod \"rate-test-model-0ac302-kserve-67bb4d8875-qznnz\" (UID: \"0256de55-42b3-4b85-a6ab-959a51325e8e\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:22:47.176618 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.176510    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0256de55-42b3-4b85-a6ab-959a51325e8e-tmp-dir\") pod \"rate-test-model-0ac302-kserve-67bb4d8875-qznnz\" (UID: \"0256de55-42b3-4b85-a6ab-959a51325e8e\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:22:47.176618 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.176547    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0256de55-42b3-4b85-a6ab-959a51325e8e-home\") pod \"rate-test-model-0ac302-kserve-67bb4d8875-qznnz\" (UID: \"0256de55-42b3-4b85-a6ab-959a51325e8e\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:22:47.176618 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.176578    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0256de55-42b3-4b85-a6ab-959a51325e8e-tls-certs\") pod \"rate-test-model-0ac302-kserve-67bb4d8875-qznnz\" (UID: \"0256de55-42b3-4b85-a6ab-959a51325e8e\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:22:47.176770 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.176716    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0256de55-42b3-4b85-a6ab-959a51325e8e-model-cache\") pod \"rate-test-model-0ac302-kserve-67bb4d8875-qznnz\" (UID: \"0256de55-42b3-4b85-a6ab-959a51325e8e\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:22:47.176822 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.176797    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0256de55-42b3-4b85-a6ab-959a51325e8e-tmp-dir\") pod \"rate-test-model-0ac302-kserve-67bb4d8875-qznnz\" (UID: \"0256de55-42b3-4b85-a6ab-959a51325e8e\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:22:47.176867 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.176835    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0256de55-42b3-4b85-a6ab-959a51325e8e-home\") pod \"rate-test-model-0ac302-kserve-67bb4d8875-qznnz\" (UID: \"0256de55-42b3-4b85-a6ab-959a51325e8e\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:22:47.178440 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.178401    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0256de55-42b3-4b85-a6ab-959a51325e8e-dshm\") pod \"rate-test-model-0ac302-kserve-67bb4d8875-qznnz\" (UID: \"0256de55-42b3-4b85-a6ab-959a51325e8e\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:22:47.178827 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.178812    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0256de55-42b3-4b85-a6ab-959a51325e8e-tls-certs\") pod \"rate-test-model-0ac302-kserve-67bb4d8875-qznnz\" (UID: \"0256de55-42b3-4b85-a6ab-959a51325e8e\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:22:47.185511 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.185489    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hzx5\" (UniqueName: \"kubernetes.io/projected/0256de55-42b3-4b85-a6ab-959a51325e8e-kube-api-access-6hzx5\") pod \"rate-test-model-0ac302-kserve-67bb4d8875-qznnz\" (UID: \"0256de55-42b3-4b85-a6ab-959a51325e8e\") " pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:22:47.285641 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.285548    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:22:47.563181 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.563151    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"]
Sep 10 21:22:47.565491 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:22:47.565465    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0256de55_42b3_4b85_a6ab_959a51325e8e.slice/crio-5ce2d3428e96228ff1f2366ab53705eb0fa59e4cf75a22d530bc6313ee5a0a7d WatchSource:0}: Error finding container 5ce2d3428e96228ff1f2366ab53705eb0fa59e4cf75a22d530bc6313ee5a0a7d: Status 404 returned error can't find the container with id 5ce2d3428e96228ff1f2366ab53705eb0fa59e4cf75a22d530bc6313ee5a0a7d
Sep 10 21:22:47.892287 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.892254    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz" event={"ID":"0256de55-42b3-4b85-a6ab-959a51325e8e","Type":"ContainerStarted","Data":"736ad175c419d52b942d418a69fc58227556d0d2121d20e1163dc66d0853a880"}
Sep 10 21:22:47.892768 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.892300    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz" event={"ID":"0256de55-42b3-4b85-a6ab-959a51325e8e","Type":"ContainerStarted","Data":"5ce2d3428e96228ff1f2366ab53705eb0fa59e4cf75a22d530bc6313ee5a0a7d"}
Sep 10 21:22:47.893840 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.893818    2584 generic.go:356] "Generic (PLEG): container finished" podID="fa5e0906-ce06-45a1-bf08-4bd4005d4779" containerID="10feb7b749904cfa0d28520eef96a409ae4bb95bb27a314e3f01a36dd50dd7f0" exitCode=0
Sep 10 21:22:47.893935 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.893855    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a249a2-8bqqz" event={"ID":"fa5e0906-ce06-45a1-bf08-4bd4005d4779","Type":"ContainerDied","Data":"10feb7b749904cfa0d28520eef96a409ae4bb95bb27a314e3f01a36dd50dd7f0"}
Sep 10 21:22:47.914921 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:47.914880    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz" podStartSLOduration=1.914866832 podStartE2EDuration="1.914866832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:22:47.913076246 +0000 UTC m=+1427.578101047" watchObservedRunningTime="2026-09-10 21:22:47.914866832 +0000 UTC m=+1427.579891684"
Sep 10 21:22:49.034367 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:49.034344    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a249a2-8bqqz"
Sep 10 21:22:49.199312 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:49.199235    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa5e0906-ce06-45a1-bf08-4bd4005d4779-maas-api-service-ca\") pod \"fa5e0906-ce06-45a1-bf08-4bd4005d4779\" (UID: \"fa5e0906-ce06-45a1-bf08-4bd4005d4779\") "
Sep 10 21:22:49.199717 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:49.199688    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa5e0906-ce06-45a1-bf08-4bd4005d4779-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fa5e0906-ce06-45a1-bf08-4bd4005d4779" (UID: "fa5e0906-ce06-45a1-bf08-4bd4005d4779"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:22:49.300590 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:49.300556    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa5e0906-ce06-45a1-bf08-4bd4005d4779-maas-api-service-ca\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:22:49.903980 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:49.903951    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a249a2-8bqqz"
Sep 10 21:22:49.904167 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:49.903950    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a249a2-8bqqz" event={"ID":"fa5e0906-ce06-45a1-bf08-4bd4005d4779","Type":"ContainerDied","Data":"f5b509c28dfd6e51309e56a797daa60041f4c359eaf8ce77a891fd92c8aed19f"}
Sep 10 21:22:49.904167 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:49.904077    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5b509c28dfd6e51309e56a797daa60041f4c359eaf8ce77a891fd92c8aed19f"
Sep 10 21:22:51.568119 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:51.568069    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn" podUID="8e94e56c-d489-43ba-b44c-c656a25b5225" containerName="main" containerID="cri-o://3e2677f7f60a96955c200563b971a409728e5f8909310e9d62b1a3446d64c714" gracePeriod=45
Sep 10 21:22:51.816568 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:51.816545    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:22:51.915901 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:51.915864    2584 generic.go:356] "Generic (PLEG): container finished" podID="8e94e56c-d489-43ba-b44c-c656a25b5225" containerID="3e2677f7f60a96955c200563b971a409728e5f8909310e9d62b1a3446d64c714" exitCode=0
Sep 10 21:22:51.916071 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:51.915909    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"
Sep 10 21:22:51.916071 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:51.915918    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn" event={"ID":"8e94e56c-d489-43ba-b44c-c656a25b5225","Type":"ContainerDied","Data":"3e2677f7f60a96955c200563b971a409728e5f8909310e9d62b1a3446d64c714"}
Sep 10 21:22:51.916071 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:51.915949    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn" event={"ID":"8e94e56c-d489-43ba-b44c-c656a25b5225","Type":"ContainerDied","Data":"bb862cf820638b7715210b96f8617617f8eea7fe6742043b4222662cba273547"}
Sep 10 21:22:51.916071 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:51.915965    2584 scope.go:117] "RemoveContainer" containerID="3e2677f7f60a96955c200563b971a409728e5f8909310e9d62b1a3446d64c714"
Sep 10 21:22:51.924554 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:51.924536    2584 scope.go:117] "RemoveContainer" containerID="3e2677f7f60a96955c200563b971a409728e5f8909310e9d62b1a3446d64c714"
Sep 10 21:22:51.924852 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:22:51.924831    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e2677f7f60a96955c200563b971a409728e5f8909310e9d62b1a3446d64c714\": container with ID starting with 3e2677f7f60a96955c200563b971a409728e5f8909310e9d62b1a3446d64c714 not found: ID does not exist" containerID="3e2677f7f60a96955c200563b971a409728e5f8909310e9d62b1a3446d64c714"
Sep 10 21:22:51.924904 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:51.924873    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e2677f7f60a96955c200563b971a409728e5f8909310e9d62b1a3446d64c714"} err="failed to get container status \"3e2677f7f60a96955c200563b971a409728e5f8909310e9d62b1a3446d64c714\": rpc error: code = NotFound desc = could not find container \"3e2677f7f60a96955c200563b971a409728e5f8909310e9d62b1a3446d64c714\": container with ID starting with 3e2677f7f60a96955c200563b971a409728e5f8909310e9d62b1a3446d64c714 not found: ID does not exist"
Sep 10 21:22:51.929167 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:51.929150    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e94e56c-d489-43ba-b44c-c656a25b5225-dshm\") pod \"8e94e56c-d489-43ba-b44c-c656a25b5225\" (UID: \"8e94e56c-d489-43ba-b44c-c656a25b5225\") "
Sep 10 21:22:51.929277 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:51.929264    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e94e56c-d489-43ba-b44c-c656a25b5225-model-cache\") pod \"8e94e56c-d489-43ba-b44c-c656a25b5225\" (UID: \"8e94e56c-d489-43ba-b44c-c656a25b5225\") "
Sep 10 21:22:51.929340 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:51.929292    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e94e56c-d489-43ba-b44c-c656a25b5225-home\") pod \"8e94e56c-d489-43ba-b44c-c656a25b5225\" (UID: \"8e94e56c-d489-43ba-b44c-c656a25b5225\") "
Sep 10 21:22:51.929340 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:51.929319    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e94e56c-d489-43ba-b44c-c656a25b5225-tmp-dir\") pod \"8e94e56c-d489-43ba-b44c-c656a25b5225\" (UID: \"8e94e56c-d489-43ba-b44c-c656a25b5225\") "
Sep 10 21:22:51.929513 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:51.929345    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e94e56c-d489-43ba-b44c-c656a25b5225-tls-certs\") pod \"8e94e56c-d489-43ba-b44c-c656a25b5225\" (UID: \"8e94e56c-d489-43ba-b44c-c656a25b5225\") "
Sep 10 21:22:51.929513 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:51.929391    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jv54p\" (UniqueName: \"kubernetes.io/projected/8e94e56c-d489-43ba-b44c-c656a25b5225-kube-api-access-jv54p\") pod \"8e94e56c-d489-43ba-b44c-c656a25b5225\" (UID: \"8e94e56c-d489-43ba-b44c-c656a25b5225\") "
Sep 10 21:22:51.929634 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:51.929537    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e94e56c-d489-43ba-b44c-c656a25b5225-model-cache" (OuterVolumeSpecName: "model-cache") pod "8e94e56c-d489-43ba-b44c-c656a25b5225" (UID: "8e94e56c-d489-43ba-b44c-c656a25b5225"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:22:51.929634 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:51.929555    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e94e56c-d489-43ba-b44c-c656a25b5225-home" (OuterVolumeSpecName: "home") pod "8e94e56c-d489-43ba-b44c-c656a25b5225" (UID: "8e94e56c-d489-43ba-b44c-c656a25b5225"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:22:51.929739 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:51.929628    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e94e56c-d489-43ba-b44c-c656a25b5225-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8e94e56c-d489-43ba-b44c-c656a25b5225" (UID: "8e94e56c-d489-43ba-b44c-c656a25b5225"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:22:51.929818 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:51.929798    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e94e56c-d489-43ba-b44c-c656a25b5225-model-cache\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:22:51.929877 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:51.929820    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e94e56c-d489-43ba-b44c-c656a25b5225-home\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:22:51.929877 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:51.929830    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e94e56c-d489-43ba-b44c-c656a25b5225-tmp-dir\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:22:51.931294 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:51.931273    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e94e56c-d489-43ba-b44c-c656a25b5225-dshm" (OuterVolumeSpecName: "dshm") pod "8e94e56c-d489-43ba-b44c-c656a25b5225" (UID: "8e94e56c-d489-43ba-b44c-c656a25b5225"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:22:51.931503 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:51.931483    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e94e56c-d489-43ba-b44c-c656a25b5225-kube-api-access-jv54p" (OuterVolumeSpecName: "kube-api-access-jv54p") pod "8e94e56c-d489-43ba-b44c-c656a25b5225" (UID: "8e94e56c-d489-43ba-b44c-c656a25b5225"). InnerVolumeSpecName "kube-api-access-jv54p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:22:51.931553 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:51.931506    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e94e56c-d489-43ba-b44c-c656a25b5225-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e94e56c-d489-43ba-b44c-c656a25b5225" (UID: "8e94e56c-d489-43ba-b44c-c656a25b5225"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:22:52.031283 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.031245    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e94e56c-d489-43ba-b44c-c656a25b5225-tls-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:22:52.031283 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.031280    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv54p\" (UniqueName: \"kubernetes.io/projected/8e94e56c-d489-43ba-b44c-c656a25b5225-kube-api-access-jv54p\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:22:52.031466 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.031295    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e94e56c-d489-43ba-b44c-c656a25b5225-dshm\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:22:52.241503 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.241468    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"]
Sep 10 21:22:52.243807 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.243784    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0ac302/sub-test-model-0ac302-kserve-58f59b7d5c-w94dn"]
Sep 10 21:22:52.275848 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.275813    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8" podUID="c9c40364-f7af-4050-906d-718f584a8cb9" containerName="main" containerID="cri-o://f90e6cf97a5717f0d9d801dc5c361dd1b1abb4acaa3c60e814ffdd78dd1f4b9c" gracePeriod=45
Sep 10 21:22:52.506468 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.506445    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:22:52.638743 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.638713    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c40364-f7af-4050-906d-718f584a8cb9-model-cache\") pod \"c9c40364-f7af-4050-906d-718f584a8cb9\" (UID: \"c9c40364-f7af-4050-906d-718f584a8cb9\") "
Sep 10 21:22:52.639162 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.638802    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6fvg\" (UniqueName: \"kubernetes.io/projected/c9c40364-f7af-4050-906d-718f584a8cb9-kube-api-access-h6fvg\") pod \"c9c40364-f7af-4050-906d-718f584a8cb9\" (UID: \"c9c40364-f7af-4050-906d-718f584a8cb9\") "
Sep 10 21:22:52.639162 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.638853    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c40364-f7af-4050-906d-718f584a8cb9-tls-certs\") pod \"c9c40364-f7af-4050-906d-718f584a8cb9\" (UID: \"c9c40364-f7af-4050-906d-718f584a8cb9\") "
Sep 10 21:22:52.639162 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.638902    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9c40364-f7af-4050-906d-718f584a8cb9-dshm\") pod \"c9c40364-f7af-4050-906d-718f584a8cb9\" (UID: \"c9c40364-f7af-4050-906d-718f584a8cb9\") "
Sep 10 21:22:52.639162 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.638942    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9c40364-f7af-4050-906d-718f584a8cb9-home\") pod \"c9c40364-f7af-4050-906d-718f584a8cb9\" (UID: \"c9c40364-f7af-4050-906d-718f584a8cb9\") "
Sep 10 21:22:52.639162 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.638987    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9c40364-f7af-4050-906d-718f584a8cb9-tmp-dir\") pod \"c9c40364-f7af-4050-906d-718f584a8cb9\" (UID: \"c9c40364-f7af-4050-906d-718f584a8cb9\") "
Sep 10 21:22:52.639162 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.639004    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c40364-f7af-4050-906d-718f584a8cb9-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9c40364-f7af-4050-906d-718f584a8cb9" (UID: "c9c40364-f7af-4050-906d-718f584a8cb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:22:52.639494 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.639228    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c40364-f7af-4050-906d-718f584a8cb9-home" (OuterVolumeSpecName: "home") pod "c9c40364-f7af-4050-906d-718f584a8cb9" (UID: "c9c40364-f7af-4050-906d-718f584a8cb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:22:52.639494 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.639246    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c40364-f7af-4050-906d-718f584a8cb9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c9c40364-f7af-4050-906d-718f584a8cb9" (UID: "c9c40364-f7af-4050-906d-718f584a8cb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:22:52.639494 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.639416    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c40364-f7af-4050-906d-718f584a8cb9-model-cache\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:22:52.639494 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.639455    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9c40364-f7af-4050-906d-718f584a8cb9-home\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:22:52.639494 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.639464    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9c40364-f7af-4050-906d-718f584a8cb9-tmp-dir\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:22:52.641127 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.641096    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c40364-f7af-4050-906d-718f584a8cb9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9c40364-f7af-4050-906d-718f584a8cb9" (UID: "c9c40364-f7af-4050-906d-718f584a8cb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:22:52.641227 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.641107    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c40364-f7af-4050-906d-718f584a8cb9-dshm" (OuterVolumeSpecName: "dshm") pod "c9c40364-f7af-4050-906d-718f584a8cb9" (UID: "c9c40364-f7af-4050-906d-718f584a8cb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:22:52.641227 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.641150    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c40364-f7af-4050-906d-718f584a8cb9-kube-api-access-h6fvg" (OuterVolumeSpecName: "kube-api-access-h6fvg") pod "c9c40364-f7af-4050-906d-718f584a8cb9" (UID: "c9c40364-f7af-4050-906d-718f584a8cb9"). InnerVolumeSpecName "kube-api-access-h6fvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:22:52.740189 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.740151    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6fvg\" (UniqueName: \"kubernetes.io/projected/c9c40364-f7af-4050-906d-718f584a8cb9-kube-api-access-h6fvg\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:22:52.740189 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.740188    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c40364-f7af-4050-906d-718f584a8cb9-tls-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:22:52.740372 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.740197    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9c40364-f7af-4050-906d-718f584a8cb9-dshm\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:22:52.840738 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.840655    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e94e56c-d489-43ba-b44c-c656a25b5225" path="/var/lib/kubelet/pods/8e94e56c-d489-43ba-b44c-c656a25b5225/volumes"
Sep 10 21:22:52.923293 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.923260    2584 generic.go:356] "Generic (PLEG): container finished" podID="c9c40364-f7af-4050-906d-718f584a8cb9" containerID="f90e6cf97a5717f0d9d801dc5c361dd1b1abb4acaa3c60e814ffdd78dd1f4b9c" exitCode=0
Sep 10 21:22:52.923293 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.923296    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8" event={"ID":"c9c40364-f7af-4050-906d-718f584a8cb9","Type":"ContainerDied","Data":"f90e6cf97a5717f0d9d801dc5c361dd1b1abb4acaa3c60e814ffdd78dd1f4b9c"}
Sep 10 21:22:52.923562 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.923306    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"
Sep 10 21:22:52.923562 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.923322    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8" event={"ID":"c9c40364-f7af-4050-906d-718f584a8cb9","Type":"ContainerDied","Data":"98556a59b44cec2fa0d8d89586c6c1a59cdeeb05b22dd45aab6ae60bf8e6ed65"}
Sep 10 21:22:52.923562 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.923342    2584 scope.go:117] "RemoveContainer" containerID="f90e6cf97a5717f0d9d801dc5c361dd1b1abb4acaa3c60e814ffdd78dd1f4b9c"
Sep 10 21:22:52.931925 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.931905    2584 scope.go:117] "RemoveContainer" containerID="f90e6cf97a5717f0d9d801dc5c361dd1b1abb4acaa3c60e814ffdd78dd1f4b9c"
Sep 10 21:22:52.932184 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:22:52.932165    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f90e6cf97a5717f0d9d801dc5c361dd1b1abb4acaa3c60e814ffdd78dd1f4b9c\": container with ID starting with f90e6cf97a5717f0d9d801dc5c361dd1b1abb4acaa3c60e814ffdd78dd1f4b9c not found: ID does not exist" containerID="f90e6cf97a5717f0d9d801dc5c361dd1b1abb4acaa3c60e814ffdd78dd1f4b9c"
Sep 10 21:22:52.932257 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.932196    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f90e6cf97a5717f0d9d801dc5c361dd1b1abb4acaa3c60e814ffdd78dd1f4b9c"} err="failed to get container status \"f90e6cf97a5717f0d9d801dc5c361dd1b1abb4acaa3c60e814ffdd78dd1f4b9c\": rpc error: code = NotFound desc = could not find container \"f90e6cf97a5717f0d9d801dc5c361dd1b1abb4acaa3c60e814ffdd78dd1f4b9c\": container with ID starting with f90e6cf97a5717f0d9d801dc5c361dd1b1abb4acaa3c60e814ffdd78dd1f4b9c not found: ID does not exist"
Sep 10 21:22:52.942181 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.942158    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"]
Sep 10 21:22:52.946737 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:52.946709    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-16bbec/sub-test-model-16bbec-kserve-554fd8d7b9-sljs8"]
Sep 10 21:22:54.846521 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:54.846489    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9c40364-f7af-4050-906d-718f584a8cb9" path="/var/lib/kubelet/pods/c9c40364-f7af-4050-906d-718f584a8cb9/volumes"
Sep 10 21:22:57.286520 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:57.286483    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:22:57.286520 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:57.286530    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:22:57.299163 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:57.299140    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:22:57.962085 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:22:57.962055    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:23:01.037847 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:01.037815    2584 scope.go:117] "RemoveContainer" containerID="c3eb7ede8cedb98a7f2c8d8902077eba7e33c38c4c222aeb5f6ec5fa7c500900"
Sep 10 21:23:01.943126 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:01.943085    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a249a2-8bqqz"]
Sep 10 21:23:01.956237 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:01.956208    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a249a2-8bqqz"]
Sep 10 21:23:02.840257 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:02.840217    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa5e0906-ce06-45a1-bf08-4bd4005d4779" path="/var/lib/kubelet/pods/fa5e0906-ce06-45a1-bf08-4bd4005d4779/volumes"
Sep 10 21:23:05.771900 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.771860    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"]
Sep 10 21:23:05.772533 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.772511    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9c40364-f7af-4050-906d-718f584a8cb9" containerName="main"
Sep 10 21:23:05.772533 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.772535    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9c40364-f7af-4050-906d-718f584a8cb9" containerName="main"
Sep 10 21:23:05.772708 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.772563    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e94e56c-d489-43ba-b44c-c656a25b5225" containerName="main"
Sep 10 21:23:05.772708 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.772571    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e94e56c-d489-43ba-b44c-c656a25b5225" containerName="main"
Sep 10 21:23:05.772708 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.772581    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa5e0906-ce06-45a1-bf08-4bd4005d4779" containerName="revoke-keys"
Sep 10 21:23:05.772708 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.772590    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa5e0906-ce06-45a1-bf08-4bd4005d4779" containerName="revoke-keys"
Sep 10 21:23:05.772708 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.772706    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa5e0906-ce06-45a1-bf08-4bd4005d4779" containerName="revoke-keys"
Sep 10 21:23:05.772950 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.772720    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9c40364-f7af-4050-906d-718f584a8cb9" containerName="main"
Sep 10 21:23:05.772950 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.772730    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e94e56c-d489-43ba-b44c-c656a25b5225" containerName="main"
Sep 10 21:23:05.776262 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.776226    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:23:05.779469 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.779439    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-16bbec\"/\"default-dockercfg-nwr5g\""
Sep 10 21:23:05.779469 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.779455    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-16bbec\"/\"rate-test-model-16bbec-kserve-self-signed-certs\""
Sep 10 21:23:05.779645 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.779469    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-16bbec\"/\"openshift-service-ca.crt\""
Sep 10 21:23:05.779645 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.779439    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-16bbec\"/\"kube-root-ca.crt\""
Sep 10 21:23:05.784742 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.784713    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"]
Sep 10 21:23:05.873288 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.873247    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4455ad-0996-4f4c-923f-29a8fa595140-tmp-dir\") pod \"rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l\" (UID: \"7e4455ad-0996-4f4c-923f-29a8fa595140\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:23:05.873487 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.873358    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e4455ad-0996-4f4c-923f-29a8fa595140-model-cache\") pod \"rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l\" (UID: \"7e4455ad-0996-4f4c-923f-29a8fa595140\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:23:05.873487 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.873379    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e4455ad-0996-4f4c-923f-29a8fa595140-tls-certs\") pod \"rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l\" (UID: \"7e4455ad-0996-4f4c-923f-29a8fa595140\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:23:05.873487 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.873402    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtpjq\" (UniqueName: \"kubernetes.io/projected/7e4455ad-0996-4f4c-923f-29a8fa595140-kube-api-access-mtpjq\") pod \"rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l\" (UID: \"7e4455ad-0996-4f4c-923f-29a8fa595140\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:23:05.873610 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.873506    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e4455ad-0996-4f4c-923f-29a8fa595140-home\") pod \"rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l\" (UID: \"7e4455ad-0996-4f4c-923f-29a8fa595140\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:23:05.873610 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.873559    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e4455ad-0996-4f4c-923f-29a8fa595140-dshm\") pod \"rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l\" (UID: \"7e4455ad-0996-4f4c-923f-29a8fa595140\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:23:05.975017 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.974973    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e4455ad-0996-4f4c-923f-29a8fa595140-model-cache\") pod \"rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l\" (UID: \"7e4455ad-0996-4f4c-923f-29a8fa595140\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:23:05.975017 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.975016    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e4455ad-0996-4f4c-923f-29a8fa595140-tls-certs\") pod \"rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l\" (UID: \"7e4455ad-0996-4f4c-923f-29a8fa595140\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:23:05.975269 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.975044    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtpjq\" (UniqueName: \"kubernetes.io/projected/7e4455ad-0996-4f4c-923f-29a8fa595140-kube-api-access-mtpjq\") pod \"rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l\" (UID: \"7e4455ad-0996-4f4c-923f-29a8fa595140\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:23:05.975269 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.975089    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e4455ad-0996-4f4c-923f-29a8fa595140-home\") pod \"rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l\" (UID: \"7e4455ad-0996-4f4c-923f-29a8fa595140\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:23:05.975269 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.975120    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e4455ad-0996-4f4c-923f-29a8fa595140-dshm\") pod \"rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l\" (UID: \"7e4455ad-0996-4f4c-923f-29a8fa595140\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:23:05.975269 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.975147    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4455ad-0996-4f4c-923f-29a8fa595140-tmp-dir\") pod \"rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l\" (UID: \"7e4455ad-0996-4f4c-923f-29a8fa595140\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:23:05.975627 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.975479    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e4455ad-0996-4f4c-923f-29a8fa595140-model-cache\") pod \"rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l\" (UID: \"7e4455ad-0996-4f4c-923f-29a8fa595140\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:23:05.975627 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.975562    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e4455ad-0996-4f4c-923f-29a8fa595140-home\") pod \"rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l\" (UID: \"7e4455ad-0996-4f4c-923f-29a8fa595140\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:23:05.975726 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.975628    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4455ad-0996-4f4c-923f-29a8fa595140-tmp-dir\") pod \"rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l\" (UID: \"7e4455ad-0996-4f4c-923f-29a8fa595140\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:23:05.977493 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.977472    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e4455ad-0996-4f4c-923f-29a8fa595140-dshm\") pod \"rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l\" (UID: \"7e4455ad-0996-4f4c-923f-29a8fa595140\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:23:05.977784 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.977767    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e4455ad-0996-4f4c-923f-29a8fa595140-tls-certs\") pod \"rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l\" (UID: \"7e4455ad-0996-4f4c-923f-29a8fa595140\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:23:05.984196 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:05.984172    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtpjq\" (UniqueName: \"kubernetes.io/projected/7e4455ad-0996-4f4c-923f-29a8fa595140-kube-api-access-mtpjq\") pod \"rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l\" (UID: \"7e4455ad-0996-4f4c-923f-29a8fa595140\") " pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:23:06.086244 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:06.086147    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:23:06.359314 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:06.359285    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"]
Sep 10 21:23:06.360779 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:23:06.360752    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e4455ad_0996_4f4c_923f_29a8fa595140.slice/crio-fe047e699c3ccf4fb1cee669d17b4441ee1488a937f2baad61f7c30b1cada62b WatchSource:0}: Error finding container fe047e699c3ccf4fb1cee669d17b4441ee1488a937f2baad61f7c30b1cada62b: Status 404 returned error can't find the container with id fe047e699c3ccf4fb1cee669d17b4441ee1488a937f2baad61f7c30b1cada62b
Sep 10 21:23:06.996030 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:06.995989    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l" event={"ID":"7e4455ad-0996-4f4c-923f-29a8fa595140","Type":"ContainerStarted","Data":"39c057b695fa6be3b81202f7953b13d8427e257785a3ba27ce2a80caaa974600"}
Sep 10 21:23:06.996595 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:06.996045    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l" event={"ID":"7e4455ad-0996-4f4c-923f-29a8fa595140","Type":"ContainerStarted","Data":"fe047e699c3ccf4fb1cee669d17b4441ee1488a937f2baad61f7c30b1cada62b"}
Sep 10 21:23:07.014732 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:07.014685    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l" podStartSLOduration=2.014670887 podStartE2EDuration="2.014670887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:23:07.013539996 +0000 UTC m=+1446.678564796" watchObservedRunningTime="2026-09-10 21:23:07.014670887 +0000 UTC m=+1446.679695686"
Sep 10 21:23:08.905523 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:08.905488    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"]
Sep 10 21:23:08.905894 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:08.905744    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7" podUID="bd12952d-7efc-450a-a457-bc039f995340" containerName="istio-proxy" containerID="cri-o://15a8131ac467fdf56459bd4f6bc623a92b08448b52ac399ad7973e98650325ee" gracePeriod=30
Sep 10 21:23:09.291600 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:09.291513    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf7171-8r5jt"]
Sep 10 21:23:09.294117 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:09.294095    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf7171-8r5jt"
Sep 10 21:23:09.296476 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:09.296446    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4gs5\""
Sep 10 21:23:09.296578 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:09.296506    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:23:09.304379 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:09.304353    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf7171-8r5jt"]
Sep 10 21:23:09.411827 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:09.411789    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d39479b-ae3e-4e65-92d6-53bdd72e229b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cf7171-8r5jt\" (UID: \"1d39479b-ae3e-4e65-92d6-53bdd72e229b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf7171-8r5jt"
Sep 10 21:23:09.513209 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:09.513177    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d39479b-ae3e-4e65-92d6-53bdd72e229b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cf7171-8r5jt\" (UID: \"1d39479b-ae3e-4e65-92d6-53bdd72e229b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf7171-8r5jt"
Sep 10 21:23:09.513831 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:09.513812    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d39479b-ae3e-4e65-92d6-53bdd72e229b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cf7171-8r5jt\" (UID: \"1d39479b-ae3e-4e65-92d6-53bdd72e229b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf7171-8r5jt"
Sep 10 21:23:09.605389 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:09.605291    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf7171-8r5jt"
Sep 10 21:23:09.876442 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:09.876395    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf7171-8r5jt"]
Sep 10 21:23:09.878181 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:23:09.878150    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d39479b_ae3e_4e65_92d6_53bdd72e229b.slice/crio-2298a6d1e1bbd8a4737d3ec4e57495c5900f1affcdf86a97ee3b11157f01cf5e WatchSource:0}: Error finding container 2298a6d1e1bbd8a4737d3ec4e57495c5900f1affcdf86a97ee3b11157f01cf5e: Status 404 returned error can't find the container with id 2298a6d1e1bbd8a4737d3ec4e57495c5900f1affcdf86a97ee3b11157f01cf5e
Sep 10 21:23:10.014258 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:10.014215    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf7171-8r5jt" event={"ID":"1d39479b-ae3e-4e65-92d6-53bdd72e229b","Type":"ContainerStarted","Data":"2298a6d1e1bbd8a4737d3ec4e57495c5900f1affcdf86a97ee3b11157f01cf5e"}
Sep 10 21:23:11.021127 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:11.021088    2584 generic.go:356] "Generic (PLEG): container finished" podID="1d39479b-ae3e-4e65-92d6-53bdd72e229b" containerID="8e6f1bca6fb5c690e4fcaf32eb67e0b52affe38cb1b6c18ec9549393a0e8cbed" exitCode=0
Sep 10 21:23:11.021521 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:11.021173    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf7171-8r5jt" event={"ID":"1d39479b-ae3e-4e65-92d6-53bdd72e229b","Type":"ContainerDied","Data":"8e6f1bca6fb5c690e4fcaf32eb67e0b52affe38cb1b6c18ec9549393a0e8cbed"}
Sep 10 21:23:12.155177 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:12.155150    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf7171-8r5jt"
Sep 10 21:23:12.241933 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:12.241900    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d39479b-ae3e-4e65-92d6-53bdd72e229b-maas-api-service-ca\") pod \"1d39479b-ae3e-4e65-92d6-53bdd72e229b\" (UID: \"1d39479b-ae3e-4e65-92d6-53bdd72e229b\") "
Sep 10 21:23:12.242302 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:12.242275    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d39479b-ae3e-4e65-92d6-53bdd72e229b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1d39479b-ae3e-4e65-92d6-53bdd72e229b" (UID: "1d39479b-ae3e-4e65-92d6-53bdd72e229b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:23:12.343660 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:12.343579    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d39479b-ae3e-4e65-92d6-53bdd72e229b-maas-api-service-ca\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:23:12.995046 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:12.995009    2584 patch_prober.go:28] interesting pod/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 21:23:12.995194 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:12.995065    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7" podUID="bd12952d-7efc-450a-a457-bc039f995340" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 21:23:13.033773 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:13.033738    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf7171-8r5jt" event={"ID":"1d39479b-ae3e-4e65-92d6-53bdd72e229b","Type":"ContainerDied","Data":"2298a6d1e1bbd8a4737d3ec4e57495c5900f1affcdf86a97ee3b11157f01cf5e"}
Sep 10 21:23:13.033773 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:13.033771    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf7171-8r5jt"
Sep 10 21:23:13.033984 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:13.033789    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2298a6d1e1bbd8a4737d3ec4e57495c5900f1affcdf86a97ee3b11157f01cf5e"
Sep 10 21:23:14.041861 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.041827    2584 generic.go:356] "Generic (PLEG): container finished" podID="bd12952d-7efc-450a-a457-bc039f995340" containerID="15a8131ac467fdf56459bd4f6bc623a92b08448b52ac399ad7973e98650325ee" exitCode=0
Sep 10 21:23:14.042250 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.041905    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7" event={"ID":"bd12952d-7efc-450a-a457-bc039f995340","Type":"ContainerDied","Data":"15a8131ac467fdf56459bd4f6bc623a92b08448b52ac399ad7973e98650325ee"}
Sep 10 21:23:14.042250 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.041995    2584 scope.go:117] "RemoveContainer" containerID="531773008d0886620082702789773fb3b4660d38ba031c5f09b3d3f6ea68c80c"
Sep 10 21:23:14.148969 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.148941    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:23:14.264682 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.264595    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-istio-data\") pod \"bd12952d-7efc-450a-a457-bc039f995340\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") "
Sep 10 21:23:14.264835 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.264724    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd12952d-7efc-450a-a457-bc039f995340-istiod-ca-cert\") pod \"bd12952d-7efc-450a-a457-bc039f995340\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") "
Sep 10 21:23:14.264835 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.264786    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd12952d-7efc-450a-a457-bc039f995340-istio-podinfo\") pod \"bd12952d-7efc-450a-a457-bc039f995340\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") "
Sep 10 21:23:14.264835 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.264822    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-workload-certs\") pod \"bd12952d-7efc-450a-a457-bc039f995340\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") "
Sep 10 21:23:14.264975 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.264863    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxnfc\" (UniqueName: \"kubernetes.io/projected/bd12952d-7efc-450a-a457-bc039f995340-kube-api-access-nxnfc\") pod \"bd12952d-7efc-450a-a457-bc039f995340\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") "
Sep 10 21:23:14.264975 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.264889    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-credential-socket\") pod \"bd12952d-7efc-450a-a457-bc039f995340\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") "
Sep 10 21:23:14.264975 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.264898    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-istio-data" (OuterVolumeSpecName: "istio-data") pod "bd12952d-7efc-450a-a457-bc039f995340" (UID: "bd12952d-7efc-450a-a457-bc039f995340"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:23:14.264975 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.264948    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-istio-envoy\") pod \"bd12952d-7efc-450a-a457-bc039f995340\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") "
Sep 10 21:23:14.265181 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.264986    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd12952d-7efc-450a-a457-bc039f995340-istio-token\") pod \"bd12952d-7efc-450a-a457-bc039f995340\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") "
Sep 10 21:23:14.265181 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.265024    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-workload-socket\") pod \"bd12952d-7efc-450a-a457-bc039f995340\" (UID: \"bd12952d-7efc-450a-a457-bc039f995340\") "
Sep 10 21:23:14.265297 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.265170    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bd12952d-7efc-450a-a457-bc039f995340" (UID: "bd12952d-7efc-450a-a457-bc039f995340"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:23:14.265297 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.265206    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bd12952d-7efc-450a-a457-bc039f995340" (UID: "bd12952d-7efc-450a-a457-bc039f995340"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:23:14.265414 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.265382    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bd12952d-7efc-450a-a457-bc039f995340" (UID: "bd12952d-7efc-450a-a457-bc039f995340"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:23:14.265556 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.265533    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd12952d-7efc-450a-a457-bc039f995340-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bd12952d-7efc-450a-a457-bc039f995340" (UID: "bd12952d-7efc-450a-a457-bc039f995340"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:23:14.265624 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.265594    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-workload-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:23:14.265624 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.265615    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-credential-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:23:14.265735 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.265628    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-workload-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:23:14.265735 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.265642    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-istio-data\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:23:14.267186 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.267152    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd12952d-7efc-450a-a457-bc039f995340-kube-api-access-nxnfc" (OuterVolumeSpecName: "kube-api-access-nxnfc") pod "bd12952d-7efc-450a-a457-bc039f995340" (UID: "bd12952d-7efc-450a-a457-bc039f995340"). InnerVolumeSpecName "kube-api-access-nxnfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:23:14.267301 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.267204    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd12952d-7efc-450a-a457-bc039f995340-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bd12952d-7efc-450a-a457-bc039f995340" (UID: "bd12952d-7efc-450a-a457-bc039f995340"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:23:14.267301 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.267224    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd12952d-7efc-450a-a457-bc039f995340-istio-token" (OuterVolumeSpecName: "istio-token") pod "bd12952d-7efc-450a-a457-bc039f995340" (UID: "bd12952d-7efc-450a-a457-bc039f995340"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:23:14.267301 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.267229    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bd12952d-7efc-450a-a457-bc039f995340" (UID: "bd12952d-7efc-450a-a457-bc039f995340"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:23:14.366323 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.366278    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd12952d-7efc-450a-a457-bc039f995340-istio-envoy\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:23:14.366323 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.366315    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd12952d-7efc-450a-a457-bc039f995340-istio-token\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:23:14.366323 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.366324    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd12952d-7efc-450a-a457-bc039f995340-istiod-ca-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:23:14.366323 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.366332    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd12952d-7efc-450a-a457-bc039f995340-istio-podinfo\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:23:14.366323 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:14.366340    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxnfc\" (UniqueName: \"kubernetes.io/projected/bd12952d-7efc-450a-a457-bc039f995340-kube-api-access-nxnfc\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:23:15.047876 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:15.047842    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"
Sep 10 21:23:15.048279 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:15.047888    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7" event={"ID":"bd12952d-7efc-450a-a457-bc039f995340","Type":"ContainerDied","Data":"b7d3ce0909e5e59f39fdd8b791ad60af644046f9afbb40a819972010cd3e830b"}
Sep 10 21:23:15.048279 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:15.047949    2584 scope.go:117] "RemoveContainer" containerID="15a8131ac467fdf56459bd4f6bc623a92b08448b52ac399ad7973e98650325ee"
Sep 10 21:23:15.070362 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:15.070332    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"]
Sep 10 21:23:15.077039 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:15.077011    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a249a2-openshift-default-7cb8d5bf98-xj9n7"]
Sep 10 21:23:16.086945 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:16.086912    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:23:16.086945 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:16.086954    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:23:16.099580 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:16.099551    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:23:16.840488 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:16.840451    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd12952d-7efc-450a-a457-bc039f995340" path="/var/lib/kubelet/pods/bd12952d-7efc-450a-a457-bc039f995340/volumes"
Sep 10 21:23:17.069535 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:17.069507    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:23:25.300573 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:25.300534    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf7171-8r5jt"]
Sep 10 21:23:25.307951 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:25.307915    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf7171-8r5jt"]
Sep 10 21:23:26.839976 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:26.839938    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d39479b-ae3e-4e65-92d6-53bdd72e229b" path="/var/lib/kubelet/pods/1d39479b-ae3e-4e65-92d6-53bdd72e229b/volumes"
Sep 10 21:23:32.435272 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:32.435236    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"]
Sep 10 21:23:32.435776 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:32.435552    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2" podUID="4d9d1849-4946-4693-910a-eecce8049eee" containerName="istio-proxy" containerID="cri-o://c45cf181d08d14bbe1cb49b2c290a1b029fa94c43e583f8713dea7a7008293dc" gracePeriod=30
Sep 10 21:23:32.807056 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:32.806972    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-501c57-jl7fn"]
Sep 10 21:23:32.807343 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:32.807331    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd12952d-7efc-450a-a457-bc039f995340" containerName="istio-proxy"
Sep 10 21:23:32.807388 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:32.807345    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd12952d-7efc-450a-a457-bc039f995340" containerName="istio-proxy"
Sep 10 21:23:32.807388 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:32.807364    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd12952d-7efc-450a-a457-bc039f995340" containerName="istio-proxy"
Sep 10 21:23:32.807388 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:32.807371    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd12952d-7efc-450a-a457-bc039f995340" containerName="istio-proxy"
Sep 10 21:23:32.807388 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:32.807384    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d39479b-ae3e-4e65-92d6-53bdd72e229b" containerName="revoke-keys"
Sep 10 21:23:32.807575 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:32.807390    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d39479b-ae3e-4e65-92d6-53bdd72e229b" containerName="revoke-keys"
Sep 10 21:23:32.807575 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:32.807466    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d39479b-ae3e-4e65-92d6-53bdd72e229b" containerName="revoke-keys"
Sep 10 21:23:32.807575 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:32.807477    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd12952d-7efc-450a-a457-bc039f995340" containerName="istio-proxy"
Sep 10 21:23:32.810496 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:32.810478    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-501c57-jl7fn"
Sep 10 21:23:32.812757 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:32.812733    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4gs5\""
Sep 10 21:23:32.812890 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:32.812796    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:23:32.819104 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:32.819085    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-501c57-jl7fn"]
Sep 10 21:23:32.855212 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:32.855178    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea5421d-c9bb-493e-adf8-f45005ffd953-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-501c57-jl7fn\" (UID: \"cea5421d-c9bb-493e-adf8-f45005ffd953\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-501c57-jl7fn"
Sep 10 21:23:32.955937 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:32.955891    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea5421d-c9bb-493e-adf8-f45005ffd953-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-501c57-jl7fn\" (UID: \"cea5421d-c9bb-493e-adf8-f45005ffd953\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-501c57-jl7fn"
Sep 10 21:23:32.956629 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:32.956603    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea5421d-c9bb-493e-adf8-f45005ffd953-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-501c57-jl7fn\" (UID: \"cea5421d-c9bb-493e-adf8-f45005ffd953\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-501c57-jl7fn"
Sep 10 21:23:33.121971 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:33.121943    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-501c57-jl7fn"
Sep 10 21:23:33.398632 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:33.398607    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-501c57-jl7fn"]
Sep 10 21:23:33.400789 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:23:33.400764    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea5421d_c9bb_493e_adf8_f45005ffd953.slice/crio-87fa38c73e0272313b32119785ef983c72c3447cef96b8da2c6cac24d7fe4300 WatchSource:0}: Error finding container 87fa38c73e0272313b32119785ef983c72c3447cef96b8da2c6cac24d7fe4300: Status 404 returned error can't find the container with id 87fa38c73e0272313b32119785ef983c72c3447cef96b8da2c6cac24d7fe4300
Sep 10 21:23:34.151473 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:34.151440    2584 generic.go:356] "Generic (PLEG): container finished" podID="cea5421d-c9bb-493e-adf8-f45005ffd953" containerID="34ab67e02b67d9a7750857f6723c41985b93072819c9a625cfd1e782e9575852" exitCode=0
Sep 10 21:23:34.151840 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:34.151476    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-501c57-jl7fn" event={"ID":"cea5421d-c9bb-493e-adf8-f45005ffd953","Type":"ContainerDied","Data":"34ab67e02b67d9a7750857f6723c41985b93072819c9a625cfd1e782e9575852"}
Sep 10 21:23:34.151840 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:34.151544    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-501c57-jl7fn" event={"ID":"cea5421d-c9bb-493e-adf8-f45005ffd953","Type":"ContainerStarted","Data":"87fa38c73e0272313b32119785ef983c72c3447cef96b8da2c6cac24d7fe4300"}
Sep 10 21:23:35.285861 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:35.285839    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-501c57-jl7fn"
Sep 10 21:23:35.381466 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:35.381399    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea5421d-c9bb-493e-adf8-f45005ffd953-maas-api-service-ca\") pod \"cea5421d-c9bb-493e-adf8-f45005ffd953\" (UID: \"cea5421d-c9bb-493e-adf8-f45005ffd953\") "
Sep 10 21:23:35.381746 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:35.381721    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cea5421d-c9bb-493e-adf8-f45005ffd953-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cea5421d-c9bb-493e-adf8-f45005ffd953" (UID: "cea5421d-c9bb-493e-adf8-f45005ffd953"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:23:35.482871 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:35.482777    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea5421d-c9bb-493e-adf8-f45005ffd953-maas-api-service-ca\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:23:36.162565 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:36.162527    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-501c57-jl7fn" event={"ID":"cea5421d-c9bb-493e-adf8-f45005ffd953","Type":"ContainerDied","Data":"87fa38c73e0272313b32119785ef983c72c3447cef96b8da2c6cac24d7fe4300"}
Sep 10 21:23:36.162745 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:36.162583    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87fa38c73e0272313b32119785ef983c72c3447cef96b8da2c6cac24d7fe4300"
Sep 10 21:23:36.162745 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:36.162545    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-501c57-jl7fn"
Sep 10 21:23:37.686494 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.686470    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:23:37.805260 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.805164    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d9d1849-4946-4693-910a-eecce8049eee-istio-token\") pod \"4d9d1849-4946-4693-910a-eecce8049eee\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") "
Sep 10 21:23:37.805260 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.805231    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d9d1849-4946-4693-910a-eecce8049eee-istiod-ca-cert\") pod \"4d9d1849-4946-4693-910a-eecce8049eee\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") "
Sep 10 21:23:37.805641 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.805260    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d9d1849-4946-4693-910a-eecce8049eee-istio-podinfo\") pod \"4d9d1849-4946-4693-910a-eecce8049eee\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") "
Sep 10 21:23:37.805641 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.805327    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-istio-data\") pod \"4d9d1849-4946-4693-910a-eecce8049eee\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") "
Sep 10 21:23:37.805641 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.805380    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-credential-socket\") pod \"4d9d1849-4946-4693-910a-eecce8049eee\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") "
Sep 10 21:23:37.805641 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.805408    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-workload-certs\") pod \"4d9d1849-4946-4693-910a-eecce8049eee\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") "
Sep 10 21:23:37.805641 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.805489    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j487p\" (UniqueName: \"kubernetes.io/projected/4d9d1849-4946-4693-910a-eecce8049eee-kube-api-access-j487p\") pod \"4d9d1849-4946-4693-910a-eecce8049eee\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") "
Sep 10 21:23:37.805641 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.805550    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-istio-envoy\") pod \"4d9d1849-4946-4693-910a-eecce8049eee\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") "
Sep 10 21:23:37.805641 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.805617    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-workload-socket\") pod \"4d9d1849-4946-4693-910a-eecce8049eee\" (UID: \"4d9d1849-4946-4693-910a-eecce8049eee\") "
Sep 10 21:23:37.806068 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.805733    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-istio-data" (OuterVolumeSpecName: "istio-data") pod "4d9d1849-4946-4693-910a-eecce8049eee" (UID: "4d9d1849-4946-4693-910a-eecce8049eee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:23:37.806068 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.805767    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d9d1849-4946-4693-910a-eecce8049eee-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4d9d1849-4946-4693-910a-eecce8049eee" (UID: "4d9d1849-4946-4693-910a-eecce8049eee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:23:37.806068 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.805779    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4d9d1849-4946-4693-910a-eecce8049eee" (UID: "4d9d1849-4946-4693-910a-eecce8049eee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:23:37.806068 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.805953    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4d9d1849-4946-4693-910a-eecce8049eee" (UID: "4d9d1849-4946-4693-910a-eecce8049eee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:23:37.806068 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.805982    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4d9d1849-4946-4693-910a-eecce8049eee" (UID: "4d9d1849-4946-4693-910a-eecce8049eee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:23:37.806068 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.806003    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d9d1849-4946-4693-910a-eecce8049eee-istiod-ca-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:23:37.806068 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.806022    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-istio-data\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:23:37.806068 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.806035    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-workload-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:23:37.808147 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.808120    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9d1849-4946-4693-910a-eecce8049eee-kube-api-access-j487p" (OuterVolumeSpecName: "kube-api-access-j487p") pod "4d9d1849-4946-4693-910a-eecce8049eee" (UID: "4d9d1849-4946-4693-910a-eecce8049eee"). InnerVolumeSpecName "kube-api-access-j487p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:23:37.808147 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.808134    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d9d1849-4946-4693-910a-eecce8049eee-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4d9d1849-4946-4693-910a-eecce8049eee" (UID: "4d9d1849-4946-4693-910a-eecce8049eee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:23:37.808340 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.808253    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9d1849-4946-4693-910a-eecce8049eee-istio-token" (OuterVolumeSpecName: "istio-token") pod "4d9d1849-4946-4693-910a-eecce8049eee" (UID: "4d9d1849-4946-4693-910a-eecce8049eee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:23:37.808392 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.808369    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4d9d1849-4946-4693-910a-eecce8049eee" (UID: "4d9d1849-4946-4693-910a-eecce8049eee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:23:37.907131 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.907095    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-istio-envoy\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:23:37.907131 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.907130    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-workload-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:23:37.907382 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.907145    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d9d1849-4946-4693-910a-eecce8049eee-istio-token\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:23:37.907382 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.907161    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d9d1849-4946-4693-910a-eecce8049eee-istio-podinfo\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:23:37.907382 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.907175    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9d1849-4946-4693-910a-eecce8049eee-credential-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:23:37.907382 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:37.907190    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j487p\" (UniqueName: \"kubernetes.io/projected/4d9d1849-4946-4693-910a-eecce8049eee-kube-api-access-j487p\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:23:38.174608 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:38.174574    2584 generic.go:356] "Generic (PLEG): container finished" podID="4d9d1849-4946-4693-910a-eecce8049eee" containerID="c45cf181d08d14bbe1cb49b2c290a1b029fa94c43e583f8713dea7a7008293dc" exitCode=0
Sep 10 21:23:38.174776 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:38.174625    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"
Sep 10 21:23:38.174776 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:38.174664    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2" event={"ID":"4d9d1849-4946-4693-910a-eecce8049eee","Type":"ContainerDied","Data":"c45cf181d08d14bbe1cb49b2c290a1b029fa94c43e583f8713dea7a7008293dc"}
Sep 10 21:23:38.174776 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:38.174724    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2" event={"ID":"4d9d1849-4946-4693-910a-eecce8049eee","Type":"ContainerDied","Data":"dbf4827386fac7cc08ac06a70f96c541a712f01a4858437f00abab104911a8ff"}
Sep 10 21:23:38.174776 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:38.174756    2584 scope.go:117] "RemoveContainer" containerID="c45cf181d08d14bbe1cb49b2c290a1b029fa94c43e583f8713dea7a7008293dc"
Sep 10 21:23:38.184331 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:38.184314    2584 scope.go:117] "RemoveContainer" containerID="c22153a0eef611e0b66755fcc02040a47df67915b37e94b2235defd9cc74295d"
Sep 10 21:23:38.192195 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:38.192177    2584 scope.go:117] "RemoveContainer" containerID="c45cf181d08d14bbe1cb49b2c290a1b029fa94c43e583f8713dea7a7008293dc"
Sep 10 21:23:38.192467 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:23:38.192447    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c45cf181d08d14bbe1cb49b2c290a1b029fa94c43e583f8713dea7a7008293dc\": container with ID starting with c45cf181d08d14bbe1cb49b2c290a1b029fa94c43e583f8713dea7a7008293dc not found: ID does not exist" containerID="c45cf181d08d14bbe1cb49b2c290a1b029fa94c43e583f8713dea7a7008293dc"
Sep 10 21:23:38.192541 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:38.192480    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c45cf181d08d14bbe1cb49b2c290a1b029fa94c43e583f8713dea7a7008293dc"} err="failed to get container status \"c45cf181d08d14bbe1cb49b2c290a1b029fa94c43e583f8713dea7a7008293dc\": rpc error: code = NotFound desc = could not find container \"c45cf181d08d14bbe1cb49b2c290a1b029fa94c43e583f8713dea7a7008293dc\": container with ID starting with c45cf181d08d14bbe1cb49b2c290a1b029fa94c43e583f8713dea7a7008293dc not found: ID does not exist"
Sep 10 21:23:38.192541 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:38.192504    2584 scope.go:117] "RemoveContainer" containerID="c22153a0eef611e0b66755fcc02040a47df67915b37e94b2235defd9cc74295d"
Sep 10 21:23:38.192753 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:23:38.192737    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c22153a0eef611e0b66755fcc02040a47df67915b37e94b2235defd9cc74295d\": container with ID starting with c22153a0eef611e0b66755fcc02040a47df67915b37e94b2235defd9cc74295d not found: ID does not exist" containerID="c22153a0eef611e0b66755fcc02040a47df67915b37e94b2235defd9cc74295d"
Sep 10 21:23:38.192810 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:38.192761    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c22153a0eef611e0b66755fcc02040a47df67915b37e94b2235defd9cc74295d"} err="failed to get container status \"c22153a0eef611e0b66755fcc02040a47df67915b37e94b2235defd9cc74295d\": rpc error: code = NotFound desc = could not find container \"c22153a0eef611e0b66755fcc02040a47df67915b37e94b2235defd9cc74295d\": container with ID starting with c22153a0eef611e0b66755fcc02040a47df67915b37e94b2235defd9cc74295d not found: ID does not exist"
Sep 10 21:23:38.196786 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:38.196763    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"]
Sep 10 21:23:38.200368 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:38.200348    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-cf7171-openshift-default-6f84fd57d8-dr9w2"]
Sep 10 21:23:38.840142 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:38.840107    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d9d1849-4946-4693-910a-eecce8049eee" path="/var/lib/kubelet/pods/4d9d1849-4946-4693-910a-eecce8049eee/volumes"
Sep 10 21:23:43.594308 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:43.594271    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-501c57-jl7fn"]
Sep 10 21:23:43.597624 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:43.597595    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-501c57-jl7fn"]
Sep 10 21:23:44.840661 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:44.840584    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cea5421d-c9bb-493e-adf8-f45005ffd953" path="/var/lib/kubelet/pods/cea5421d-c9bb-493e-adf8-f45005ffd953/volumes"
Sep 10 21:23:50.493107 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:50.493071    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4ad710-xmp9f"]
Sep 10 21:23:50.493522 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:50.493466    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d9d1849-4946-4693-910a-eecce8049eee" containerName="istio-proxy"
Sep 10 21:23:50.493522 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:50.493483    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d9d1849-4946-4693-910a-eecce8049eee" containerName="istio-proxy"
Sep 10 21:23:50.493522 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:50.493498    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cea5421d-c9bb-493e-adf8-f45005ffd953" containerName="revoke-keys"
Sep 10 21:23:50.493522 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:50.493504    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="cea5421d-c9bb-493e-adf8-f45005ffd953" containerName="revoke-keys"
Sep 10 21:23:50.493522 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:50.493516    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d9d1849-4946-4693-910a-eecce8049eee" containerName="istio-proxy"
Sep 10 21:23:50.493522 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:50.493523    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d9d1849-4946-4693-910a-eecce8049eee" containerName="istio-proxy"
Sep 10 21:23:50.493813 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:50.493607    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd12952d-7efc-450a-a457-bc039f995340" containerName="istio-proxy"
Sep 10 21:23:50.493813 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:50.493615    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="cea5421d-c9bb-493e-adf8-f45005ffd953" containerName="revoke-keys"
Sep 10 21:23:50.493813 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:50.493624    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d9d1849-4946-4693-910a-eecce8049eee" containerName="istio-proxy"
Sep 10 21:23:50.493813 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:50.493630    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d9d1849-4946-4693-910a-eecce8049eee" containerName="istio-proxy"
Sep 10 21:23:50.498282 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:50.498266    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4ad710-xmp9f"
Sep 10 21:23:50.500604 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:50.500580    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:23:50.500746 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:50.500721    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4gs5\""
Sep 10 21:23:50.504953 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:50.504929    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4ad710-xmp9f"]
Sep 10 21:23:50.637574 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:50.637543    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8089512a-56eb-4fad-8ac4-8e79db806619-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4ad710-xmp9f\" (UID: \"8089512a-56eb-4fad-8ac4-8e79db806619\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4ad710-xmp9f"
Sep 10 21:23:50.738354 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:50.738300    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8089512a-56eb-4fad-8ac4-8e79db806619-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4ad710-xmp9f\" (UID: \"8089512a-56eb-4fad-8ac4-8e79db806619\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4ad710-xmp9f"
Sep 10 21:23:50.738924 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:50.738902    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8089512a-56eb-4fad-8ac4-8e79db806619-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4ad710-xmp9f\" (UID: \"8089512a-56eb-4fad-8ac4-8e79db806619\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4ad710-xmp9f"
Sep 10 21:23:50.810379 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:50.810305    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4ad710-xmp9f"
Sep 10 21:23:51.085472 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:51.085447    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4ad710-xmp9f"]
Sep 10 21:23:51.087982 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:23:51.087955    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8089512a_56eb_4fad_8ac4_8e79db806619.slice/crio-49778be5778fd0a0e98bcab41857d43e37c14c8a83ca5b9c1403d481d5a06ba8 WatchSource:0}: Error finding container 49778be5778fd0a0e98bcab41857d43e37c14c8a83ca5b9c1403d481d5a06ba8: Status 404 returned error can't find the container with id 49778be5778fd0a0e98bcab41857d43e37c14c8a83ca5b9c1403d481d5a06ba8
Sep 10 21:23:51.244500 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:51.244462    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4ad710-xmp9f" event={"ID":"8089512a-56eb-4fad-8ac4-8e79db806619","Type":"ContainerStarted","Data":"49778be5778fd0a0e98bcab41857d43e37c14c8a83ca5b9c1403d481d5a06ba8"}
Sep 10 21:23:52.250841 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:52.250807    2584 generic.go:356] "Generic (PLEG): container finished" podID="8089512a-56eb-4fad-8ac4-8e79db806619" containerID="7ffe7401e91913b21ad42e4788cff03d56f71a5cdf31942d3145b3153e051832" exitCode=0
Sep 10 21:23:52.251215 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:52.250870    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4ad710-xmp9f" event={"ID":"8089512a-56eb-4fad-8ac4-8e79db806619","Type":"ContainerDied","Data":"7ffe7401e91913b21ad42e4788cff03d56f71a5cdf31942d3145b3153e051832"}
Sep 10 21:23:53.379821 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:53.379798    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4ad710-xmp9f"
Sep 10 21:23:53.467790 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:53.467754    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8089512a-56eb-4fad-8ac4-8e79db806619-maas-api-service-ca\") pod \"8089512a-56eb-4fad-8ac4-8e79db806619\" (UID: \"8089512a-56eb-4fad-8ac4-8e79db806619\") "
Sep 10 21:23:53.468156 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:53.468132    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8089512a-56eb-4fad-8ac4-8e79db806619-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8089512a-56eb-4fad-8ac4-8e79db806619" (UID: "8089512a-56eb-4fad-8ac4-8e79db806619"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:23:53.568837 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:53.568753    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8089512a-56eb-4fad-8ac4-8e79db806619-maas-api-service-ca\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:23:54.262833 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:54.262793    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4ad710-xmp9f"
Sep 10 21:23:54.262984 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:54.262843    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4ad710-xmp9f" event={"ID":"8089512a-56eb-4fad-8ac4-8e79db806619","Type":"ContainerDied","Data":"49778be5778fd0a0e98bcab41857d43e37c14c8a83ca5b9c1403d481d5a06ba8"}
Sep 10 21:23:54.262984 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:54.262885    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49778be5778fd0a0e98bcab41857d43e37c14c8a83ca5b9c1403d481d5a06ba8"
Sep 10 21:23:55.384865 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:55.384828    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"]
Sep 10 21:23:56.343407 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:56.343372    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"]
Sep 10 21:23:57.117806 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.117775    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"]
Sep 10 21:23:57.118182 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.118170    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8089512a-56eb-4fad-8ac4-8e79db806619" containerName="revoke-keys"
Sep 10 21:23:57.118228 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.118184    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8089512a-56eb-4fad-8ac4-8e79db806619" containerName="revoke-keys"
Sep 10 21:23:57.118272 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.118248    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8089512a-56eb-4fad-8ac4-8e79db806619" containerName="revoke-keys"
Sep 10 21:23:57.121521 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.121499    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.123776 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.123747    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3bbf7d-openshift-default-dockercfg-4w7g4\""
Sep 10 21:23:57.133949 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.133926    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"]
Sep 10 21:23:57.304690 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.304650    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8khzw\" (UniqueName: \"kubernetes.io/projected/31ef0c45-7695-4bfc-9979-ca253fba416e-kube-api-access-8khzw\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.304690 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.304697    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-istio-data\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.304975 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.304770    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31ef0c45-7695-4bfc-9979-ca253fba416e-istio-token\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.304975 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.304847    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31ef0c45-7695-4bfc-9979-ca253fba416e-istio-podinfo\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.304975 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.304909    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31ef0c45-7695-4bfc-9979-ca253fba416e-istiod-ca-cert\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.304975 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.304944    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-workload-socket\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.305147 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.304974    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-workload-certs\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.305147 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.305026    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-istio-envoy\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.305147 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.305085    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-credential-socket\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.406386 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.406345    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8khzw\" (UniqueName: \"kubernetes.io/projected/31ef0c45-7695-4bfc-9979-ca253fba416e-kube-api-access-8khzw\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.406386 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.406387    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-istio-data\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.406648 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.406409    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31ef0c45-7695-4bfc-9979-ca253fba416e-istio-token\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.406648 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.406485    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31ef0c45-7695-4bfc-9979-ca253fba416e-istio-podinfo\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.406648 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.406544    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31ef0c45-7695-4bfc-9979-ca253fba416e-istiod-ca-cert\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.406648 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.406571    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-workload-socket\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.406648 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.406596    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-workload-certs\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.406648 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.406626    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-istio-envoy\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.406972 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.406682    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-credential-socket\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.406972 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.406891    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-istio-data\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.407072 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.406961    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-workload-socket\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.407072 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.407052    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-credential-socket\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.407154 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.407129    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-workload-certs\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.407362 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.407344    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31ef0c45-7695-4bfc-9979-ca253fba416e-istiod-ca-cert\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.408963 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.408940    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-istio-envoy\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.409177 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.409158    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31ef0c45-7695-4bfc-9979-ca253fba416e-istio-podinfo\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.415475 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.415456    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31ef0c45-7695-4bfc-9979-ca253fba416e-istio-token\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.415571 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.415519    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8khzw\" (UniqueName: \"kubernetes.io/projected/31ef0c45-7695-4bfc-9979-ca253fba416e-kube-api-access-8khzw\") pod \"e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") " pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.431408 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.431386    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:57.708148 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.708124    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"]
Sep 10 21:23:57.710008 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:23:57.709965    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ef0c45_7695_4bfc_9979_ca253fba416e.slice/crio-3beb96176b728ab5cb5041573fef8ed5120640505ee4a8ff171661ad6d462705 WatchSource:0}: Error finding container 3beb96176b728ab5cb5041573fef8ed5120640505ee4a8ff171661ad6d462705: Status 404 returned error can't find the container with id 3beb96176b728ab5cb5041573fef8ed5120640505ee4a8ff171661ad6d462705
Sep 10 21:23:57.712275 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.712233    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:23:57.712381 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.712303    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:23:57.712381 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:57.712341    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:23:58.285042 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:58.284997    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv" event={"ID":"31ef0c45-7695-4bfc-9979-ca253fba416e","Type":"ContainerStarted","Data":"88d9c18a50ea165b2ad4cb79592921632d3dd8162319d46b09e0ef932f281632"}
Sep 10 21:23:58.285417 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:58.285061    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv" event={"ID":"31ef0c45-7695-4bfc-9979-ca253fba416e","Type":"ContainerStarted","Data":"3beb96176b728ab5cb5041573fef8ed5120640505ee4a8ff171661ad6d462705"}
Sep 10 21:23:58.306078 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:58.306028    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv" podStartSLOduration=1.30601226 podStartE2EDuration="1.30601226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:23:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:23:58.304457833 +0000 UTC m=+1497.969482633" watchObservedRunningTime="2026-09-10 21:23:58.30601226 +0000 UTC m=+1497.971037058"
Sep 10 21:23:58.432306 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:58.432269    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:58.437224 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:58.437201    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:59.289940 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:59.289901    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:23:59.290919 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:23:59.290899    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:24:00.859684 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:00.859652    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-jkkfm_fd81843e-6283-43e3-a571-d5e47be26e38/console-operator/1.log"
Sep 10 21:24:00.866068 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:00.865889    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-jkkfm_fd81843e-6283-43e3-a571-d5e47be26e38/console-operator/1.log"
Sep 10 21:24:01.201405 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:01.201377    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4ad710-xmp9f"]
Sep 10 21:24:01.206049 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:01.206024    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4ad710-xmp9f"]
Sep 10 21:24:02.841081 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:02.841048    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8089512a-56eb-4fad-8ac4-8e79db806619" path="/var/lib/kubelet/pods/8089512a-56eb-4fad-8ac4-8e79db806619/volumes"
Sep 10 21:24:08.685274 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:08.685237    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"]
Sep 10 21:24:08.685863 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:08.685538    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh" podUID="64121917-eb0e-4b8a-91f3-69655dac1b55" containerName="istio-proxy" containerID="cri-o://62f50709b3e7b7cff78f1e2920eb10b5fd85490de90d8dcaf17471852e3d3469" gracePeriod=30
Sep 10 21:24:10.418148 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:10.417962    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz" podUID="0256de55-42b3-4b85-a6ab-959a51325e8e" containerName="main" containerID="cri-o://736ad175c419d52b942d418a69fc58227556d0d2121d20e1163dc66d0853a880" gracePeriod=45
Sep 10 21:24:10.682466 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:10.682396    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:24:10.738073 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:10.738036    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0256de55-42b3-4b85-a6ab-959a51325e8e-tls-certs\") pod \"0256de55-42b3-4b85-a6ab-959a51325e8e\" (UID: \"0256de55-42b3-4b85-a6ab-959a51325e8e\") "
Sep 10 21:24:10.738279 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:10.738087    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0256de55-42b3-4b85-a6ab-959a51325e8e-model-cache\") pod \"0256de55-42b3-4b85-a6ab-959a51325e8e\" (UID: \"0256de55-42b3-4b85-a6ab-959a51325e8e\") "
Sep 10 21:24:10.738279 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:10.738106    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0256de55-42b3-4b85-a6ab-959a51325e8e-home\") pod \"0256de55-42b3-4b85-a6ab-959a51325e8e\" (UID: \"0256de55-42b3-4b85-a6ab-959a51325e8e\") "
Sep 10 21:24:10.738279 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:10.738142    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0256de55-42b3-4b85-a6ab-959a51325e8e-tmp-dir\") pod \"0256de55-42b3-4b85-a6ab-959a51325e8e\" (UID: \"0256de55-42b3-4b85-a6ab-959a51325e8e\") "
Sep 10 21:24:10.738279 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:10.738223    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0256de55-42b3-4b85-a6ab-959a51325e8e-dshm\") pod \"0256de55-42b3-4b85-a6ab-959a51325e8e\" (UID: \"0256de55-42b3-4b85-a6ab-959a51325e8e\") "
Sep 10 21:24:10.738279 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:10.738251    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hzx5\" (UniqueName: \"kubernetes.io/projected/0256de55-42b3-4b85-a6ab-959a51325e8e-kube-api-access-6hzx5\") pod \"0256de55-42b3-4b85-a6ab-959a51325e8e\" (UID: \"0256de55-42b3-4b85-a6ab-959a51325e8e\") "
Sep 10 21:24:10.738598 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:10.738370    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0256de55-42b3-4b85-a6ab-959a51325e8e-model-cache" (OuterVolumeSpecName: "model-cache") pod "0256de55-42b3-4b85-a6ab-959a51325e8e" (UID: "0256de55-42b3-4b85-a6ab-959a51325e8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:24:10.738598 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:10.738464    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0256de55-42b3-4b85-a6ab-959a51325e8e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0256de55-42b3-4b85-a6ab-959a51325e8e" (UID: "0256de55-42b3-4b85-a6ab-959a51325e8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:24:10.738712 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:10.738626    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0256de55-42b3-4b85-a6ab-959a51325e8e-model-cache\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:24:10.738712 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:10.738639    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0256de55-42b3-4b85-a6ab-959a51325e8e-tmp-dir\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:24:10.738806 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:10.738781    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0256de55-42b3-4b85-a6ab-959a51325e8e-home" (OuterVolumeSpecName: "home") pod "0256de55-42b3-4b85-a6ab-959a51325e8e" (UID: "0256de55-42b3-4b85-a6ab-959a51325e8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:24:10.740540 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:10.740512    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0256de55-42b3-4b85-a6ab-959a51325e8e-kube-api-access-6hzx5" (OuterVolumeSpecName: "kube-api-access-6hzx5") pod "0256de55-42b3-4b85-a6ab-959a51325e8e" (UID: "0256de55-42b3-4b85-a6ab-959a51325e8e"). InnerVolumeSpecName "kube-api-access-6hzx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:24:10.740777 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:10.740759    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0256de55-42b3-4b85-a6ab-959a51325e8e-dshm" (OuterVolumeSpecName: "dshm") pod "0256de55-42b3-4b85-a6ab-959a51325e8e" (UID: "0256de55-42b3-4b85-a6ab-959a51325e8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:24:10.740817 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:10.740779    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0256de55-42b3-4b85-a6ab-959a51325e8e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0256de55-42b3-4b85-a6ab-959a51325e8e" (UID: "0256de55-42b3-4b85-a6ab-959a51325e8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:24:10.839921 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:10.839886    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0256de55-42b3-4b85-a6ab-959a51325e8e-home\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:24:10.839921 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:10.839920    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0256de55-42b3-4b85-a6ab-959a51325e8e-dshm\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:24:10.840114 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:10.839935    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hzx5\" (UniqueName: \"kubernetes.io/projected/0256de55-42b3-4b85-a6ab-959a51325e8e-kube-api-access-6hzx5\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:24:10.840114 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:10.839953    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0256de55-42b3-4b85-a6ab-959a51325e8e-tls-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:24:11.351877 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.351842    2584 generic.go:356] "Generic (PLEG): container finished" podID="0256de55-42b3-4b85-a6ab-959a51325e8e" containerID="736ad175c419d52b942d418a69fc58227556d0d2121d20e1163dc66d0853a880" exitCode=0
Sep 10 21:24:11.352050 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.351883    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"
Sep 10 21:24:11.352050 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.351929    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz" event={"ID":"0256de55-42b3-4b85-a6ab-959a51325e8e","Type":"ContainerDied","Data":"736ad175c419d52b942d418a69fc58227556d0d2121d20e1163dc66d0853a880"}
Sep 10 21:24:11.352050 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.351973    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz" event={"ID":"0256de55-42b3-4b85-a6ab-959a51325e8e","Type":"ContainerDied","Data":"5ce2d3428e96228ff1f2366ab53705eb0fa59e4cf75a22d530bc6313ee5a0a7d"}
Sep 10 21:24:11.352050 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.351990    2584 scope.go:117] "RemoveContainer" containerID="736ad175c419d52b942d418a69fc58227556d0d2121d20e1163dc66d0853a880"
Sep 10 21:24:11.360698 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.360681    2584 scope.go:117] "RemoveContainer" containerID="736ad175c419d52b942d418a69fc58227556d0d2121d20e1163dc66d0853a880"
Sep 10 21:24:11.361034 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:24:11.361010    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"736ad175c419d52b942d418a69fc58227556d0d2121d20e1163dc66d0853a880\": container with ID starting with 736ad175c419d52b942d418a69fc58227556d0d2121d20e1163dc66d0853a880 not found: ID does not exist" containerID="736ad175c419d52b942d418a69fc58227556d0d2121d20e1163dc66d0853a880"
Sep 10 21:24:11.361126 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.361040    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"736ad175c419d52b942d418a69fc58227556d0d2121d20e1163dc66d0853a880"} err="failed to get container status \"736ad175c419d52b942d418a69fc58227556d0d2121d20e1163dc66d0853a880\": rpc error: code = NotFound desc = could not find container \"736ad175c419d52b942d418a69fc58227556d0d2121d20e1163dc66d0853a880\": container with ID starting with 736ad175c419d52b942d418a69fc58227556d0d2121d20e1163dc66d0853a880 not found: ID does not exist"
Sep 10 21:24:11.362305 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.362280    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l" podUID="7e4455ad-0996-4f4c-923f-29a8fa595140" containerName="main" containerID="cri-o://39c057b695fa6be3b81202f7953b13d8427e257785a3ba27ce2a80caaa974600" gracePeriod=45
Sep 10 21:24:11.367966 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.367943    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"]
Sep 10 21:24:11.369719 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.369701    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0ac302/rate-test-model-0ac302-kserve-67bb4d8875-qznnz"]
Sep 10 21:24:11.653205 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.653180    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:24:11.749029 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.748992    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e4455ad-0996-4f4c-923f-29a8fa595140-model-cache\") pod \"7e4455ad-0996-4f4c-923f-29a8fa595140\" (UID: \"7e4455ad-0996-4f4c-923f-29a8fa595140\") "
Sep 10 21:24:11.749219 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.749050    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4455ad-0996-4f4c-923f-29a8fa595140-tmp-dir\") pod \"7e4455ad-0996-4f4c-923f-29a8fa595140\" (UID: \"7e4455ad-0996-4f4c-923f-29a8fa595140\") "
Sep 10 21:24:11.749219 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.749077    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e4455ad-0996-4f4c-923f-29a8fa595140-home\") pod \"7e4455ad-0996-4f4c-923f-29a8fa595140\" (UID: \"7e4455ad-0996-4f4c-923f-29a8fa595140\") "
Sep 10 21:24:11.749219 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.749106    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtpjq\" (UniqueName: \"kubernetes.io/projected/7e4455ad-0996-4f4c-923f-29a8fa595140-kube-api-access-mtpjq\") pod \"7e4455ad-0996-4f4c-923f-29a8fa595140\" (UID: \"7e4455ad-0996-4f4c-923f-29a8fa595140\") "
Sep 10 21:24:11.749219 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.749170    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e4455ad-0996-4f4c-923f-29a8fa595140-tls-certs\") pod \"7e4455ad-0996-4f4c-923f-29a8fa595140\" (UID: \"7e4455ad-0996-4f4c-923f-29a8fa595140\") "
Sep 10 21:24:11.749500 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.749276    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4455ad-0996-4f4c-923f-29a8fa595140-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e4455ad-0996-4f4c-923f-29a8fa595140" (UID: "7e4455ad-0996-4f4c-923f-29a8fa595140"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:24:11.749500 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.749290    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4455ad-0996-4f4c-923f-29a8fa595140-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e4455ad-0996-4f4c-923f-29a8fa595140" (UID: "7e4455ad-0996-4f4c-923f-29a8fa595140"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:24:11.749500 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.749304    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e4455ad-0996-4f4c-923f-29a8fa595140-dshm\") pod \"7e4455ad-0996-4f4c-923f-29a8fa595140\" (UID: \"7e4455ad-0996-4f4c-923f-29a8fa595140\") "
Sep 10 21:24:11.749500 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.749377    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4455ad-0996-4f4c-923f-29a8fa595140-home" (OuterVolumeSpecName: "home") pod "7e4455ad-0996-4f4c-923f-29a8fa595140" (UID: "7e4455ad-0996-4f4c-923f-29a8fa595140"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:24:11.749715 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.749699    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e4455ad-0996-4f4c-923f-29a8fa595140-model-cache\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:24:11.749760 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.749721    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4455ad-0996-4f4c-923f-29a8fa595140-tmp-dir\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:24:11.749760 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.749733    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e4455ad-0996-4f4c-923f-29a8fa595140-home\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:24:11.751295 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.751270    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4455ad-0996-4f4c-923f-29a8fa595140-kube-api-access-mtpjq" (OuterVolumeSpecName: "kube-api-access-mtpjq") pod "7e4455ad-0996-4f4c-923f-29a8fa595140" (UID: "7e4455ad-0996-4f4c-923f-29a8fa595140"). InnerVolumeSpecName "kube-api-access-mtpjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:24:11.751725 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.751708    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4455ad-0996-4f4c-923f-29a8fa595140-dshm" (OuterVolumeSpecName: "dshm") pod "7e4455ad-0996-4f4c-923f-29a8fa595140" (UID: "7e4455ad-0996-4f4c-923f-29a8fa595140"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:24:11.751792 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.751728    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e4455ad-0996-4f4c-923f-29a8fa595140-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e4455ad-0996-4f4c-923f-29a8fa595140" (UID: "7e4455ad-0996-4f4c-923f-29a8fa595140"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:24:11.850737 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.850702    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtpjq\" (UniqueName: \"kubernetes.io/projected/7e4455ad-0996-4f4c-923f-29a8fa595140-kube-api-access-mtpjq\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:24:11.850737 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.850731    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e4455ad-0996-4f4c-923f-29a8fa595140-tls-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:24:11.850737 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:11.850742    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e4455ad-0996-4f4c-923f-29a8fa595140-dshm\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:24:12.359070 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:12.359035    2584 generic.go:356] "Generic (PLEG): container finished" podID="7e4455ad-0996-4f4c-923f-29a8fa595140" containerID="39c057b695fa6be3b81202f7953b13d8427e257785a3ba27ce2a80caaa974600" exitCode=0
Sep 10 21:24:12.359070 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:12.359081    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"
Sep 10 21:24:12.359298 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:12.359122    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l" event={"ID":"7e4455ad-0996-4f4c-923f-29a8fa595140","Type":"ContainerDied","Data":"39c057b695fa6be3b81202f7953b13d8427e257785a3ba27ce2a80caaa974600"}
Sep 10 21:24:12.359298 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:12.359162    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l" event={"ID":"7e4455ad-0996-4f4c-923f-29a8fa595140","Type":"ContainerDied","Data":"fe047e699c3ccf4fb1cee669d17b4441ee1488a937f2baad61f7c30b1cada62b"}
Sep 10 21:24:12.359298 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:12.359181    2584 scope.go:117] "RemoveContainer" containerID="39c057b695fa6be3b81202f7953b13d8427e257785a3ba27ce2a80caaa974600"
Sep 10 21:24:12.369470 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:12.369454    2584 scope.go:117] "RemoveContainer" containerID="39c057b695fa6be3b81202f7953b13d8427e257785a3ba27ce2a80caaa974600"
Sep 10 21:24:12.369761 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:24:12.369744    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c057b695fa6be3b81202f7953b13d8427e257785a3ba27ce2a80caaa974600\": container with ID starting with 39c057b695fa6be3b81202f7953b13d8427e257785a3ba27ce2a80caaa974600 not found: ID does not exist" containerID="39c057b695fa6be3b81202f7953b13d8427e257785a3ba27ce2a80caaa974600"
Sep 10 21:24:12.369823 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:12.369768    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c057b695fa6be3b81202f7953b13d8427e257785a3ba27ce2a80caaa974600"} err="failed to get container status \"39c057b695fa6be3b81202f7953b13d8427e257785a3ba27ce2a80caaa974600\": rpc error: code = NotFound desc = could not find container \"39c057b695fa6be3b81202f7953b13d8427e257785a3ba27ce2a80caaa974600\": container with ID starting with 39c057b695fa6be3b81202f7953b13d8427e257785a3ba27ce2a80caaa974600 not found: ID does not exist"
Sep 10 21:24:12.382522 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:12.382498    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"]
Sep 10 21:24:12.386188 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:12.386166    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-16bbec/rate-test-model-16bbec-kserve-7bbbb65f95-gwr5l"]
Sep 10 21:24:12.840353 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:12.840318    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0256de55-42b3-4b85-a6ab-959a51325e8e" path="/var/lib/kubelet/pods/0256de55-42b3-4b85-a6ab-959a51325e8e/volumes"
Sep 10 21:24:12.840823 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:12.840808    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e4455ad-0996-4f4c-923f-29a8fa595140" path="/var/lib/kubelet/pods/7e4455ad-0996-4f4c-923f-29a8fa595140/volumes"
Sep 10 21:24:13.972992 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:13.972963    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:24:14.071888 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.071857    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64121917-eb0e-4b8a-91f3-69655dac1b55-istio-token\") pod \"64121917-eb0e-4b8a-91f3-69655dac1b55\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") "
Sep 10 21:24:14.072052 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.071924    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-istio-data\") pod \"64121917-eb0e-4b8a-91f3-69655dac1b55\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") "
Sep 10 21:24:14.072052 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.071954    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5q5vx\" (UniqueName: \"kubernetes.io/projected/64121917-eb0e-4b8a-91f3-69655dac1b55-kube-api-access-5q5vx\") pod \"64121917-eb0e-4b8a-91f3-69655dac1b55\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") "
Sep 10 21:24:14.072052 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.071982    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64121917-eb0e-4b8a-91f3-69655dac1b55-istio-podinfo\") pod \"64121917-eb0e-4b8a-91f3-69655dac1b55\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") "
Sep 10 21:24:14.072052 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.072014    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-istio-envoy\") pod \"64121917-eb0e-4b8a-91f3-69655dac1b55\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") "
Sep 10 21:24:14.072052 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.072052    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-workload-certs\") pod \"64121917-eb0e-4b8a-91f3-69655dac1b55\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") "
Sep 10 21:24:14.072355 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.072077    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64121917-eb0e-4b8a-91f3-69655dac1b55-istiod-ca-cert\") pod \"64121917-eb0e-4b8a-91f3-69655dac1b55\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") "
Sep 10 21:24:14.072355 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.072165    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-credential-socket\") pod \"64121917-eb0e-4b8a-91f3-69655dac1b55\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") "
Sep 10 21:24:14.072355 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.072211    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-workload-socket\") pod \"64121917-eb0e-4b8a-91f3-69655dac1b55\" (UID: \"64121917-eb0e-4b8a-91f3-69655dac1b55\") "
Sep 10 21:24:14.072355 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.072300    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-istio-data" (OuterVolumeSpecName: "istio-data") pod "64121917-eb0e-4b8a-91f3-69655dac1b55" (UID: "64121917-eb0e-4b8a-91f3-69655dac1b55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:24:14.072355 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.072325    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "64121917-eb0e-4b8a-91f3-69655dac1b55" (UID: "64121917-eb0e-4b8a-91f3-69655dac1b55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:24:14.072666 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.072565    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "64121917-eb0e-4b8a-91f3-69655dac1b55" (UID: "64121917-eb0e-4b8a-91f3-69655dac1b55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:24:14.072666 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.072609    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "64121917-eb0e-4b8a-91f3-69655dac1b55" (UID: "64121917-eb0e-4b8a-91f3-69655dac1b55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:24:14.072666 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.072652    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-workload-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:24:14.072786 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.072670    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-istio-data\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:24:14.072786 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.072672    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64121917-eb0e-4b8a-91f3-69655dac1b55-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "64121917-eb0e-4b8a-91f3-69655dac1b55" (UID: "64121917-eb0e-4b8a-91f3-69655dac1b55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:24:14.074252 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.074232    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "64121917-eb0e-4b8a-91f3-69655dac1b55" (UID: "64121917-eb0e-4b8a-91f3-69655dac1b55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:24:14.074387 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.074362    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64121917-eb0e-4b8a-91f3-69655dac1b55-kube-api-access-5q5vx" (OuterVolumeSpecName: "kube-api-access-5q5vx") pod "64121917-eb0e-4b8a-91f3-69655dac1b55" (UID: "64121917-eb0e-4b8a-91f3-69655dac1b55"). InnerVolumeSpecName "kube-api-access-5q5vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:24:14.074530 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.074515    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64121917-eb0e-4b8a-91f3-69655dac1b55-istio-token" (OuterVolumeSpecName: "istio-token") pod "64121917-eb0e-4b8a-91f3-69655dac1b55" (UID: "64121917-eb0e-4b8a-91f3-69655dac1b55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:24:14.074720 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.074678    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64121917-eb0e-4b8a-91f3-69655dac1b55-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "64121917-eb0e-4b8a-91f3-69655dac1b55" (UID: "64121917-eb0e-4b8a-91f3-69655dac1b55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:24:14.173184 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.173143    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64121917-eb0e-4b8a-91f3-69655dac1b55-istio-token\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:24:14.173184 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.173178    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q5vx\" (UniqueName: \"kubernetes.io/projected/64121917-eb0e-4b8a-91f3-69655dac1b55-kube-api-access-5q5vx\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:24:14.173184 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.173189    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64121917-eb0e-4b8a-91f3-69655dac1b55-istio-podinfo\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:24:14.173184 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.173197    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-istio-envoy\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:24:14.173514 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.173206    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64121917-eb0e-4b8a-91f3-69655dac1b55-istiod-ca-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:24:14.173514 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.173214    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-credential-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:24:14.173514 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.173224    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64121917-eb0e-4b8a-91f3-69655dac1b55-workload-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:24:14.372126 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.372094    2584 generic.go:356] "Generic (PLEG): container finished" podID="64121917-eb0e-4b8a-91f3-69655dac1b55" containerID="62f50709b3e7b7cff78f1e2920eb10b5fd85490de90d8dcaf17471852e3d3469" exitCode=0
Sep 10 21:24:14.372286 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.372144    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh" event={"ID":"64121917-eb0e-4b8a-91f3-69655dac1b55","Type":"ContainerDied","Data":"62f50709b3e7b7cff78f1e2920eb10b5fd85490de90d8dcaf17471852e3d3469"}
Sep 10 21:24:14.372286 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.372190    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh" event={"ID":"64121917-eb0e-4b8a-91f3-69655dac1b55","Type":"ContainerDied","Data":"504dcb6ea4d41c54f07e92b47828436d72591a77cea9f71119d1eef746e7c4cb"}
Sep 10 21:24:14.372286 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.372202    2584 scope.go:117] "RemoveContainer" containerID="62f50709b3e7b7cff78f1e2920eb10b5fd85490de90d8dcaf17471852e3d3469"
Sep 10 21:24:14.372286 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.372189    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"
Sep 10 21:24:14.382624 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.382598    2584 scope.go:117] "RemoveContainer" containerID="0d8ecbae1257923e00a925863def93a27750c4d687429915a64e52e24e238aff"
Sep 10 21:24:14.390526 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.390505    2584 scope.go:117] "RemoveContainer" containerID="62f50709b3e7b7cff78f1e2920eb10b5fd85490de90d8dcaf17471852e3d3469"
Sep 10 21:24:14.390807 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:24:14.390788    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62f50709b3e7b7cff78f1e2920eb10b5fd85490de90d8dcaf17471852e3d3469\": container with ID starting with 62f50709b3e7b7cff78f1e2920eb10b5fd85490de90d8dcaf17471852e3d3469 not found: ID does not exist" containerID="62f50709b3e7b7cff78f1e2920eb10b5fd85490de90d8dcaf17471852e3d3469"
Sep 10 21:24:14.390890 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.390818    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f50709b3e7b7cff78f1e2920eb10b5fd85490de90d8dcaf17471852e3d3469"} err="failed to get container status \"62f50709b3e7b7cff78f1e2920eb10b5fd85490de90d8dcaf17471852e3d3469\": rpc error: code = NotFound desc = could not find container \"62f50709b3e7b7cff78f1e2920eb10b5fd85490de90d8dcaf17471852e3d3469\": container with ID starting with 62f50709b3e7b7cff78f1e2920eb10b5fd85490de90d8dcaf17471852e3d3469 not found: ID does not exist"
Sep 10 21:24:14.390890 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.390841    2584 scope.go:117] "RemoveContainer" containerID="0d8ecbae1257923e00a925863def93a27750c4d687429915a64e52e24e238aff"
Sep 10 21:24:14.391105 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:24:14.391087    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8ecbae1257923e00a925863def93a27750c4d687429915a64e52e24e238aff\": container with ID starting with 0d8ecbae1257923e00a925863def93a27750c4d687429915a64e52e24e238aff not found: ID does not exist" containerID="0d8ecbae1257923e00a925863def93a27750c4d687429915a64e52e24e238aff"
Sep 10 21:24:14.391143 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.391110    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8ecbae1257923e00a925863def93a27750c4d687429915a64e52e24e238aff"} err="failed to get container status \"0d8ecbae1257923e00a925863def93a27750c4d687429915a64e52e24e238aff\": rpc error: code = NotFound desc = could not find container \"0d8ecbae1257923e00a925863def93a27750c4d687429915a64e52e24e238aff\": container with ID starting with 0d8ecbae1257923e00a925863def93a27750c4d687429915a64e52e24e238aff not found: ID does not exist"
Sep 10 21:24:14.394998 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.394976    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"]
Sep 10 21:24:14.400147 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.400119    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4ad710-openshift-default-676f47554b-kdvgh"]
Sep 10 21:24:14.841307 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:14.841275    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64121917-eb0e-4b8a-91f3-69655dac1b55" path="/var/lib/kubelet/pods/64121917-eb0e-4b8a-91f3-69655dac1b55/volumes"
Sep 10 21:24:15.331589 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:15.331555    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 21:24:15.331973 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:15.331622    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv" podUID="31ef0c45-7695-4bfc-9979-ca253fba416e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 21:24:17.390241 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:17.390209    2584 generic.go:356] "Generic (PLEG): container finished" podID="31ef0c45-7695-4bfc-9979-ca253fba416e" containerID="88d9c18a50ea165b2ad4cb79592921632d3dd8162319d46b09e0ef932f281632" exitCode=0
Sep 10 21:24:17.390757 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:17.390262    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv" event={"ID":"31ef0c45-7695-4bfc-9979-ca253fba416e","Type":"ContainerDied","Data":"88d9c18a50ea165b2ad4cb79592921632d3dd8162319d46b09e0ef932f281632"}
Sep 10 21:24:17.390838 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:17.390816    2584 scope.go:117] "RemoveContainer" containerID="88d9c18a50ea165b2ad4cb79592921632d3dd8162319d46b09e0ef932f281632"
Sep 10 21:24:17.391967 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:17.391931    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:24:17.392071 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:17.391999    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:24:17.392071 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:17.392029    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:24:18.396649 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:18.396609    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv" event={"ID":"31ef0c45-7695-4bfc-9979-ca253fba416e","Type":"ContainerStarted","Data":"fb338f4d9ecc69ffb83fcad5c11716d4d362dfd9b4501a9c425798b9a060b05a"}
Sep 10 21:24:18.431755 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:18.431716    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:24:18.431950 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:18.431924    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 10 21:24:18.432070 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:18.431995    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv" podUID="31ef0c45-7695-4bfc-9979-ca253fba416e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 10 21:24:19.432548 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:19.432511    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 10 21:24:19.432948 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:19.432588    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv" podUID="31ef0c45-7695-4bfc-9979-ca253fba416e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 10 21:24:20.432713 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:20.432675    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 10 21:24:20.433187 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:20.432731    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv" podUID="31ef0c45-7695-4bfc-9979-ca253fba416e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 10 21:24:21.432130 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:21.432091    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 10 21:24:21.432326 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:21.432160    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv" podUID="31ef0c45-7695-4bfc-9979-ca253fba416e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 10 21:24:22.432590 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:22.432551    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 10 21:24:22.432988 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:22.432612    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv" podUID="31ef0c45-7695-4bfc-9979-ca253fba416e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 10 21:24:23.527281 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:23.527199    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:24:23.528057 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:23.527656    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:24:23.528877 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:23.528856    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:24:26.276717 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.276678    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"]
Sep 10 21:24:26.277237 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.277215    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e4455ad-0996-4f4c-923f-29a8fa595140" containerName="main"
Sep 10 21:24:26.277237 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.277236    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e4455ad-0996-4f4c-923f-29a8fa595140" containerName="main"
Sep 10 21:24:26.277400 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.277248    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64121917-eb0e-4b8a-91f3-69655dac1b55" containerName="istio-proxy"
Sep 10 21:24:26.277400 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.277259    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="64121917-eb0e-4b8a-91f3-69655dac1b55" containerName="istio-proxy"
Sep 10 21:24:26.277400 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.277270    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0256de55-42b3-4b85-a6ab-959a51325e8e" containerName="main"
Sep 10 21:24:26.277400 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.277278    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0256de55-42b3-4b85-a6ab-959a51325e8e" containerName="main"
Sep 10 21:24:26.277400 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.277293    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64121917-eb0e-4b8a-91f3-69655dac1b55" containerName="istio-proxy"
Sep 10 21:24:26.277400 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.277299    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="64121917-eb0e-4b8a-91f3-69655dac1b55" containerName="istio-proxy"
Sep 10 21:24:26.277400 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.277387    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="64121917-eb0e-4b8a-91f3-69655dac1b55" containerName="istio-proxy"
Sep 10 21:24:26.277400 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.277398    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0256de55-42b3-4b85-a6ab-959a51325e8e" containerName="main"
Sep 10 21:24:26.277400 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.277405    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e4455ad-0996-4f4c-923f-29a8fa595140" containerName="main"
Sep 10 21:24:26.279754 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.279737    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.283884 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.283864    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a581b5-openshift-default-dockercfg-vzh7b\""
Sep 10 21:24:26.292807 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.292784    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"]
Sep 10 21:24:26.396162 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.396118    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istio-podinfo\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.396361 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.396198    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-workload-certs\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.396361 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.396230    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istiod-ca-cert\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.396361 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.396279    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-credential-socket\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.396553 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.396373    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x82rn\" (UniqueName: \"kubernetes.io/projected/8018c9ab-ec1a-49c6-8664-175138d0d6e0-kube-api-access-x82rn\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.396553 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.396452    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istio-token\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.396553 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.396506    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-workload-socket\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.396553 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.396531    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istio-data\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.396763 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.396591    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istio-envoy\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.497764 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.497730    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-workload-certs\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.497908 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.497767    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istiod-ca-cert\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.497908 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.497793    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-credential-socket\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.497908 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.497819    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x82rn\" (UniqueName: \"kubernetes.io/projected/8018c9ab-ec1a-49c6-8664-175138d0d6e0-kube-api-access-x82rn\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.497908 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.497850    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istio-token\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.497908 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.497882    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-workload-socket\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.497908 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.497896    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istio-data\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.498232 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.497938    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istio-envoy\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.498232 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.497991    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istio-podinfo\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.498232 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.498149    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-workload-certs\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.498355 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.498259    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-workload-socket\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.498355 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.498337    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istio-data\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.498452 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.498380    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-credential-socket\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.498560 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.498525    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istiod-ca-cert\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.500344 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.500321    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istio-envoy\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.500489 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.500471    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istio-podinfo\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.508361 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.508338    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x82rn\" (UniqueName: \"kubernetes.io/projected/8018c9ab-ec1a-49c6-8664-175138d0d6e0-kube-api-access-x82rn\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.520959 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.520935    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istio-token\") pod \"e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") " pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.590242 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.590156    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:26.868010 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.867985    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"]
Sep 10 21:24:26.870372 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:24:26.870344    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8018c9ab_ec1a_49c6_8664_175138d0d6e0.slice/crio-2aee16d840a2e716154ce30c5c9e5c2f948dd4269a4d472f425e5eb4e2c69550 WatchSource:0}: Error finding container 2aee16d840a2e716154ce30c5c9e5c2f948dd4269a4d472f425e5eb4e2c69550: Status 404 returned error can't find the container with id 2aee16d840a2e716154ce30c5c9e5c2f948dd4269a4d472f425e5eb4e2c69550
Sep 10 21:24:26.872663 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.872580    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:24:26.872663 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.872645    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:24:26.872792 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:26.872677    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:24:27.453752 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:27.453718    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs" event={"ID":"8018c9ab-ec1a-49c6-8664-175138d0d6e0","Type":"ContainerStarted","Data":"7309d5da08e84f65895eb60edf138c5e64c636c820f813f97addc89c6d7d5471"}
Sep 10 21:24:27.454139 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:27.453772    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs" event={"ID":"8018c9ab-ec1a-49c6-8664-175138d0d6e0","Type":"ContainerStarted","Data":"2aee16d840a2e716154ce30c5c9e5c2f948dd4269a4d472f425e5eb4e2c69550"}
Sep 10 21:24:27.478395 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:27.478339    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs" podStartSLOduration=1.4783246540000001 podStartE2EDuration="1.478324654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:24:27.476238473 +0000 UTC m=+1527.141263274" watchObservedRunningTime="2026-09-10 21:24:27.478324654 +0000 UTC m=+1527.143349452"
Sep 10 21:24:27.590744 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:27.590712    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:27.598964 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:27.598934    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:28.462461 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:28.460349    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:28.465516 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:28.465487    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:47.570217 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:47.570180    2584 generic.go:356] "Generic (PLEG): container finished" podID="8018c9ab-ec1a-49c6-8664-175138d0d6e0" containerID="7309d5da08e84f65895eb60edf138c5e64c636c820f813f97addc89c6d7d5471" exitCode=0
Sep 10 21:24:47.570734 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:47.570251    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs" event={"ID":"8018c9ab-ec1a-49c6-8664-175138d0d6e0","Type":"ContainerDied","Data":"7309d5da08e84f65895eb60edf138c5e64c636c820f813f97addc89c6d7d5471"}
Sep 10 21:24:47.570820 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:47.570802    2584 scope.go:117] "RemoveContainer" containerID="7309d5da08e84f65895eb60edf138c5e64c636c820f813f97addc89c6d7d5471"
Sep 10 21:24:47.571994 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:47.571967    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:24:47.572104 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:47.572049    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:24:47.572104 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:47.572093    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:24:48.578203 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:48.578172    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs" event={"ID":"8018c9ab-ec1a-49c6-8664-175138d0d6e0","Type":"ContainerStarted","Data":"bfa5402dc16e2b7d7b7e70f362e0fe191e69c7e922853453b9210516441df45a"}
Sep 10 21:24:48.590867 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:48.590842    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:48.591035 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:48.591013    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 10 21:24:48.591106 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:48.591070    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs" podUID="8018c9ab-ec1a-49c6-8664-175138d0d6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 10 21:24:49.591026 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:49.590991    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 10 21:24:49.591413 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:49.591043    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs" podUID="8018c9ab-ec1a-49c6-8664-175138d0d6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 10 21:24:50.591595 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:50.591560    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 10 21:24:50.591980 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:50.591617    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs" podUID="8018c9ab-ec1a-49c6-8664-175138d0d6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 10 21:24:51.590755 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:51.590720    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 10 21:24:51.590938 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:51.590773    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs" podUID="8018c9ab-ec1a-49c6-8664-175138d0d6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 10 21:24:52.591559 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:52.591522    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 10 21:24:52.591947 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:52.591576    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs" podUID="8018c9ab-ec1a-49c6-8664-175138d0d6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 10 21:24:53.653537 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:53.653500    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:53.653996 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:53.653976    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:24:53.654805 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:24:53.654784    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:25:11.672075 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.672043    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"]
Sep 10 21:25:11.672693 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.672669    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="64121917-eb0e-4b8a-91f3-69655dac1b55" containerName="istio-proxy"
Sep 10 21:25:11.675826 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.675807    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:25:11.678934 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.678909    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-3bbf7d\"/\"kube-root-ca.crt\""
Sep 10 21:25:11.678934 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.678924    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-3bbf7d\"/\"openshift-service-ca.crt\""
Sep 10 21:25:11.679112 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.678966    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-3bbf7d\"/\"default-dockercfg-l657d\""
Sep 10 21:25:11.679246 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.679227    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-3bbf7d\"/\"ipp-route-3bbf7d-kserve-self-signed-certs\""
Sep 10 21:25:11.687537 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.687513    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"]
Sep 10 21:25:11.857872 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.857835    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmqsh\" (UniqueName: \"kubernetes.io/projected/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-kube-api-access-wmqsh\") pod \"ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg\" (UID: \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\") " pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:25:11.858043 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.857919    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-home\") pod \"ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg\" (UID: \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\") " pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:25:11.858043 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.857939    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-model-cache\") pod \"ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg\" (UID: \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\") " pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:25:11.858043 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.857959    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-tls-certs\") pod \"ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg\" (UID: \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\") " pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:25:11.858043 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.858016    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-tmp-dir\") pod \"ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg\" (UID: \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\") " pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:25:11.858182 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.858063    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-dshm\") pod \"ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg\" (UID: \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\") " pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:25:11.958880 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.958797    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-dshm\") pod \"ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg\" (UID: \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\") " pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:25:11.958880 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.958857    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmqsh\" (UniqueName: \"kubernetes.io/projected/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-kube-api-access-wmqsh\") pod \"ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg\" (UID: \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\") " pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:25:11.959066 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.958911    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-home\") pod \"ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg\" (UID: \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\") " pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:25:11.959066 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.958927    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-model-cache\") pod \"ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg\" (UID: \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\") " pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:25:11.959066 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.959042    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-tls-certs\") pod \"ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg\" (UID: \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\") " pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:25:11.959188 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.959108    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-tmp-dir\") pod \"ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg\" (UID: \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\") " pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:25:11.959314 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.959293    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-home\") pod \"ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg\" (UID: \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\") " pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:25:11.959412 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.959338    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-model-cache\") pod \"ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg\" (UID: \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\") " pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:25:11.959555 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.959534    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-tmp-dir\") pod \"ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg\" (UID: \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\") " pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:25:11.961184 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.961157    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-dshm\") pod \"ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg\" (UID: \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\") " pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:25:11.961473 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.961455    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-tls-certs\") pod \"ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg\" (UID: \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\") " pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:25:11.967085 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.967061    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmqsh\" (UniqueName: \"kubernetes.io/projected/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-kube-api-access-wmqsh\") pod \"ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg\" (UID: \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\") " pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:25:11.988114 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:11.988089    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:25:12.258493 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:12.258465    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"]
Sep 10 21:25:12.260534 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:25:12.260506    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9f9f346_eb8c_4f3c_9bc9_aa12edeec3fd.slice/crio-8937c86dcfff632716f048bc53cb4de852baca539f73a7466258e61d1a523c0c WatchSource:0}: Error finding container 8937c86dcfff632716f048bc53cb4de852baca539f73a7466258e61d1a523c0c: Status 404 returned error can't find the container with id 8937c86dcfff632716f048bc53cb4de852baca539f73a7466258e61d1a523c0c
Sep 10 21:25:12.697936 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:12.697901    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg" event={"ID":"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd","Type":"ContainerStarted","Data":"abdd0ad584ec384c90d8d0d313be411812bca45a0633bbde3291351d16c99c2b"}
Sep 10 21:25:12.697936 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:12.697942    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg" event={"ID":"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd","Type":"ContainerStarted","Data":"8937c86dcfff632716f048bc53cb4de852baca539f73a7466258e61d1a523c0c"}
Sep 10 21:25:12.716200 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:12.716140    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg" podStartSLOduration=1.716123886 podStartE2EDuration="1.716123886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:25:12.715399175 +0000 UTC m=+1572.380423976" watchObservedRunningTime="2026-09-10 21:25:12.716123886 +0000 UTC m=+1572.381148687"
Sep 10 21:25:21.989006 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:21.988966    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:25:21.989552 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:21.989122    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:25:22.001492 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:22.001468    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:25:22.761245 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:22.761212    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:25:36.381188 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.381153    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"]
Sep 10 21:25:36.385181 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.385163    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.387580 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.387550    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-d24c8a-openshift-default-dockercfg-wtmkt\""
Sep 10 21:25:36.397318 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.397289    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"]
Sep 10 21:25:36.503383 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.503343    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istio-token\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.503383 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.503382    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjqt7\" (UniqueName: \"kubernetes.io/projected/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-kube-api-access-sjqt7\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.503625 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.503416    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-workload-socket\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.503625 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.503499    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istio-podinfo\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.503625 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.503543    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istio-data\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.503625 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.503562    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istiod-ca-cert\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.503760 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.503664    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-credential-socket\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.503760 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.503698    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istio-envoy\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.503760 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.503719    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-workload-certs\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.604838 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.604790    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-credential-socket\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.604838 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.604837    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istio-envoy\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.605086 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.604931    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-workload-certs\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.605086 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.604987    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istio-token\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.605086 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.605004    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjqt7\" (UniqueName: \"kubernetes.io/projected/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-kube-api-access-sjqt7\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.605266 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.605102    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-workload-socket\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.605266 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.605169    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istio-podinfo\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.605266 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.605219    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istio-data\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.605266 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.605224    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-credential-socket\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.605266 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.605246    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istiod-ca-cert\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.605562 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.605487    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-workload-socket\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.605562 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.605534    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istio-data\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.605722 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.605701    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-workload-certs\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.605908 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.605892    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istiod-ca-cert\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.607221 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.607201    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istio-envoy\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.607569 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.607549    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istio-podinfo\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.613036 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.613014    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istio-token\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.613322 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.613303    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjqt7\" (UniqueName: \"kubernetes.io/projected/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-kube-api-access-sjqt7\") pod \"e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") " pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.695949 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.695862    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:36.973135 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.973060    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"]
Sep 10 21:25:36.976320 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:25:36.976292    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod294baf3a_2b92_423c_92ac_b90dd2eeaf4e.slice/crio-fdb218d024ec784acf2ec321ed8fef7b2d249527e327847b2e9602d3d4be17b7 WatchSource:0}: Error finding container fdb218d024ec784acf2ec321ed8fef7b2d249527e327847b2e9602d3d4be17b7: Status 404 returned error can't find the container with id fdb218d024ec784acf2ec321ed8fef7b2d249527e327847b2e9602d3d4be17b7
Sep 10 21:25:36.978565 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.978528    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:25:36.978646 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.978602    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:25:36.978646 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:36.978632    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:25:37.829345 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:37.829308    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh" event={"ID":"294baf3a-2b92-423c-92ac-b90dd2eeaf4e","Type":"ContainerStarted","Data":"2a550349126afc180c814f8f55f2c5f9fdaae11af5b0d79e8939646e6a1b9a1b"}
Sep 10 21:25:37.829863 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:37.829376    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh" event={"ID":"294baf3a-2b92-423c-92ac-b90dd2eeaf4e","Type":"ContainerStarted","Data":"fdb218d024ec784acf2ec321ed8fef7b2d249527e327847b2e9602d3d4be17b7"}
Sep 10 21:25:37.850605 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:37.850545    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh" podStartSLOduration=1.850530432 podStartE2EDuration="1.850530432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:25:37.84850417 +0000 UTC m=+1597.513528971" watchObservedRunningTime="2026-09-10 21:25:37.850530432 +0000 UTC m=+1597.515555231"
Sep 10 21:25:38.696439 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:38.696401    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:38.701402 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:38.701375    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:38.841451 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:38.841400    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:38.841840 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:38.841486    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:25:58.947781 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:58.947746    2584 generic.go:356] "Generic (PLEG): container finished" podID="294baf3a-2b92-423c-92ac-b90dd2eeaf4e" containerID="2a550349126afc180c814f8f55f2c5f9fdaae11af5b0d79e8939646e6a1b9a1b" exitCode=0
Sep 10 21:25:58.948219 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:58.947818    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh" event={"ID":"294baf3a-2b92-423c-92ac-b90dd2eeaf4e","Type":"ContainerDied","Data":"2a550349126afc180c814f8f55f2c5f9fdaae11af5b0d79e8939646e6a1b9a1b"}
Sep 10 21:25:58.948282 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:58.948230    2584 scope.go:117] "RemoveContainer" containerID="2a550349126afc180c814f8f55f2c5f9fdaae11af5b0d79e8939646e6a1b9a1b"
Sep 10 21:25:58.949329 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:58.949306    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:25:58.949392 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:58.949373    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:25:58.949447 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:58.949404    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 10 21:25:59.955980 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:25:59.955947    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh" event={"ID":"294baf3a-2b92-423c-92ac-b90dd2eeaf4e","Type":"ContainerStarted","Data":"372f8d094b0c2ceca6ba2ab354c8fb73da749105d9ff0e9651b30c040f5c9af9"}
Sep 10 21:26:00.696982 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:00.696947    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:26:00.697195 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:00.697174    2584 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 10 21:26:00.697243 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:00.697219    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh" podUID="294baf3a-2b92-423c-92ac-b90dd2eeaf4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 10 21:26:01.117336 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:01.117231    2584 scope.go:117] "RemoveContainer" containerID="64d208ffc46188badbb174e62609dbf6d3f7e78be38fd535409db3983fb01486"
Sep 10 21:26:01.696919 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:01.696881    2584 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 10 21:26:01.697098 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:01.696939    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh" podUID="294baf3a-2b92-423c-92ac-b90dd2eeaf4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 10 21:26:02.696869 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:02.696837    2584 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 10 21:26:02.697242 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:02.696894    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh" podUID="294baf3a-2b92-423c-92ac-b90dd2eeaf4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 10 21:26:03.696264 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:03.696227    2584 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 10 21:26:03.696450 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:03.696289    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh" podUID="294baf3a-2b92-423c-92ac-b90dd2eeaf4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 10 21:26:04.696692 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:04.696655    2584 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 10 21:26:04.697077 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:04.696717    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh" podUID="294baf3a-2b92-423c-92ac-b90dd2eeaf4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 10 21:26:05.578707 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:05.578669    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d24c8a-rl9xl"]
Sep 10 21:26:05.582057 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:05.582038    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d24c8a-rl9xl"
Sep 10 21:26:05.584272 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:05.584246    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:26:05.584400 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:05.584321    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4gs5\""
Sep 10 21:26:05.590043 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:05.590014    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d24c8a-rl9xl"]
Sep 10 21:26:05.700970 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:05.700943    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:26:05.701442 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:05.701212    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:26:05.701960 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:05.701943    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:26:05.712764 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:05.712735    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9006427-c201-4dd6-b693-d9599c28acdc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-d24c8a-rl9xl\" (UID: \"d9006427-c201-4dd6-b693-d9599c28acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d24c8a-rl9xl"
Sep 10 21:26:05.814066 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:05.814027    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9006427-c201-4dd6-b693-d9599c28acdc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-d24c8a-rl9xl\" (UID: \"d9006427-c201-4dd6-b693-d9599c28acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d24c8a-rl9xl"
Sep 10 21:26:05.814723 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:05.814695    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9006427-c201-4dd6-b693-d9599c28acdc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-d24c8a-rl9xl\" (UID: \"d9006427-c201-4dd6-b693-d9599c28acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d24c8a-rl9xl"
Sep 10 21:26:05.893974 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:05.893942    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d24c8a-rl9xl"
Sep 10 21:26:06.166629 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:06.166599    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d24c8a-rl9xl"]
Sep 10 21:26:06.168848 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:26:06.168816    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9006427_c201_4dd6_b693_d9599c28acdc.slice/crio-d58bf9f19aaf8716e64be399cce35b9cb5bfc1473dbefa71b2bf73ad74fd4644 WatchSource:0}: Error finding container d58bf9f19aaf8716e64be399cce35b9cb5bfc1473dbefa71b2bf73ad74fd4644: Status 404 returned error can't find the container with id d58bf9f19aaf8716e64be399cce35b9cb5bfc1473dbefa71b2bf73ad74fd4644
Sep 10 21:26:06.995500 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:06.995456    2584 generic.go:356] "Generic (PLEG): container finished" podID="d9006427-c201-4dd6-b693-d9599c28acdc" containerID="afef252cce01d00fe6fc9dca2812fe2e6d2706679a152f123ac0c9976aff3d99" exitCode=0
Sep 10 21:26:06.995924 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:06.995529    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d24c8a-rl9xl" event={"ID":"d9006427-c201-4dd6-b693-d9599c28acdc","Type":"ContainerDied","Data":"afef252cce01d00fe6fc9dca2812fe2e6d2706679a152f123ac0c9976aff3d99"}
Sep 10 21:26:06.995924 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:06.995613    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d24c8a-rl9xl" event={"ID":"d9006427-c201-4dd6-b693-d9599c28acdc","Type":"ContainerStarted","Data":"d58bf9f19aaf8716e64be399cce35b9cb5bfc1473dbefa71b2bf73ad74fd4644"}
Sep 10 21:26:08.134299 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:08.134276    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d24c8a-rl9xl"
Sep 10 21:26:08.238837 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:08.238803    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9006427-c201-4dd6-b693-d9599c28acdc-maas-api-service-ca\") pod \"d9006427-c201-4dd6-b693-d9599c28acdc\" (UID: \"d9006427-c201-4dd6-b693-d9599c28acdc\") "
Sep 10 21:26:08.239226 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:08.239200    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9006427-c201-4dd6-b693-d9599c28acdc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d9006427-c201-4dd6-b693-d9599c28acdc" (UID: "d9006427-c201-4dd6-b693-d9599c28acdc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:26:08.339946 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:08.339850    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9006427-c201-4dd6-b693-d9599c28acdc-maas-api-service-ca\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:26:09.008849 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:09.008815    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d24c8a-rl9xl" event={"ID":"d9006427-c201-4dd6-b693-d9599c28acdc","Type":"ContainerDied","Data":"d58bf9f19aaf8716e64be399cce35b9cb5bfc1473dbefa71b2bf73ad74fd4644"}
Sep 10 21:26:09.009015 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:09.008854    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d24c8a-rl9xl"
Sep 10 21:26:09.009015 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:09.008864    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d58bf9f19aaf8716e64be399cce35b9cb5bfc1473dbefa71b2bf73ad74fd4644"
Sep 10 21:26:11.700298 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:11.700264    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d24c8a-rl9xl"]
Sep 10 21:26:11.702668 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:11.702642    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d24c8a-rl9xl"]
Sep 10 21:26:12.840573 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:12.840543    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9006427-c201-4dd6-b693-d9599c28acdc" path="/var/lib/kubelet/pods/d9006427-c201-4dd6-b693-d9599c28acdc/volumes"
Sep 10 21:26:18.628807 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:18.628773    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"]
Sep 10 21:26:18.629285 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:18.628985    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh" podUID="294baf3a-2b92-423c-92ac-b90dd2eeaf4e" containerName="istio-proxy" containerID="cri-o://372f8d094b0c2ceca6ba2ab354c8fb73da749105d9ff0e9651b30c040f5c9af9" gracePeriod=30
Sep 10 21:26:20.703437 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:20.703384    2584 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 21:26:20.703817 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:20.703480    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh" podUID="294baf3a-2b92-423c-92ac-b90dd2eeaf4e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 21:26:22.449506 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:22.449467    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3bbf7d-qkhlg"]
Sep 10 21:26:22.449934 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:22.449890    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9006427-c201-4dd6-b693-d9599c28acdc" containerName="revoke-keys"
Sep 10 21:26:22.449934 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:22.449903    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9006427-c201-4dd6-b693-d9599c28acdc" containerName="revoke-keys"
Sep 10 21:26:22.450024 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:22.449960    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9006427-c201-4dd6-b693-d9599c28acdc" containerName="revoke-keys"
Sep 10 21:26:22.453295 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:22.453277    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3bbf7d-qkhlg"
Sep 10 21:26:22.455842 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:22.455819    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4gs5\""
Sep 10 21:26:22.455955 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:22.455845    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:26:22.467414 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:22.467385    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3bbf7d-qkhlg"]
Sep 10 21:26:22.490668 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:22.490635    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/983886f9-96c0-43aa-b942-729c6e000726-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3bbf7d-qkhlg\" (UID: \"983886f9-96c0-43aa-b942-729c6e000726\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3bbf7d-qkhlg"
Sep 10 21:26:22.592166 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:22.592107    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/983886f9-96c0-43aa-b942-729c6e000726-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3bbf7d-qkhlg\" (UID: \"983886f9-96c0-43aa-b942-729c6e000726\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3bbf7d-qkhlg"
Sep 10 21:26:22.592813 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:22.592793    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/983886f9-96c0-43aa-b942-729c6e000726-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3bbf7d-qkhlg\" (UID: \"983886f9-96c0-43aa-b942-729c6e000726\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3bbf7d-qkhlg"
Sep 10 21:26:22.763526 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:22.763437    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3bbf7d-qkhlg"
Sep 10 21:26:23.036843 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.036786    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3bbf7d-qkhlg"]
Sep 10 21:26:23.039192 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:26:23.039170    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983886f9_96c0_43aa_b942_729c6e000726.slice/crio-a96f0544c758cfef419156e1a39d00555a515a491bd8435b741e69337a8284b5 WatchSource:0}: Error finding container a96f0544c758cfef419156e1a39d00555a515a491bd8435b741e69337a8284b5: Status 404 returned error can't find the container with id a96f0544c758cfef419156e1a39d00555a515a491bd8435b741e69337a8284b5
Sep 10 21:26:23.086770 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.086742    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3bbf7d-qkhlg" event={"ID":"983886f9-96c0-43aa-b942-729c6e000726","Type":"ContainerStarted","Data":"a96f0544c758cfef419156e1a39d00555a515a491bd8435b741e69337a8284b5"}
Sep 10 21:26:23.875935 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.875908    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:26:23.905332 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.905300    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istio-envoy\") pod \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") "
Sep 10 21:26:23.905567 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.905345    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istio-podinfo\") pod \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") "
Sep 10 21:26:23.905567 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.905386    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istio-token\") pod \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") "
Sep 10 21:26:23.905567 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.905452    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-workload-certs\") pod \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") "
Sep 10 21:26:23.905567 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.905502    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istio-data\") pod \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") "
Sep 10 21:26:23.905810 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.905577    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-workload-socket\") pod \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") "
Sep 10 21:26:23.905810 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.905629    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-credential-socket\") pod \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") "
Sep 10 21:26:23.905810 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.905665    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjqt7\" (UniqueName: \"kubernetes.io/projected/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-kube-api-access-sjqt7\") pod \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") "
Sep 10 21:26:23.905810 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.905743    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istiod-ca-cert\") pod \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\" (UID: \"294baf3a-2b92-423c-92ac-b90dd2eeaf4e\") "
Sep 10 21:26:23.906538 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.905978    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "294baf3a-2b92-423c-92ac-b90dd2eeaf4e" (UID: "294baf3a-2b92-423c-92ac-b90dd2eeaf4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:26:23.906538 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.906079    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istio-data" (OuterVolumeSpecName: "istio-data") pod "294baf3a-2b92-423c-92ac-b90dd2eeaf4e" (UID: "294baf3a-2b92-423c-92ac-b90dd2eeaf4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:26:23.906538 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.906124    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "294baf3a-2b92-423c-92ac-b90dd2eeaf4e" (UID: "294baf3a-2b92-423c-92ac-b90dd2eeaf4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:26:23.906538 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.906243    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "294baf3a-2b92-423c-92ac-b90dd2eeaf4e" (UID: "294baf3a-2b92-423c-92ac-b90dd2eeaf4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:26:23.906776 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.906678    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-workload-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:26:23.906776 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.906702    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-credential-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:26:23.906776 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.906716    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-workload-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:26:23.906776 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.906731    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istio-data\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:26:23.906996 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.906883    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "294baf3a-2b92-423c-92ac-b90dd2eeaf4e" (UID: "294baf3a-2b92-423c-92ac-b90dd2eeaf4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:26:23.907845 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.907823    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "294baf3a-2b92-423c-92ac-b90dd2eeaf4e" (UID: "294baf3a-2b92-423c-92ac-b90dd2eeaf4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:26:23.908040 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.908024    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istio-token" (OuterVolumeSpecName: "istio-token") pod "294baf3a-2b92-423c-92ac-b90dd2eeaf4e" (UID: "294baf3a-2b92-423c-92ac-b90dd2eeaf4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:26:23.908121 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.908108    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "294baf3a-2b92-423c-92ac-b90dd2eeaf4e" (UID: "294baf3a-2b92-423c-92ac-b90dd2eeaf4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:26:23.908751 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:23.908725    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-kube-api-access-sjqt7" (OuterVolumeSpecName: "kube-api-access-sjqt7") pod "294baf3a-2b92-423c-92ac-b90dd2eeaf4e" (UID: "294baf3a-2b92-423c-92ac-b90dd2eeaf4e"). InnerVolumeSpecName "kube-api-access-sjqt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:26:24.007626 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:24.007534    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjqt7\" (UniqueName: \"kubernetes.io/projected/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-kube-api-access-sjqt7\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:26:24.007626 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:24.007567    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istiod-ca-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:26:24.007626 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:24.007578    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istio-envoy\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:26:24.007626 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:24.007586    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istio-podinfo\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:26:24.007626 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:24.007593    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/294baf3a-2b92-423c-92ac-b90dd2eeaf4e-istio-token\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:26:24.093941 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:24.093904    2584 generic.go:356] "Generic (PLEG): container finished" podID="294baf3a-2b92-423c-92ac-b90dd2eeaf4e" containerID="372f8d094b0c2ceca6ba2ab354c8fb73da749105d9ff0e9651b30c040f5c9af9" exitCode=0
Sep 10 21:26:24.094134 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:24.093952    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"
Sep 10 21:26:24.094134 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:24.093991    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh" event={"ID":"294baf3a-2b92-423c-92ac-b90dd2eeaf4e","Type":"ContainerDied","Data":"372f8d094b0c2ceca6ba2ab354c8fb73da749105d9ff0e9651b30c040f5c9af9"}
Sep 10 21:26:24.094134 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:24.094041    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh" event={"ID":"294baf3a-2b92-423c-92ac-b90dd2eeaf4e","Type":"ContainerDied","Data":"fdb218d024ec784acf2ec321ed8fef7b2d249527e327847b2e9602d3d4be17b7"}
Sep 10 21:26:24.094134 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:24.094064    2584 scope.go:117] "RemoveContainer" containerID="372f8d094b0c2ceca6ba2ab354c8fb73da749105d9ff0e9651b30c040f5c9af9"
Sep 10 21:26:24.095783 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:24.095760    2584 generic.go:356] "Generic (PLEG): container finished" podID="983886f9-96c0-43aa-b942-729c6e000726" containerID="637e81f526a373795e1fe317c27b8b341cec40f18788396af0925983395c90fa" exitCode=0
Sep 10 21:26:24.095900 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:24.095850    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3bbf7d-qkhlg" event={"ID":"983886f9-96c0-43aa-b942-729c6e000726","Type":"ContainerDied","Data":"637e81f526a373795e1fe317c27b8b341cec40f18788396af0925983395c90fa"}
Sep 10 21:26:24.104150 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:24.104132    2584 scope.go:117] "RemoveContainer" containerID="2a550349126afc180c814f8f55f2c5f9fdaae11af5b0d79e8939646e6a1b9a1b"
Sep 10 21:26:24.112385 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:24.112364    2584 scope.go:117] "RemoveContainer" containerID="372f8d094b0c2ceca6ba2ab354c8fb73da749105d9ff0e9651b30c040f5c9af9"
Sep 10 21:26:24.112795 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:26:24.112765    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"372f8d094b0c2ceca6ba2ab354c8fb73da749105d9ff0e9651b30c040f5c9af9\": container with ID starting with 372f8d094b0c2ceca6ba2ab354c8fb73da749105d9ff0e9651b30c040f5c9af9 not found: ID does not exist" containerID="372f8d094b0c2ceca6ba2ab354c8fb73da749105d9ff0e9651b30c040f5c9af9"
Sep 10 21:26:24.112921 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:24.112806    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"372f8d094b0c2ceca6ba2ab354c8fb73da749105d9ff0e9651b30c040f5c9af9"} err="failed to get container status \"372f8d094b0c2ceca6ba2ab354c8fb73da749105d9ff0e9651b30c040f5c9af9\": rpc error: code = NotFound desc = could not find container \"372f8d094b0c2ceca6ba2ab354c8fb73da749105d9ff0e9651b30c040f5c9af9\": container with ID starting with 372f8d094b0c2ceca6ba2ab354c8fb73da749105d9ff0e9651b30c040f5c9af9 not found: ID does not exist"
Sep 10 21:26:24.112921 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:24.112882    2584 scope.go:117] "RemoveContainer" containerID="2a550349126afc180c814f8f55f2c5f9fdaae11af5b0d79e8939646e6a1b9a1b"
Sep 10 21:26:24.113188 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:26:24.113165    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a550349126afc180c814f8f55f2c5f9fdaae11af5b0d79e8939646e6a1b9a1b\": container with ID starting with 2a550349126afc180c814f8f55f2c5f9fdaae11af5b0d79e8939646e6a1b9a1b not found: ID does not exist" containerID="2a550349126afc180c814f8f55f2c5f9fdaae11af5b0d79e8939646e6a1b9a1b"
Sep 10 21:26:24.113324 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:24.113194    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a550349126afc180c814f8f55f2c5f9fdaae11af5b0d79e8939646e6a1b9a1b"} err="failed to get container status \"2a550349126afc180c814f8f55f2c5f9fdaae11af5b0d79e8939646e6a1b9a1b\": rpc error: code = NotFound desc = could not find container \"2a550349126afc180c814f8f55f2c5f9fdaae11af5b0d79e8939646e6a1b9a1b\": container with ID starting with 2a550349126afc180c814f8f55f2c5f9fdaae11af5b0d79e8939646e6a1b9a1b not found: ID does not exist"
Sep 10 21:26:24.126651 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:24.126622    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"]
Sep 10 21:26:24.132918 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:24.132883    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d24c8a-openshift-default-5c66688cfb-hnzhh"]
Sep 10 21:26:24.843569 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:24.843529    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="294baf3a-2b92-423c-92ac-b90dd2eeaf4e" path="/var/lib/kubelet/pods/294baf3a-2b92-423c-92ac-b90dd2eeaf4e/volumes"
Sep 10 21:26:25.234260 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:25.234231    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3bbf7d-qkhlg"
Sep 10 21:26:25.318810 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:25.318779    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/983886f9-96c0-43aa-b942-729c6e000726-maas-api-service-ca\") pod \"983886f9-96c0-43aa-b942-729c6e000726\" (UID: \"983886f9-96c0-43aa-b942-729c6e000726\") "
Sep 10 21:26:25.319168 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:25.319132    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/983886f9-96c0-43aa-b942-729c6e000726-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "983886f9-96c0-43aa-b942-729c6e000726" (UID: "983886f9-96c0-43aa-b942-729c6e000726"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:26:25.420439 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:25.420388    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/983886f9-96c0-43aa-b942-729c6e000726-maas-api-service-ca\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:26:26.108332 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:26.108300    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3bbf7d-qkhlg"
Sep 10 21:26:26.108507 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:26.108338    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3bbf7d-qkhlg" event={"ID":"983886f9-96c0-43aa-b942-729c6e000726","Type":"ContainerDied","Data":"a96f0544c758cfef419156e1a39d00555a515a491bd8435b741e69337a8284b5"}
Sep 10 21:26:26.108507 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:26.108382    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a96f0544c758cfef419156e1a39d00555a515a491bd8435b741e69337a8284b5"
Sep 10 21:26:38.200315 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:38.200280    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3bbf7d-qkhlg"]
Sep 10 21:26:38.202833 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:38.202809    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3bbf7d-qkhlg"]
Sep 10 21:26:38.840926 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:38.840890    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="983886f9-96c0-43aa-b942-729c6e000726" path="/var/lib/kubelet/pods/983886f9-96c0-43aa-b942-729c6e000726/volumes"
Sep 10 21:26:43.589165 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:43.589127    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"]
Sep 10 21:26:58.608343 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:58.608303    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg" podUID="e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd" containerName="main" containerID="cri-o://abdd0ad584ec384c90d8d0d313be411812bca45a0633bbde3291351d16c99c2b" gracePeriod=45
Sep 10 21:26:58.864952 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:58.864890    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:26:58.960615 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:58.960577    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-home\") pod \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\" (UID: \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\") "
Sep 10 21:26:58.960806 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:58.960642    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-dshm\") pod \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\" (UID: \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\") "
Sep 10 21:26:58.960806 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:58.960679    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-tmp-dir\") pod \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\" (UID: \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\") "
Sep 10 21:26:58.960806 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:58.960712    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-tls-certs\") pod \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\" (UID: \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\") "
Sep 10 21:26:58.960806 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:58.960783    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-model-cache\") pod \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\" (UID: \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\") "
Sep 10 21:26:58.961033 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:58.960838    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmqsh\" (UniqueName: \"kubernetes.io/projected/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-kube-api-access-wmqsh\") pod \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\" (UID: \"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd\") "
Sep 10 21:26:58.961033 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:58.960934    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-home" (OuterVolumeSpecName: "home") pod "e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd" (UID: "e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:26:58.961033 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:58.960951    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd" (UID: "e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:26:58.961174 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:58.961085    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd" (UID: "e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:26:58.961235 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:58.961217    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-tmp-dir\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:26:58.961235 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:58.961234    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-model-cache\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:26:58.961345 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:58.961243    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-home\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:26:58.962942 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:58.962919    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd" (UID: "e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:26:58.963018 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:58.962951    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-dshm" (OuterVolumeSpecName: "dshm") pod "e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd" (UID: "e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:26:58.963018 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:58.963008    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-kube-api-access-wmqsh" (OuterVolumeSpecName: "kube-api-access-wmqsh") pod "e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd" (UID: "e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd"). InnerVolumeSpecName "kube-api-access-wmqsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:26:59.062771 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:59.062733    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmqsh\" (UniqueName: \"kubernetes.io/projected/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-kube-api-access-wmqsh\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:26:59.062771 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:59.062764    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-dshm\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:26:59.062771 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:59.062774    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd-tls-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:26:59.281186 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:59.281154    2584 generic.go:356] "Generic (PLEG): container finished" podID="e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd" containerID="abdd0ad584ec384c90d8d0d313be411812bca45a0633bbde3291351d16c99c2b" exitCode=0
Sep 10 21:26:59.281369 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:59.281199    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg" event={"ID":"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd","Type":"ContainerDied","Data":"abdd0ad584ec384c90d8d0d313be411812bca45a0633bbde3291351d16c99c2b"}
Sep 10 21:26:59.281369 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:59.281202    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"
Sep 10 21:26:59.281369 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:59.281225    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg" event={"ID":"e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd","Type":"ContainerDied","Data":"8937c86dcfff632716f048bc53cb4de852baca539f73a7466258e61d1a523c0c"}
Sep 10 21:26:59.281369 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:59.281242    2584 scope.go:117] "RemoveContainer" containerID="abdd0ad584ec384c90d8d0d313be411812bca45a0633bbde3291351d16c99c2b"
Sep 10 21:26:59.290224 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:59.290206    2584 scope.go:117] "RemoveContainer" containerID="abdd0ad584ec384c90d8d0d313be411812bca45a0633bbde3291351d16c99c2b"
Sep 10 21:26:59.290499 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:26:59.290479    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abdd0ad584ec384c90d8d0d313be411812bca45a0633bbde3291351d16c99c2b\": container with ID starting with abdd0ad584ec384c90d8d0d313be411812bca45a0633bbde3291351d16c99c2b not found: ID does not exist" containerID="abdd0ad584ec384c90d8d0d313be411812bca45a0633bbde3291351d16c99c2b"
Sep 10 21:26:59.290558 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:59.290510    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abdd0ad584ec384c90d8d0d313be411812bca45a0633bbde3291351d16c99c2b"} err="failed to get container status \"abdd0ad584ec384c90d8d0d313be411812bca45a0633bbde3291351d16c99c2b\": rpc error: code = NotFound desc = could not find container \"abdd0ad584ec384c90d8d0d313be411812bca45a0633bbde3291351d16c99c2b\": container with ID starting with abdd0ad584ec384c90d8d0d313be411812bca45a0633bbde3291351d16c99c2b not found: ID does not exist"
Sep 10 21:26:59.302007 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:59.301985    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"]
Sep 10 21:26:59.305482 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:26:59.305462    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-3bbf7d/ipp-route-3bbf7d-kserve-6bbdb87b79-s8krg"]
Sep 10 21:27:00.840717 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:00.840681    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd" path="/var/lib/kubelet/pods/e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd/volumes"
Sep 10 21:27:33.087819 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:33.087732    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"]
Sep 10 21:27:33.088352 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:33.088012    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv" podUID="31ef0c45-7695-4bfc-9979-ca253fba416e" containerName="istio-proxy" containerID="cri-o://fb338f4d9ecc69ffb83fcad5c11716d4d362dfd9b4501a9c425798b9a060b05a" gracePeriod=30
Sep 10 21:27:33.473464 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:33.473416    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a581b5-52xml"]
Sep 10 21:27:33.473840 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:33.473825    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="294baf3a-2b92-423c-92ac-b90dd2eeaf4e" containerName="istio-proxy"
Sep 10 21:27:33.473888 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:33.473843    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="294baf3a-2b92-423c-92ac-b90dd2eeaf4e" containerName="istio-proxy"
Sep 10 21:27:33.473888 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:33.473867    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd" containerName="main"
Sep 10 21:27:33.473888 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:33.473872    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd" containerName="main"
Sep 10 21:27:33.473888 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:33.473880    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="983886f9-96c0-43aa-b942-729c6e000726" containerName="revoke-keys"
Sep 10 21:27:33.473888 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:33.473885    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="983886f9-96c0-43aa-b942-729c6e000726" containerName="revoke-keys"
Sep 10 21:27:33.474046 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:33.473951    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="294baf3a-2b92-423c-92ac-b90dd2eeaf4e" containerName="istio-proxy"
Sep 10 21:27:33.474046 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:33.473960    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="983886f9-96c0-43aa-b942-729c6e000726" containerName="revoke-keys"
Sep 10 21:27:33.474046 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:33.473967    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9f9f346-eb8c-4f3c-9bc9-aa12edeec3fd" containerName="main"
Sep 10 21:27:33.476975 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:33.476955    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a581b5-52xml"
Sep 10 21:27:33.479199 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:33.479180    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4gs5\""
Sep 10 21:27:33.479302 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:33.479205    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:27:33.486216 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:33.486168    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a581b5-52xml"]
Sep 10 21:27:33.511604 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:33.511564    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf15784-677b-4ccb-8683-af7b9fecdba7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a581b5-52xml\" (UID: \"dbf15784-677b-4ccb-8683-af7b9fecdba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a581b5-52xml"
Sep 10 21:27:33.612896 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:33.612858    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf15784-677b-4ccb-8683-af7b9fecdba7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a581b5-52xml\" (UID: \"dbf15784-677b-4ccb-8683-af7b9fecdba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a581b5-52xml"
Sep 10 21:27:33.613509 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:33.613488    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf15784-677b-4ccb-8683-af7b9fecdba7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a581b5-52xml\" (UID: \"dbf15784-677b-4ccb-8683-af7b9fecdba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a581b5-52xml"
Sep 10 21:27:33.788696 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:33.788599    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a581b5-52xml"
Sep 10 21:27:34.061445 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:34.061346    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a581b5-52xml"]
Sep 10 21:27:34.064489 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:27:34.064458    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf15784_677b_4ccb_8683_af7b9fecdba7.slice/crio-0f9c3af786f8e0886d640b3375b8115e6be6a6987fce2d66a91971e5b51f8616 WatchSource:0}: Error finding container 0f9c3af786f8e0886d640b3375b8115e6be6a6987fce2d66a91971e5b51f8616: Status 404 returned error can't find the container with id 0f9c3af786f8e0886d640b3375b8115e6be6a6987fce2d66a91971e5b51f8616
Sep 10 21:27:34.467926 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:34.467884    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a581b5-52xml" event={"ID":"dbf15784-677b-4ccb-8683-af7b9fecdba7","Type":"ContainerStarted","Data":"3d827a71ad9fb3d84da4617f990b8bf7a57717a8632ce840c081e518825b3c59"}
Sep 10 21:27:34.468347 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:34.467955    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a581b5-52xml" event={"ID":"dbf15784-677b-4ccb-8683-af7b9fecdba7","Type":"ContainerStarted","Data":"0f9c3af786f8e0886d640b3375b8115e6be6a6987fce2d66a91971e5b51f8616"}
Sep 10 21:27:34.487977 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:34.487925    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a581b5-52xml" podStartSLOduration=1.4879081570000001 podStartE2EDuration="1.487908157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:27:34.486369141 +0000 UTC m=+1714.151393941" watchObservedRunningTime="2026-09-10 21:27:34.487908157 +0000 UTC m=+1714.152932986"
Sep 10 21:27:35.474609 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:35.474571    2584 generic.go:356] "Generic (PLEG): container finished" podID="dbf15784-677b-4ccb-8683-af7b9fecdba7" containerID="3d827a71ad9fb3d84da4617f990b8bf7a57717a8632ce840c081e518825b3c59" exitCode=0
Sep 10 21:27:35.475076 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:35.474657    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a581b5-52xml" event={"ID":"dbf15784-677b-4ccb-8683-af7b9fecdba7","Type":"ContainerDied","Data":"3d827a71ad9fb3d84da4617f990b8bf7a57717a8632ce840c081e518825b3c59"}
Sep 10 21:27:36.612593 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:36.612568    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a581b5-52xml"
Sep 10 21:27:36.643946 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:36.643919    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf15784-677b-4ccb-8683-af7b9fecdba7-maas-api-service-ca\") pod \"dbf15784-677b-4ccb-8683-af7b9fecdba7\" (UID: \"dbf15784-677b-4ccb-8683-af7b9fecdba7\") "
Sep 10 21:27:36.644304 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:36.644281    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbf15784-677b-4ccb-8683-af7b9fecdba7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dbf15784-677b-4ccb-8683-af7b9fecdba7" (UID: "dbf15784-677b-4ccb-8683-af7b9fecdba7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:27:36.644353 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:36.644346    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf15784-677b-4ccb-8683-af7b9fecdba7-maas-api-service-ca\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:27:37.485937 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:37.485906    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a581b5-52xml"
Sep 10 21:27:37.485937 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:37.485927    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a581b5-52xml" event={"ID":"dbf15784-677b-4ccb-8683-af7b9fecdba7","Type":"ContainerDied","Data":"0f9c3af786f8e0886d640b3375b8115e6be6a6987fce2d66a91971e5b51f8616"}
Sep 10 21:27:37.486169 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:37.485980    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f9c3af786f8e0886d640b3375b8115e6be6a6987fce2d66a91971e5b51f8616"
Sep 10 21:27:38.331106 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.331081    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:27:38.361157 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.361079    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31ef0c45-7695-4bfc-9979-ca253fba416e-istio-podinfo\") pod \"31ef0c45-7695-4bfc-9979-ca253fba416e\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") "
Sep 10 21:27:38.361157 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.361119    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-istio-envoy\") pod \"31ef0c45-7695-4bfc-9979-ca253fba416e\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") "
Sep 10 21:27:38.361157 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.361156    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31ef0c45-7695-4bfc-9979-ca253fba416e-istio-token\") pod \"31ef0c45-7695-4bfc-9979-ca253fba416e\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") "
Sep 10 21:27:38.361418 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.361205    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-istio-data\") pod \"31ef0c45-7695-4bfc-9979-ca253fba416e\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") "
Sep 10 21:27:38.361418 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.361236    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-credential-socket\") pod \"31ef0c45-7695-4bfc-9979-ca253fba416e\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") "
Sep 10 21:27:38.361418 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.361279    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31ef0c45-7695-4bfc-9979-ca253fba416e-istiod-ca-cert\") pod \"31ef0c45-7695-4bfc-9979-ca253fba416e\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") "
Sep 10 21:27:38.361418 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.361316    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-workload-socket\") pod \"31ef0c45-7695-4bfc-9979-ca253fba416e\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") "
Sep 10 21:27:38.361418 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.361373    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8khzw\" (UniqueName: \"kubernetes.io/projected/31ef0c45-7695-4bfc-9979-ca253fba416e-kube-api-access-8khzw\") pod \"31ef0c45-7695-4bfc-9979-ca253fba416e\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") "
Sep 10 21:27:38.361418 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.361401    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-workload-certs\") pod \"31ef0c45-7695-4bfc-9979-ca253fba416e\" (UID: \"31ef0c45-7695-4bfc-9979-ca253fba416e\") "
Sep 10 21:27:38.361778 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.361517    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-istio-data" (OuterVolumeSpecName: "istio-data") pod "31ef0c45-7695-4bfc-9979-ca253fba416e" (UID: "31ef0c45-7695-4bfc-9979-ca253fba416e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:27:38.361778 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.361616    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "31ef0c45-7695-4bfc-9979-ca253fba416e" (UID: "31ef0c45-7695-4bfc-9979-ca253fba416e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:27:38.361778 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.361758    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "31ef0c45-7695-4bfc-9979-ca253fba416e" (UID: "31ef0c45-7695-4bfc-9979-ca253fba416e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:27:38.361950 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.361802    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-istio-data\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:27:38.361950 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.361821    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-workload-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:27:38.361950 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.361852    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ef0c45-7695-4bfc-9979-ca253fba416e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "31ef0c45-7695-4bfc-9979-ca253fba416e" (UID: "31ef0c45-7695-4bfc-9979-ca253fba416e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:27:38.361950 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.361923    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "31ef0c45-7695-4bfc-9979-ca253fba416e" (UID: "31ef0c45-7695-4bfc-9979-ca253fba416e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:27:38.363482 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.363450    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "31ef0c45-7695-4bfc-9979-ca253fba416e" (UID: "31ef0c45-7695-4bfc-9979-ca253fba416e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:27:38.363596 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.363567    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ef0c45-7695-4bfc-9979-ca253fba416e-istio-token" (OuterVolumeSpecName: "istio-token") pod "31ef0c45-7695-4bfc-9979-ca253fba416e" (UID: "31ef0c45-7695-4bfc-9979-ca253fba416e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:27:38.363992 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.363971    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ef0c45-7695-4bfc-9979-ca253fba416e-kube-api-access-8khzw" (OuterVolumeSpecName: "kube-api-access-8khzw") pod "31ef0c45-7695-4bfc-9979-ca253fba416e" (UID: "31ef0c45-7695-4bfc-9979-ca253fba416e"). InnerVolumeSpecName "kube-api-access-8khzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:27:38.363992 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.363986    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31ef0c45-7695-4bfc-9979-ca253fba416e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "31ef0c45-7695-4bfc-9979-ca253fba416e" (UID: "31ef0c45-7695-4bfc-9979-ca253fba416e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:27:38.462649 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.462617    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-workload-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:27:38.462649 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.462644    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31ef0c45-7695-4bfc-9979-ca253fba416e-istio-podinfo\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:27:38.462649 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.462652    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-istio-envoy\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:27:38.462649 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.462660    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31ef0c45-7695-4bfc-9979-ca253fba416e-istio-token\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:27:38.462904 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.462668    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ef0c45-7695-4bfc-9979-ca253fba416e-credential-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:27:38.462904 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.462678    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31ef0c45-7695-4bfc-9979-ca253fba416e-istiod-ca-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:27:38.462904 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.462686    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8khzw\" (UniqueName: \"kubernetes.io/projected/31ef0c45-7695-4bfc-9979-ca253fba416e-kube-api-access-8khzw\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:27:38.493148 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.493114    2584 generic.go:356] "Generic (PLEG): container finished" podID="31ef0c45-7695-4bfc-9979-ca253fba416e" containerID="fb338f4d9ecc69ffb83fcad5c11716d4d362dfd9b4501a9c425798b9a060b05a" exitCode=0
Sep 10 21:27:38.493283 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.493157    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv" event={"ID":"31ef0c45-7695-4bfc-9979-ca253fba416e","Type":"ContainerDied","Data":"fb338f4d9ecc69ffb83fcad5c11716d4d362dfd9b4501a9c425798b9a060b05a"}
Sep 10 21:27:38.493283 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.493187    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"
Sep 10 21:27:38.493283 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.493209    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv" event={"ID":"31ef0c45-7695-4bfc-9979-ca253fba416e","Type":"ContainerDied","Data":"3beb96176b728ab5cb5041573fef8ed5120640505ee4a8ff171661ad6d462705"}
Sep 10 21:27:38.493283 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.493238    2584 scope.go:117] "RemoveContainer" containerID="fb338f4d9ecc69ffb83fcad5c11716d4d362dfd9b4501a9c425798b9a060b05a"
Sep 10 21:27:38.502559 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.502539    2584 scope.go:117] "RemoveContainer" containerID="88d9c18a50ea165b2ad4cb79592921632d3dd8162319d46b09e0ef932f281632"
Sep 10 21:27:38.510307 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.510289    2584 scope.go:117] "RemoveContainer" containerID="fb338f4d9ecc69ffb83fcad5c11716d4d362dfd9b4501a9c425798b9a060b05a"
Sep 10 21:27:38.510607 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:27:38.510590    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb338f4d9ecc69ffb83fcad5c11716d4d362dfd9b4501a9c425798b9a060b05a\": container with ID starting with fb338f4d9ecc69ffb83fcad5c11716d4d362dfd9b4501a9c425798b9a060b05a not found: ID does not exist" containerID="fb338f4d9ecc69ffb83fcad5c11716d4d362dfd9b4501a9c425798b9a060b05a"
Sep 10 21:27:38.510678 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.510616    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb338f4d9ecc69ffb83fcad5c11716d4d362dfd9b4501a9c425798b9a060b05a"} err="failed to get container status \"fb338f4d9ecc69ffb83fcad5c11716d4d362dfd9b4501a9c425798b9a060b05a\": rpc error: code = NotFound desc = could not find container \"fb338f4d9ecc69ffb83fcad5c11716d4d362dfd9b4501a9c425798b9a060b05a\": container with ID starting with fb338f4d9ecc69ffb83fcad5c11716d4d362dfd9b4501a9c425798b9a060b05a not found: ID does not exist"
Sep 10 21:27:38.510678 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.510633    2584 scope.go:117] "RemoveContainer" containerID="88d9c18a50ea165b2ad4cb79592921632d3dd8162319d46b09e0ef932f281632"
Sep 10 21:27:38.510871 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:27:38.510851    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d9c18a50ea165b2ad4cb79592921632d3dd8162319d46b09e0ef932f281632\": container with ID starting with 88d9c18a50ea165b2ad4cb79592921632d3dd8162319d46b09e0ef932f281632 not found: ID does not exist" containerID="88d9c18a50ea165b2ad4cb79592921632d3dd8162319d46b09e0ef932f281632"
Sep 10 21:27:38.510913 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.510878    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d9c18a50ea165b2ad4cb79592921632d3dd8162319d46b09e0ef932f281632"} err="failed to get container status \"88d9c18a50ea165b2ad4cb79592921632d3dd8162319d46b09e0ef932f281632\": rpc error: code = NotFound desc = could not find container \"88d9c18a50ea165b2ad4cb79592921632d3dd8162319d46b09e0ef932f281632\": container with ID starting with 88d9c18a50ea165b2ad4cb79592921632d3dd8162319d46b09e0ef932f281632 not found: ID does not exist"
Sep 10 21:27:38.521123 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.521101    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"]
Sep 10 21:27:38.525436 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.525399    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3bbf7d-openshift-default-5f4d7bf4-xw2zv"]
Sep 10 21:27:38.840804 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:38.840772    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31ef0c45-7695-4bfc-9979-ca253fba416e" path="/var/lib/kubelet/pods/31ef0c45-7695-4bfc-9979-ca253fba416e/volumes"
Sep 10 21:27:44.650731 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:44.650693    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a581b5-52xml"]
Sep 10 21:27:44.654949 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:44.654923    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a581b5-52xml"]
Sep 10 21:27:44.840784 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:44.840703    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbf15784-677b-4ccb-8683-af7b9fecdba7" path="/var/lib/kubelet/pods/dbf15784-677b-4ccb-8683-af7b9fecdba7/volumes"
Sep 10 21:27:51.326874 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:51.326835    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"]
Sep 10 21:27:51.327354 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:51.327112    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs" podUID="8018c9ab-ec1a-49c6-8664-175138d0d6e0" containerName="istio-proxy" containerID="cri-o://bfa5402dc16e2b7d7b7e70f362e0fe191e69c7e922853453b9210516441df45a" gracePeriod=30
Sep 10 21:27:51.771187 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:51.771152    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0ac302-nzpnf"]
Sep 10 21:27:51.771554 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:51.771540    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ef0c45-7695-4bfc-9979-ca253fba416e" containerName="istio-proxy"
Sep 10 21:27:51.771617 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:51.771556    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ef0c45-7695-4bfc-9979-ca253fba416e" containerName="istio-proxy"
Sep 10 21:27:51.771617 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:51.771572    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ef0c45-7695-4bfc-9979-ca253fba416e" containerName="istio-proxy"
Sep 10 21:27:51.771617 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:51.771578    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ef0c45-7695-4bfc-9979-ca253fba416e" containerName="istio-proxy"
Sep 10 21:27:51.771617 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:51.771590    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="294baf3a-2b92-423c-92ac-b90dd2eeaf4e" containerName="istio-proxy"
Sep 10 21:27:51.771617 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:51.771595    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="294baf3a-2b92-423c-92ac-b90dd2eeaf4e" containerName="istio-proxy"
Sep 10 21:27:51.771617 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:51.771616    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbf15784-677b-4ccb-8683-af7b9fecdba7" containerName="revoke-keys"
Sep 10 21:27:51.771800 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:51.771623    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbf15784-677b-4ccb-8683-af7b9fecdba7" containerName="revoke-keys"
Sep 10 21:27:51.771800 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:51.771696    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ef0c45-7695-4bfc-9979-ca253fba416e" containerName="istio-proxy"
Sep 10 21:27:51.771800 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:51.771707    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="294baf3a-2b92-423c-92ac-b90dd2eeaf4e" containerName="istio-proxy"
Sep 10 21:27:51.771800 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:51.771714    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbf15784-677b-4ccb-8683-af7b9fecdba7" containerName="revoke-keys"
Sep 10 21:27:51.776476 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:51.776459    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0ac302-nzpnf"
Sep 10 21:27:51.778725 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:51.778704    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4gs5\""
Sep 10 21:27:51.778828 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:51.778745    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:27:51.782321 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:51.782296    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0ac302-nzpnf"]
Sep 10 21:27:51.898001 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:51.897965    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ee739e6-9533-45e5-abf4-a4df4992c409-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0ac302-nzpnf\" (UID: \"6ee739e6-9533-45e5-abf4-a4df4992c409\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0ac302-nzpnf"
Sep 10 21:27:51.998622 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:51.998583    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ee739e6-9533-45e5-abf4-a4df4992c409-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0ac302-nzpnf\" (UID: \"6ee739e6-9533-45e5-abf4-a4df4992c409\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0ac302-nzpnf"
Sep 10 21:27:51.999184 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:51.999164    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ee739e6-9533-45e5-abf4-a4df4992c409-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0ac302-nzpnf\" (UID: \"6ee739e6-9533-45e5-abf4-a4df4992c409\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0ac302-nzpnf"
Sep 10 21:27:52.088934 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:52.088857    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0ac302-nzpnf"
Sep 10 21:27:52.358642 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:52.358616    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0ac302-nzpnf"]
Sep 10 21:27:52.361057 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:27:52.361034    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee739e6_9533_45e5_abf4_a4df4992c409.slice/crio-3ae9a0fc843e9cbb275b4951c9db053d1628434108da88c8e18c47f92d5f4222 WatchSource:0}: Error finding container 3ae9a0fc843e9cbb275b4951c9db053d1628434108da88c8e18c47f92d5f4222: Status 404 returned error can't find the container with id 3ae9a0fc843e9cbb275b4951c9db053d1628434108da88c8e18c47f92d5f4222
Sep 10 21:27:52.567474 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:52.567440    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0ac302-nzpnf" event={"ID":"6ee739e6-9533-45e5-abf4-a4df4992c409","Type":"ContainerStarted","Data":"3ae9a0fc843e9cbb275b4951c9db053d1628434108da88c8e18c47f92d5f4222"}
Sep 10 21:27:53.573980 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:53.573942    2584 generic.go:356] "Generic (PLEG): container finished" podID="6ee739e6-9533-45e5-abf4-a4df4992c409" containerID="c1d618ae6bc81fc3b1645740e2c054c9bad07c26bd7a503c8a08170211e4ad91" exitCode=0
Sep 10 21:27:53.573980 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:53.573972    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0ac302-nzpnf" event={"ID":"6ee739e6-9533-45e5-abf4-a4df4992c409","Type":"ContainerDied","Data":"c1d618ae6bc81fc3b1645740e2c054c9bad07c26bd7a503c8a08170211e4ad91"}
Sep 10 21:27:53.656374 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:53.656338    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 21:27:53.656593 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:53.656405    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs" podUID="8018c9ab-ec1a-49c6-8664-175138d0d6e0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 21:27:54.707768 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:54.707740    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0ac302-nzpnf"
Sep 10 21:27:54.824955 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:54.824920    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ee739e6-9533-45e5-abf4-a4df4992c409-maas-api-service-ca\") pod \"6ee739e6-9533-45e5-abf4-a4df4992c409\" (UID: \"6ee739e6-9533-45e5-abf4-a4df4992c409\") "
Sep 10 21:27:54.825265 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:54.825243    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ee739e6-9533-45e5-abf4-a4df4992c409-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ee739e6-9533-45e5-abf4-a4df4992c409" (UID: "6ee739e6-9533-45e5-abf4-a4df4992c409"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:27:54.925943 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:54.925908    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ee739e6-9533-45e5-abf4-a4df4992c409-maas-api-service-ca\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:27:55.586629 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:55.586595    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0ac302-nzpnf"
Sep 10 21:27:55.586835 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:55.586628    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0ac302-nzpnf" event={"ID":"6ee739e6-9533-45e5-abf4-a4df4992c409","Type":"ContainerDied","Data":"3ae9a0fc843e9cbb275b4951c9db053d1628434108da88c8e18c47f92d5f4222"}
Sep 10 21:27:55.586835 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:55.586683    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ae9a0fc843e9cbb275b4951c9db053d1628434108da88c8e18c47f92d5f4222"
Sep 10 21:27:56.582893 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.582868    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:27:56.593147 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.593083    2584 generic.go:356] "Generic (PLEG): container finished" podID="8018c9ab-ec1a-49c6-8664-175138d0d6e0" containerID="bfa5402dc16e2b7d7b7e70f362e0fe191e69c7e922853453b9210516441df45a" exitCode=0
Sep 10 21:27:56.593147 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.593130    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"
Sep 10 21:27:56.593306 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.593164    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs" event={"ID":"8018c9ab-ec1a-49c6-8664-175138d0d6e0","Type":"ContainerDied","Data":"bfa5402dc16e2b7d7b7e70f362e0fe191e69c7e922853453b9210516441df45a"}
Sep 10 21:27:56.593306 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.593213    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs" event={"ID":"8018c9ab-ec1a-49c6-8664-175138d0d6e0","Type":"ContainerDied","Data":"2aee16d840a2e716154ce30c5c9e5c2f948dd4269a4d472f425e5eb4e2c69550"}
Sep 10 21:27:56.593306 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.593238    2584 scope.go:117] "RemoveContainer" containerID="bfa5402dc16e2b7d7b7e70f362e0fe191e69c7e922853453b9210516441df45a"
Sep 10 21:27:56.601621 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.601597    2584 scope.go:117] "RemoveContainer" containerID="7309d5da08e84f65895eb60edf138c5e64c636c820f813f97addc89c6d7d5471"
Sep 10 21:27:56.612283 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.612263    2584 scope.go:117] "RemoveContainer" containerID="bfa5402dc16e2b7d7b7e70f362e0fe191e69c7e922853453b9210516441df45a"
Sep 10 21:27:56.612623 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:27:56.612600    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfa5402dc16e2b7d7b7e70f362e0fe191e69c7e922853453b9210516441df45a\": container with ID starting with bfa5402dc16e2b7d7b7e70f362e0fe191e69c7e922853453b9210516441df45a not found: ID does not exist" containerID="bfa5402dc16e2b7d7b7e70f362e0fe191e69c7e922853453b9210516441df45a"
Sep 10 21:27:56.612731 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.612629    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfa5402dc16e2b7d7b7e70f362e0fe191e69c7e922853453b9210516441df45a"} err="failed to get container status \"bfa5402dc16e2b7d7b7e70f362e0fe191e69c7e922853453b9210516441df45a\": rpc error: code = NotFound desc = could not find container \"bfa5402dc16e2b7d7b7e70f362e0fe191e69c7e922853453b9210516441df45a\": container with ID starting with bfa5402dc16e2b7d7b7e70f362e0fe191e69c7e922853453b9210516441df45a not found: ID does not exist"
Sep 10 21:27:56.612731 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.612645    2584 scope.go:117] "RemoveContainer" containerID="7309d5da08e84f65895eb60edf138c5e64c636c820f813f97addc89c6d7d5471"
Sep 10 21:27:56.612922 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:27:56.612902    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7309d5da08e84f65895eb60edf138c5e64c636c820f813f97addc89c6d7d5471\": container with ID starting with 7309d5da08e84f65895eb60edf138c5e64c636c820f813f97addc89c6d7d5471 not found: ID does not exist" containerID="7309d5da08e84f65895eb60edf138c5e64c636c820f813f97addc89c6d7d5471"
Sep 10 21:27:56.612995 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.612927    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7309d5da08e84f65895eb60edf138c5e64c636c820f813f97addc89c6d7d5471"} err="failed to get container status \"7309d5da08e84f65895eb60edf138c5e64c636c820f813f97addc89c6d7d5471\": rpc error: code = NotFound desc = could not find container \"7309d5da08e84f65895eb60edf138c5e64c636c820f813f97addc89c6d7d5471\": container with ID starting with 7309d5da08e84f65895eb60edf138c5e64c636c820f813f97addc89c6d7d5471 not found: ID does not exist"
Sep 10 21:27:56.742231 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.742203    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istio-data\") pod \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") "
Sep 10 21:27:56.742410 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.742260    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-workload-certs\") pod \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") "
Sep 10 21:27:56.742410 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.742305    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istiod-ca-cert\") pod \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") "
Sep 10 21:27:56.742410 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.742340    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x82rn\" (UniqueName: \"kubernetes.io/projected/8018c9ab-ec1a-49c6-8664-175138d0d6e0-kube-api-access-x82rn\") pod \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") "
Sep 10 21:27:56.742410 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.742396    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-credential-socket\") pod \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") "
Sep 10 21:27:56.742661 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.742459    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istio-envoy\") pod \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") "
Sep 10 21:27:56.742661 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.742485    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istio-data" (OuterVolumeSpecName: "istio-data") pod "8018c9ab-ec1a-49c6-8664-175138d0d6e0" (UID: "8018c9ab-ec1a-49c6-8664-175138d0d6e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:27:56.742661 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.742505    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-workload-socket\") pod \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") "
Sep 10 21:27:56.742661 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.742538    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istio-podinfo\") pod \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") "
Sep 10 21:27:56.742661 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.742577    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8018c9ab-ec1a-49c6-8664-175138d0d6e0" (UID: "8018c9ab-ec1a-49c6-8664-175138d0d6e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:27:56.742661 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.742588    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istio-token\") pod \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\" (UID: \"8018c9ab-ec1a-49c6-8664-175138d0d6e0\") "
Sep 10 21:27:56.742987 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.742710    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8018c9ab-ec1a-49c6-8664-175138d0d6e0" (UID: "8018c9ab-ec1a-49c6-8664-175138d0d6e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:27:56.742987 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.742743    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8018c9ab-ec1a-49c6-8664-175138d0d6e0" (UID: "8018c9ab-ec1a-49c6-8664-175138d0d6e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:27:56.742987 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.742942    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8018c9ab-ec1a-49c6-8664-175138d0d6e0" (UID: "8018c9ab-ec1a-49c6-8664-175138d0d6e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:27:56.743146 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.743004    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istio-data\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:27:56.743146 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.743020    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-workload-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:27:56.743146 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.743031    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istiod-ca-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:27:56.743146 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.743045    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-credential-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:27:56.744738 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.744707    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8018c9ab-ec1a-49c6-8664-175138d0d6e0" (UID: "8018c9ab-ec1a-49c6-8664-175138d0d6e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:27:56.744865 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.744750    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8018c9ab-ec1a-49c6-8664-175138d0d6e0" (UID: "8018c9ab-ec1a-49c6-8664-175138d0d6e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:27:56.744865 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.744755    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istio-token" (OuterVolumeSpecName: "istio-token") pod "8018c9ab-ec1a-49c6-8664-175138d0d6e0" (UID: "8018c9ab-ec1a-49c6-8664-175138d0d6e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:27:56.744865 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.744771    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8018c9ab-ec1a-49c6-8664-175138d0d6e0-kube-api-access-x82rn" (OuterVolumeSpecName: "kube-api-access-x82rn") pod "8018c9ab-ec1a-49c6-8664-175138d0d6e0" (UID: "8018c9ab-ec1a-49c6-8664-175138d0d6e0"). InnerVolumeSpecName "kube-api-access-x82rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:27:56.844236 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.844137    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istio-podinfo\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:27:56.844236 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.844169    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istio-token\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:27:56.844236 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.844183    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x82rn\" (UniqueName: \"kubernetes.io/projected/8018c9ab-ec1a-49c6-8664-175138d0d6e0-kube-api-access-x82rn\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:27:56.844236 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.844194    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-istio-envoy\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:27:56.844236 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.844203    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8018c9ab-ec1a-49c6-8664-175138d0d6e0-workload-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:27:56.911300 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.911265    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"]
Sep 10 21:27:56.916697 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:56.916669    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a581b5-openshift-default-fb58bc7b-dz4bs"]
Sep 10 21:27:58.840497 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:27:58.840460    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8018c9ab-ec1a-49c6-8664-175138d0d6e0" path="/var/lib/kubelet/pods/8018c9ab-ec1a-49c6-8664-175138d0d6e0/volumes"
Sep 10 21:28:02.422312 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:02.422278    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0ac302-nzpnf"]
Sep 10 21:28:02.424981 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:02.424946    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0ac302-nzpnf"]
Sep 10 21:28:02.845588 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:02.840306    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ee739e6-9533-45e5-abf4-a4df4992c409" path="/var/lib/kubelet/pods/6ee739e6-9533-45e5-abf4-a4df4992c409/volumes"
Sep 10 21:28:09.303750 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:09.303712    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"]
Sep 10 21:28:09.304297 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:09.304008    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq" podUID="62491f9c-f6c8-484a-9488-7b38a70cd163" containerName="istio-proxy" containerID="cri-o://5798c08c7c62b3c3943755f9ec172086915bcead5b4274c3fc982c15b479cb0a" gracePeriod=30
Sep 10 21:28:10.364704 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:10.364666    2584 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 21:28:10.365062 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:10.364722    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq" podUID="62491f9c-f6c8-484a-9488-7b38a70cd163" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 21:28:14.558390 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.558362    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:28:14.617177 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.617108    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tps55\" (UniqueName: \"kubernetes.io/projected/62491f9c-f6c8-484a-9488-7b38a70cd163-kube-api-access-tps55\") pod \"62491f9c-f6c8-484a-9488-7b38a70cd163\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") "
Sep 10 21:28:14.617177 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.617147    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62491f9c-f6c8-484a-9488-7b38a70cd163-istio-token\") pod \"62491f9c-f6c8-484a-9488-7b38a70cd163\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") "
Sep 10 21:28:14.617374 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.617191    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-istio-envoy\") pod \"62491f9c-f6c8-484a-9488-7b38a70cd163\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") "
Sep 10 21:28:14.617374 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.617225    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62491f9c-f6c8-484a-9488-7b38a70cd163-istio-podinfo\") pod \"62491f9c-f6c8-484a-9488-7b38a70cd163\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") "
Sep 10 21:28:14.617374 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.617283    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-workload-certs\") pod \"62491f9c-f6c8-484a-9488-7b38a70cd163\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") "
Sep 10 21:28:14.617374 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.617312    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-workload-socket\") pod \"62491f9c-f6c8-484a-9488-7b38a70cd163\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") "
Sep 10 21:28:14.617374 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.617343    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-istio-data\") pod \"62491f9c-f6c8-484a-9488-7b38a70cd163\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") "
Sep 10 21:28:14.617758 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.617412    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62491f9c-f6c8-484a-9488-7b38a70cd163-istiod-ca-cert\") pod \"62491f9c-f6c8-484a-9488-7b38a70cd163\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") "
Sep 10 21:28:14.617934 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.617914    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-credential-socket\") pod \"62491f9c-f6c8-484a-9488-7b38a70cd163\" (UID: \"62491f9c-f6c8-484a-9488-7b38a70cd163\") "
Sep 10 21:28:14.618019 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.617692    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "62491f9c-f6c8-484a-9488-7b38a70cd163" (UID: "62491f9c-f6c8-484a-9488-7b38a70cd163"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:28:14.618019 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.617687    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "62491f9c-f6c8-484a-9488-7b38a70cd163" (UID: "62491f9c-f6c8-484a-9488-7b38a70cd163"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:28:14.618019 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.617711    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-istio-data" (OuterVolumeSpecName: "istio-data") pod "62491f9c-f6c8-484a-9488-7b38a70cd163" (UID: "62491f9c-f6c8-484a-9488-7b38a70cd163"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:28:14.618019 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.617973    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62491f9c-f6c8-484a-9488-7b38a70cd163-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "62491f9c-f6c8-484a-9488-7b38a70cd163" (UID: "62491f9c-f6c8-484a-9488-7b38a70cd163"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:28:14.618282 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.618259    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-workload-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:28:14.618282 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.618274    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "62491f9c-f6c8-484a-9488-7b38a70cd163" (UID: "62491f9c-f6c8-484a-9488-7b38a70cd163"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:28:14.618282 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.618286    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-workload-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:28:14.618541 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.618301    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-istio-data\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:28:14.618541 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.618319    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62491f9c-f6c8-484a-9488-7b38a70cd163-istiod-ca-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:28:14.619493 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.619465    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62491f9c-f6c8-484a-9488-7b38a70cd163-kube-api-access-tps55" (OuterVolumeSpecName: "kube-api-access-tps55") pod "62491f9c-f6c8-484a-9488-7b38a70cd163" (UID: "62491f9c-f6c8-484a-9488-7b38a70cd163"). InnerVolumeSpecName "kube-api-access-tps55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:28:14.619595 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.619527    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62491f9c-f6c8-484a-9488-7b38a70cd163-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "62491f9c-f6c8-484a-9488-7b38a70cd163" (UID: "62491f9c-f6c8-484a-9488-7b38a70cd163"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:28:14.619759 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.619730    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62491f9c-f6c8-484a-9488-7b38a70cd163-istio-token" (OuterVolumeSpecName: "istio-token") pod "62491f9c-f6c8-484a-9488-7b38a70cd163" (UID: "62491f9c-f6c8-484a-9488-7b38a70cd163"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:28:14.619835 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.619801    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "62491f9c-f6c8-484a-9488-7b38a70cd163" (UID: "62491f9c-f6c8-484a-9488-7b38a70cd163"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:28:14.686172 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.686145    2584 generic.go:356] "Generic (PLEG): container finished" podID="62491f9c-f6c8-484a-9488-7b38a70cd163" containerID="5798c08c7c62b3c3943755f9ec172086915bcead5b4274c3fc982c15b479cb0a" exitCode=0
Sep 10 21:28:14.686334 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.686194    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"
Sep 10 21:28:14.686334 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.686229    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq" event={"ID":"62491f9c-f6c8-484a-9488-7b38a70cd163","Type":"ContainerDied","Data":"5798c08c7c62b3c3943755f9ec172086915bcead5b4274c3fc982c15b479cb0a"}
Sep 10 21:28:14.686334 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.686284    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq" event={"ID":"62491f9c-f6c8-484a-9488-7b38a70cd163","Type":"ContainerDied","Data":"a39aaa679ec2d9a4629d8df9fe0344e67490462653513a12ff15d53e0d972ec3"}
Sep 10 21:28:14.686334 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.686308    2584 scope.go:117] "RemoveContainer" containerID="5798c08c7c62b3c3943755f9ec172086915bcead5b4274c3fc982c15b479cb0a"
Sep 10 21:28:14.696364 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.696345    2584 scope.go:117] "RemoveContainer" containerID="b3d151da81acfe8a0229f559d3f133d6c4c2d4a636cec4d445b4f5adf98fed96"
Sep 10 21:28:14.704112 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.704093    2584 scope.go:117] "RemoveContainer" containerID="5798c08c7c62b3c3943755f9ec172086915bcead5b4274c3fc982c15b479cb0a"
Sep 10 21:28:14.704379 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:28:14.704360    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5798c08c7c62b3c3943755f9ec172086915bcead5b4274c3fc982c15b479cb0a\": container with ID starting with 5798c08c7c62b3c3943755f9ec172086915bcead5b4274c3fc982c15b479cb0a not found: ID does not exist" containerID="5798c08c7c62b3c3943755f9ec172086915bcead5b4274c3fc982c15b479cb0a"
Sep 10 21:28:14.704470 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.704385    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5798c08c7c62b3c3943755f9ec172086915bcead5b4274c3fc982c15b479cb0a"} err="failed to get container status \"5798c08c7c62b3c3943755f9ec172086915bcead5b4274c3fc982c15b479cb0a\": rpc error: code = NotFound desc = could not find container \"5798c08c7c62b3c3943755f9ec172086915bcead5b4274c3fc982c15b479cb0a\": container with ID starting with 5798c08c7c62b3c3943755f9ec172086915bcead5b4274c3fc982c15b479cb0a not found: ID does not exist"
Sep 10 21:28:14.704470 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.704403    2584 scope.go:117] "RemoveContainer" containerID="b3d151da81acfe8a0229f559d3f133d6c4c2d4a636cec4d445b4f5adf98fed96"
Sep 10 21:28:14.704661 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:28:14.704644    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3d151da81acfe8a0229f559d3f133d6c4c2d4a636cec4d445b4f5adf98fed96\": container with ID starting with b3d151da81acfe8a0229f559d3f133d6c4c2d4a636cec4d445b4f5adf98fed96 not found: ID does not exist" containerID="b3d151da81acfe8a0229f559d3f133d6c4c2d4a636cec4d445b4f5adf98fed96"
Sep 10 21:28:14.704706 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.704668    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3d151da81acfe8a0229f559d3f133d6c4c2d4a636cec4d445b4f5adf98fed96"} err="failed to get container status \"b3d151da81acfe8a0229f559d3f133d6c4c2d4a636cec4d445b4f5adf98fed96\": rpc error: code = NotFound desc = could not find container \"b3d151da81acfe8a0229f559d3f133d6c4c2d4a636cec4d445b4f5adf98fed96\": container with ID starting with b3d151da81acfe8a0229f559d3f133d6c4c2d4a636cec4d445b4f5adf98fed96 not found: ID does not exist"
Sep 10 21:28:14.708647 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.708620    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"]
Sep 10 21:28:14.712613 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.712593    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0ac302-openshift-default-5db48476db-67xrq"]
Sep 10 21:28:14.719185 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.719166    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-credential-socket\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:28:14.719250 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.719188    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tps55\" (UniqueName: \"kubernetes.io/projected/62491f9c-f6c8-484a-9488-7b38a70cd163-kube-api-access-tps55\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:28:14.719250 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.719199    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62491f9c-f6c8-484a-9488-7b38a70cd163-istio-token\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:28:14.719250 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.719208    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62491f9c-f6c8-484a-9488-7b38a70cd163-istio-envoy\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:28:14.719250 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.719216    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62491f9c-f6c8-484a-9488-7b38a70cd163-istio-podinfo\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:28:14.839847 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:28:14.839815    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62491f9c-f6c8-484a-9488-7b38a70cd163" path="/var/lib/kubelet/pods/62491f9c-f6c8-484a-9488-7b38a70cd163/volumes"
Sep 10 21:29:00.893338 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:29:00.893306    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-jkkfm_fd81843e-6283-43e3-a571-d5e47be26e38/console-operator/1.log"
Sep 10 21:29:00.899447 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:29:00.899411    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-jkkfm_fd81843e-6283-43e3-a571-d5e47be26e38/console-operator/1.log"
Sep 10 21:29:01.175732 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:29:01.175647    2584 scope.go:117] "RemoveContainer" containerID="10feb7b749904cfa0d28520eef96a409ae4bb95bb27a314e3f01a36dd50dd7f0"
Sep 10 21:30:00.131288 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:00.131248    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817930-q5m84"]
Sep 10 21:30:00.131755 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:00.131621    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62491f9c-f6c8-484a-9488-7b38a70cd163" containerName="istio-proxy"
Sep 10 21:30:00.131755 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:00.131635    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="62491f9c-f6c8-484a-9488-7b38a70cd163" containerName="istio-proxy"
Sep 10 21:30:00.131755 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:00.131645    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62491f9c-f6c8-484a-9488-7b38a70cd163" containerName="istio-proxy"
Sep 10 21:30:00.131755 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:00.131651    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="62491f9c-f6c8-484a-9488-7b38a70cd163" containerName="istio-proxy"
Sep 10 21:30:00.131755 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:00.131663    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8018c9ab-ec1a-49c6-8664-175138d0d6e0" containerName="istio-proxy"
Sep 10 21:30:00.131755 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:00.131668    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8018c9ab-ec1a-49c6-8664-175138d0d6e0" containerName="istio-proxy"
Sep 10 21:30:00.131755 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:00.131680    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8018c9ab-ec1a-49c6-8664-175138d0d6e0" containerName="istio-proxy"
Sep 10 21:30:00.131755 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:00.131687    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8018c9ab-ec1a-49c6-8664-175138d0d6e0" containerName="istio-proxy"
Sep 10 21:30:00.131755 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:00.131703    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ee739e6-9533-45e5-abf4-a4df4992c409" containerName="revoke-keys"
Sep 10 21:30:00.131755 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:00.131708    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ee739e6-9533-45e5-abf4-a4df4992c409" containerName="revoke-keys"
Sep 10 21:30:00.132070 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:00.131763    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="62491f9c-f6c8-484a-9488-7b38a70cd163" containerName="istio-proxy"
Sep 10 21:30:00.132070 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:00.131775    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ef0c45-7695-4bfc-9979-ca253fba416e" containerName="istio-proxy"
Sep 10 21:30:00.132070 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:00.131783    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ee739e6-9533-45e5-abf4-a4df4992c409" containerName="revoke-keys"
Sep 10 21:30:00.132070 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:00.131790    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8018c9ab-ec1a-49c6-8664-175138d0d6e0" containerName="istio-proxy"
Sep 10 21:30:00.132070 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:00.131796    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="62491f9c-f6c8-484a-9488-7b38a70cd163" containerName="istio-proxy"
Sep 10 21:30:00.132070 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:00.131802    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8018c9ab-ec1a-49c6-8664-175138d0d6e0" containerName="istio-proxy"
Sep 10 21:30:00.134819 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:00.134801    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-q5m84"
Sep 10 21:30:00.137116 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:00.137095    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4gs5\""
Sep 10 21:30:00.141358 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:00.141336    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817930-q5m84"]
Sep 10 21:30:00.146399 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:00.146373    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-q5m84"
Sep 10 21:30:00.422032 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:00.422005    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817930-q5m84"]
Sep 10 21:30:00.424527 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:30:00.424498    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf80b598_5751_4d28_906a_d821d2ba30bd.slice/crio-380dac3e0214cb3c84a94501e0acdadbbb4006cbe7669ae8b9bc9b78478bf747 WatchSource:0}: Error finding container 380dac3e0214cb3c84a94501e0acdadbbb4006cbe7669ae8b9bc9b78478bf747: Status 404 returned error can't find the container with id 380dac3e0214cb3c84a94501e0acdadbbb4006cbe7669ae8b9bc9b78478bf747
Sep 10 21:30:01.200777 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:01.200742    2584 scope.go:117] "RemoveContainer" containerID="34ab67e02b67d9a7750857f6723c41985b93072819c9a625cfd1e782e9575852"
Sep 10 21:30:01.209242 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:01.209218    2584 scope.go:117] "RemoveContainer" containerID="7ffe7401e91913b21ad42e4788cff03d56f71a5cdf31942d3145b3153e051832"
Sep 10 21:30:01.217310 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:01.217290    2584 scope.go:117] "RemoveContainer" containerID="8e6f1bca6fb5c690e4fcaf32eb67e0b52affe38cb1b6c18ec9549393a0e8cbed"
Sep 10 21:30:01.221296 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:01.221268    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-q5m84" event={"ID":"cf80b598-5751-4d28-906a-d821d2ba30bd","Type":"ContainerStarted","Data":"8c8ab30a934d2dc110f7c4d8dab9ccbf29b8a09f234cb6fbdc41565d43ffac33"}
Sep 10 21:30:01.221496 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:01.221326    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-q5m84" event={"ID":"cf80b598-5751-4d28-906a-d821d2ba30bd","Type":"ContainerStarted","Data":"380dac3e0214cb3c84a94501e0acdadbbb4006cbe7669ae8b9bc9b78478bf747"}
Sep 10 21:30:02.228257 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:02.228215    2584 generic.go:356] "Generic (PLEG): container finished" podID="cf80b598-5751-4d28-906a-d821d2ba30bd" containerID="8c8ab30a934d2dc110f7c4d8dab9ccbf29b8a09f234cb6fbdc41565d43ffac33" exitCode=0
Sep 10 21:30:02.228652 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:02.228301    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-q5m84" event={"ID":"cf80b598-5751-4d28-906a-d821d2ba30bd","Type":"ContainerDied","Data":"8c8ab30a934d2dc110f7c4d8dab9ccbf29b8a09f234cb6fbdc41565d43ffac33"}
Sep 10 21:30:03.361854 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:03.361830    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-q5m84"
Sep 10 21:30:04.238868 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:04.238839    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-q5m84"
Sep 10 21:30:04.239039 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:04.238844    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-q5m84" event={"ID":"cf80b598-5751-4d28-906a-d821d2ba30bd","Type":"ContainerDied","Data":"380dac3e0214cb3c84a94501e0acdadbbb4006cbe7669ae8b9bc9b78478bf747"}
Sep 10 21:30:04.239039 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:30:04.238954    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="380dac3e0214cb3c84a94501e0acdadbbb4006cbe7669ae8b9bc9b78478bf747"
Sep 10 21:32:43.867984 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:43.867951    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"]
Sep 10 21:32:43.868487 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:43.868292    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf80b598-5751-4d28-906a-d821d2ba30bd" containerName="cleanup"
Sep 10 21:32:43.868487 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:43.868304    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf80b598-5751-4d28-906a-d821d2ba30bd" containerName="cleanup"
Sep 10 21:32:43.868487 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:43.868380    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf80b598-5751-4d28-906a-d821d2ba30bd" containerName="cleanup"
Sep 10 21:32:43.871495 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:43.871478    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:32:43.873395 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:43.873370    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-eae8a67b-kserve-self-signed-certs\""
Sep 10 21:32:43.881136 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:43.881112    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"]
Sep 10 21:32:43.945976 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:43.945948    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5229dff7-3810-443a-ae4c-98d194b015c2-tmp-dir\") pod \"e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5\" (UID: \"5229dff7-3810-443a-ae4c-98d194b015c2\") " pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:32:43.946094 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:43.945977    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5229dff7-3810-443a-ae4c-98d194b015c2-dshm\") pod \"e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5\" (UID: \"5229dff7-3810-443a-ae4c-98d194b015c2\") " pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:32:43.946094 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:43.945997    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5229dff7-3810-443a-ae4c-98d194b015c2-tls-certs\") pod \"e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5\" (UID: \"5229dff7-3810-443a-ae4c-98d194b015c2\") " pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:32:43.946094 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:43.946081    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5229dff7-3810-443a-ae4c-98d194b015c2-model-cache\") pod \"e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5\" (UID: \"5229dff7-3810-443a-ae4c-98d194b015c2\") " pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:32:43.946203 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:43.946112    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5229dff7-3810-443a-ae4c-98d194b015c2-home\") pod \"e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5\" (UID: \"5229dff7-3810-443a-ae4c-98d194b015c2\") " pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:32:43.946203 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:43.946128    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75bf8\" (UniqueName: \"kubernetes.io/projected/5229dff7-3810-443a-ae4c-98d194b015c2-kube-api-access-75bf8\") pod \"e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5\" (UID: \"5229dff7-3810-443a-ae4c-98d194b015c2\") " pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:32:44.047641 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:44.047602    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5229dff7-3810-443a-ae4c-98d194b015c2-model-cache\") pod \"e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5\" (UID: \"5229dff7-3810-443a-ae4c-98d194b015c2\") " pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:32:44.047823 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:44.047663    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5229dff7-3810-443a-ae4c-98d194b015c2-home\") pod \"e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5\" (UID: \"5229dff7-3810-443a-ae4c-98d194b015c2\") " pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:32:44.047823 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:44.047745    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75bf8\" (UniqueName: \"kubernetes.io/projected/5229dff7-3810-443a-ae4c-98d194b015c2-kube-api-access-75bf8\") pod \"e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5\" (UID: \"5229dff7-3810-443a-ae4c-98d194b015c2\") " pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:32:44.047942 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:44.047905    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5229dff7-3810-443a-ae4c-98d194b015c2-tmp-dir\") pod \"e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5\" (UID: \"5229dff7-3810-443a-ae4c-98d194b015c2\") " pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:32:44.047994 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:44.047955    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5229dff7-3810-443a-ae4c-98d194b015c2-dshm\") pod \"e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5\" (UID: \"5229dff7-3810-443a-ae4c-98d194b015c2\") " pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:32:44.048067 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:44.048047    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5229dff7-3810-443a-ae4c-98d194b015c2-home\") pod \"e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5\" (UID: \"5229dff7-3810-443a-ae4c-98d194b015c2\") " pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:32:44.048123 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:44.048087    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5229dff7-3810-443a-ae4c-98d194b015c2-model-cache\") pod \"e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5\" (UID: \"5229dff7-3810-443a-ae4c-98d194b015c2\") " pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:32:44.048231 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:44.048212    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5229dff7-3810-443a-ae4c-98d194b015c2-tls-certs\") pod \"e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5\" (UID: \"5229dff7-3810-443a-ae4c-98d194b015c2\") " pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:32:44.048304 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:44.048235    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5229dff7-3810-443a-ae4c-98d194b015c2-tmp-dir\") pod \"e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5\" (UID: \"5229dff7-3810-443a-ae4c-98d194b015c2\") " pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:32:44.050270 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:44.050242    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5229dff7-3810-443a-ae4c-98d194b015c2-dshm\") pod \"e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5\" (UID: \"5229dff7-3810-443a-ae4c-98d194b015c2\") " pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:32:44.050713 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:44.050666    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5229dff7-3810-443a-ae4c-98d194b015c2-tls-certs\") pod \"e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5\" (UID: \"5229dff7-3810-443a-ae4c-98d194b015c2\") " pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:32:44.055927 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:44.055906    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75bf8\" (UniqueName: \"kubernetes.io/projected/5229dff7-3810-443a-ae4c-98d194b015c2-kube-api-access-75bf8\") pod \"e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5\" (UID: \"5229dff7-3810-443a-ae4c-98d194b015c2\") " pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:32:44.180901 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:44.180869    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:32:44.456774 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:44.456750    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"]
Sep 10 21:32:44.458774 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:32:44.458745    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5229dff7_3810_443a_ae4c_98d194b015c2.slice/crio-d0687d231a408f18331beeef3feec2b5bb1ef119653e9841541d267e262ab642 WatchSource:0}: Error finding container d0687d231a408f18331beeef3feec2b5bb1ef119653e9841541d267e262ab642: Status 404 returned error can't find the container with id d0687d231a408f18331beeef3feec2b5bb1ef119653e9841541d267e262ab642
Sep 10 21:32:45.012154 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:45.012118    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5" event={"ID":"5229dff7-3810-443a-ae4c-98d194b015c2","Type":"ContainerStarted","Data":"d57814a8cdf7fb371d0c87b06c498473aff1270ef42a12aa3859c7f59e7c7138"}
Sep 10 21:32:45.012154 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:45.012160    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5" event={"ID":"5229dff7-3810-443a-ae4c-98d194b015c2","Type":"ContainerStarted","Data":"d0687d231a408f18331beeef3feec2b5bb1ef119653e9841541d267e262ab642"}
Sep 10 21:32:45.030772 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:45.030722    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5" podStartSLOduration=2.030707206 podStartE2EDuration="2.030707206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:32:45.029097635 +0000 UTC m=+2024.694122435" watchObservedRunningTime="2026-09-10 21:32:45.030707206 +0000 UTC m=+2024.695732042"
Sep 10 21:32:54.181080 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:54.181039    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:32:54.181602 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:54.181106    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:32:54.196681 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:54.196647    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:32:55.071022 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:32:55.070989    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:33:01.258631 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:01.258596    2584 scope.go:117] "RemoveContainer" containerID="afef252cce01d00fe6fc9dca2812fe2e6d2706679a152f123ac0c9976aff3d99"
Sep 10 21:33:01.267009 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:01.266990    2584 scope.go:117] "RemoveContainer" containerID="637e81f526a373795e1fe317c27b8b341cec40f18788396af0925983395c90fa"
Sep 10 21:33:41.295222 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:41.295130    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"]
Sep 10 21:33:56.314590 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:56.314529    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5" podUID="5229dff7-3810-443a-ae4c-98d194b015c2" containerName="main" containerID="cri-o://d57814a8cdf7fb371d0c87b06c498473aff1270ef42a12aa3859c7f59e7c7138" gracePeriod=45
Sep 10 21:33:56.560873 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:56.560848    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:33:56.637952 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:56.637924    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5229dff7-3810-443a-ae4c-98d194b015c2-dshm\") pod \"5229dff7-3810-443a-ae4c-98d194b015c2\" (UID: \"5229dff7-3810-443a-ae4c-98d194b015c2\") "
Sep 10 21:33:56.638120 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:56.637962    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5229dff7-3810-443a-ae4c-98d194b015c2-model-cache\") pod \"5229dff7-3810-443a-ae4c-98d194b015c2\" (UID: \"5229dff7-3810-443a-ae4c-98d194b015c2\") "
Sep 10 21:33:56.638120 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:56.638018    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5229dff7-3810-443a-ae4c-98d194b015c2-home\") pod \"5229dff7-3810-443a-ae4c-98d194b015c2\" (UID: \"5229dff7-3810-443a-ae4c-98d194b015c2\") "
Sep 10 21:33:56.638120 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:56.638094    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5229dff7-3810-443a-ae4c-98d194b015c2-tls-certs\") pod \"5229dff7-3810-443a-ae4c-98d194b015c2\" (UID: \"5229dff7-3810-443a-ae4c-98d194b015c2\") "
Sep 10 21:33:56.638302 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:56.638120    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75bf8\" (UniqueName: \"kubernetes.io/projected/5229dff7-3810-443a-ae4c-98d194b015c2-kube-api-access-75bf8\") pod \"5229dff7-3810-443a-ae4c-98d194b015c2\" (UID: \"5229dff7-3810-443a-ae4c-98d194b015c2\") "
Sep 10 21:33:56.638302 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:56.638175    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5229dff7-3810-443a-ae4c-98d194b015c2-tmp-dir\") pod \"5229dff7-3810-443a-ae4c-98d194b015c2\" (UID: \"5229dff7-3810-443a-ae4c-98d194b015c2\") "
Sep 10 21:33:56.638403 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:56.638325    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5229dff7-3810-443a-ae4c-98d194b015c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "5229dff7-3810-443a-ae4c-98d194b015c2" (UID: "5229dff7-3810-443a-ae4c-98d194b015c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:33:56.638403 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:56.638338    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5229dff7-3810-443a-ae4c-98d194b015c2-home" (OuterVolumeSpecName: "home") pod "5229dff7-3810-443a-ae4c-98d194b015c2" (UID: "5229dff7-3810-443a-ae4c-98d194b015c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:33:56.638574 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:56.638556    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5229dff7-3810-443a-ae4c-98d194b015c2-home\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:33:56.638643 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:56.638581    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5229dff7-3810-443a-ae4c-98d194b015c2-model-cache\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:33:56.638692 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:56.638658    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5229dff7-3810-443a-ae4c-98d194b015c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5229dff7-3810-443a-ae4c-98d194b015c2" (UID: "5229dff7-3810-443a-ae4c-98d194b015c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:33:56.640143 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:56.640114    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5229dff7-3810-443a-ae4c-98d194b015c2-dshm" (OuterVolumeSpecName: "dshm") pod "5229dff7-3810-443a-ae4c-98d194b015c2" (UID: "5229dff7-3810-443a-ae4c-98d194b015c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:33:56.640679 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:56.640654    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5229dff7-3810-443a-ae4c-98d194b015c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5229dff7-3810-443a-ae4c-98d194b015c2" (UID: "5229dff7-3810-443a-ae4c-98d194b015c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:33:56.640736 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:56.640678    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5229dff7-3810-443a-ae4c-98d194b015c2-kube-api-access-75bf8" (OuterVolumeSpecName: "kube-api-access-75bf8") pod "5229dff7-3810-443a-ae4c-98d194b015c2" (UID: "5229dff7-3810-443a-ae4c-98d194b015c2"). InnerVolumeSpecName "kube-api-access-75bf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:33:56.739297 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:56.739255    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5229dff7-3810-443a-ae4c-98d194b015c2-tls-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:33:56.739297 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:56.739299    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75bf8\" (UniqueName: \"kubernetes.io/projected/5229dff7-3810-443a-ae4c-98d194b015c2-kube-api-access-75bf8\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:33:56.739546 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:56.739316    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5229dff7-3810-443a-ae4c-98d194b015c2-tmp-dir\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:33:56.739546 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:56.739334    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5229dff7-3810-443a-ae4c-98d194b015c2-dshm\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 10 21:33:57.381456 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:57.381405    2584 generic.go:356] "Generic (PLEG): container finished" podID="5229dff7-3810-443a-ae4c-98d194b015c2" containerID="d57814a8cdf7fb371d0c87b06c498473aff1270ef42a12aa3859c7f59e7c7138" exitCode=0
Sep 10 21:33:57.381934 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:57.381470    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"
Sep 10 21:33:57.381934 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:57.381497    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5" event={"ID":"5229dff7-3810-443a-ae4c-98d194b015c2","Type":"ContainerDied","Data":"d57814a8cdf7fb371d0c87b06c498473aff1270ef42a12aa3859c7f59e7c7138"}
Sep 10 21:33:57.381934 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:57.381546    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5" event={"ID":"5229dff7-3810-443a-ae4c-98d194b015c2","Type":"ContainerDied","Data":"d0687d231a408f18331beeef3feec2b5bb1ef119653e9841541d267e262ab642"}
Sep 10 21:33:57.381934 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:57.381567    2584 scope.go:117] "RemoveContainer" containerID="d57814a8cdf7fb371d0c87b06c498473aff1270ef42a12aa3859c7f59e7c7138"
Sep 10 21:33:57.390117 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:57.390098    2584 scope.go:117] "RemoveContainer" containerID="d57814a8cdf7fb371d0c87b06c498473aff1270ef42a12aa3859c7f59e7c7138"
Sep 10 21:33:57.390368 ip-10-0-135-205 kubenswrapper[2584]: E0910 21:33:57.390348    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d57814a8cdf7fb371d0c87b06c498473aff1270ef42a12aa3859c7f59e7c7138\": container with ID starting with d57814a8cdf7fb371d0c87b06c498473aff1270ef42a12aa3859c7f59e7c7138 not found: ID does not exist" containerID="d57814a8cdf7fb371d0c87b06c498473aff1270ef42a12aa3859c7f59e7c7138"
Sep 10 21:33:57.390446 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:57.390377    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d57814a8cdf7fb371d0c87b06c498473aff1270ef42a12aa3859c7f59e7c7138"} err="failed to get container status \"d57814a8cdf7fb371d0c87b06c498473aff1270ef42a12aa3859c7f59e7c7138\": rpc error: code = NotFound desc = could not find container \"d57814a8cdf7fb371d0c87b06c498473aff1270ef42a12aa3859c7f59e7c7138\": container with ID starting with d57814a8cdf7fb371d0c87b06c498473aff1270ef42a12aa3859c7f59e7c7138 not found: ID does not exist"
Sep 10 21:33:57.397190 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:57.397168    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"]
Sep 10 21:33:57.401814 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:57.401794    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-eae8a67b-kserve-797c8bf7f-2ghw5"]
Sep 10 21:33:58.839969 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:33:58.839936    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5229dff7-3810-443a-ae4c-98d194b015c2" path="/var/lib/kubelet/pods/5229dff7-3810-443a-ae4c-98d194b015c2/volumes"
Sep 10 21:34:00.922998 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:34:00.922969    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-jkkfm_fd81843e-6283-43e3-a571-d5e47be26e38/console-operator/1.log"
Sep 10 21:34:00.930492 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:34:00.930468    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-jkkfm_fd81843e-6283-43e3-a571-d5e47be26e38/console-operator/1.log"
Sep 10 21:34:01.287944 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:34:01.287856    2584 scope.go:117] "RemoveContainer" containerID="3d827a71ad9fb3d84da4617f990b8bf7a57717a8632ce840c081e518825b3c59"
Sep 10 21:34:01.296226 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:34:01.296202    2584 scope.go:117] "RemoveContainer" containerID="c1d618ae6bc81fc3b1645740e2c054c9bad07c26bd7a503c8a08170211e4ad91"
Sep 10 21:34:42.442940 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:34:42.442908    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-x66cz_ffde7537-c081-431e-a26a-f10f66c20b60/manager/0.log"
Sep 10 21:34:42.699909 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:34:42.699831    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d67fdbbcb-m5jrb_cace0196-0854-40eb-b897-917323c7b8ec/copy-manifests/0.log"
Sep 10 21:34:42.712851 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:34:42.712825    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d67fdbbcb-m5jrb_cace0196-0854-40eb-b897-917323c7b8ec/manager/0.log"
Sep 10 21:34:43.537043 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:34:43.537018    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817930-q5m84_cf80b598-5751-4d28-906a-d821d2ba30bd/cleanup/0.log"
Sep 10 21:34:44.828469 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:34:44.828374    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-c7qg4_d9094ca4-097a-4074-8753-05239f5fc252/kuadrant-console-plugin/0.log"
Sep 10 21:34:45.581514 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:34:45.581484    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c994s2_19306155-ec71-4c18-9e12-a6afe565b642/istio-proxy/0.log"
Sep 10 21:34:45.825042 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:34:45.825012    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54c6c477f9-4bk2f_491006ce-4acf-49b3-9a19-210ed7756177/kube-auth-proxy/0.log"
Sep 10 21:34:46.070947 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:34:46.070913    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-wqb75_c45383bb-f8d6-4467-8aa4-af2e1a9e18fc/istio-proxy/1.log"
Sep 10 21:34:46.804253 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:34:46.804224    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-f7t75_54f7bcbf-c532-44ad-9044-95f58c58a93d/main/0.log"
Sep 10 21:34:47.165180 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:34:47.165154    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-b58zm_2b21604b-6ee2-43f9-bf26-757932579b52/main/0.log"
Sep 10 21:34:47.283826 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:34:47.283795    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkshwf_6aa2cff9-04d4-4eaf-8df6-e7310cf89623/main/0.log"
Sep 10 21:34:47.526998 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:34:47.526914    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-x9mlz_6228bd72-6ae7-4078-852d-24dffc5e437f/main/0.log"
Sep 10 21:34:56.241449 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:34:56.241349    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-k8v5s_eb76c9c7-abc6-4d93-b8e8-f68110f905a9/global-pull-secret-syncer/0.log"
Sep 10 21:34:56.352323 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:34:56.352287    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-r282d_cdafa9fe-0815-45f4-b40b-d41b196f2735/konnectivity-agent/0.log"
Sep 10 21:34:56.448627 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:34:56.448598    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-205.ec2.internal_77a4c4cd902dc458728fadabb7f791eb/haproxy/0.log"
Sep 10 21:35:00.468744 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:00.468710    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-c7qg4_d9094ca4-097a-4074-8753-05239f5fc252/kuadrant-console-plugin/0.log"
Sep 10 21:35:02.579933 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:02.579897    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-zkbqs_93462573-692a-4246-afd9-08c9430c7f53/cluster-monitoring-operator/0.log"
Sep 10 21:35:02.694394 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:02.694364    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-d6cc4887f-2xhtg_42269fc7-bacc-4820-9835-a8a814cb2134/metrics-server/0.log"
Sep 10 21:35:02.760081 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:02.760054    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2qskx_71550cf6-f533-4e66-8aa2-e9e208d5f700/node-exporter/0.log"
Sep 10 21:35:02.781682 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:02.781659    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2qskx_71550cf6-f533-4e66-8aa2-e9e208d5f700/kube-rbac-proxy/0.log"
Sep 10 21:35:02.800829 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:02.800808    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2qskx_71550cf6-f533-4e66-8aa2-e9e208d5f700/init-textfile/0.log"
Sep 10 21:35:03.075399 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:03.075374    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a48b2cb1-5c53-4002-9cc9-f54d995541dd/prometheus/0.log"
Sep 10 21:35:03.096370 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:03.096344    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a48b2cb1-5c53-4002-9cc9-f54d995541dd/config-reloader/0.log"
Sep 10 21:35:03.117673 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:03.117652    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a48b2cb1-5c53-4002-9cc9-f54d995541dd/thanos-sidecar/0.log"
Sep 10 21:35:03.142255 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:03.142234    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a48b2cb1-5c53-4002-9cc9-f54d995541dd/kube-rbac-proxy-web/0.log"
Sep 10 21:35:03.164615 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:03.164595    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a48b2cb1-5c53-4002-9cc9-f54d995541dd/kube-rbac-proxy/0.log"
Sep 10 21:35:03.183858 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:03.183838    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a48b2cb1-5c53-4002-9cc9-f54d995541dd/kube-rbac-proxy-thanos/0.log"
Sep 10 21:35:03.203951 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:03.203931    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a48b2cb1-5c53-4002-9cc9-f54d995541dd/init-config-reloader/0.log"
Sep 10 21:35:05.264404 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:05.264360    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-pjkzj_29f5e959-7b2f-42e9-98f2-a3547859bb56/networking-console-plugin/0.log"
Sep 10 21:35:05.874152 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:05.874122    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-jkkfm_fd81843e-6283-43e3-a571-d5e47be26e38/console-operator/1.log"
Sep 10 21:35:05.879637 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:05.879612    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-jkkfm_fd81843e-6283-43e3-a571-d5e47be26e38/console-operator/2.log"
Sep 10 21:35:06.868203 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:06.868167    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp"]
Sep 10 21:35:06.868787 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:06.868766    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5229dff7-3810-443a-ae4c-98d194b015c2" containerName="main"
Sep 10 21:35:06.868853 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:06.868791    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5229dff7-3810-443a-ae4c-98d194b015c2" containerName="main"
Sep 10 21:35:06.868901 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:06.868877    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5229dff7-3810-443a-ae4c-98d194b015c2" containerName="main"
Sep 10 21:35:06.872162 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:06.872138    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp"
Sep 10 21:35:06.874362 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:06.874339    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n84km\"/\"default-dockercfg-xm668\""
Sep 10 21:35:06.874859 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:06.874839    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n84km\"/\"kube-root-ca.crt\""
Sep 10 21:35:06.874934 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:06.874893    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n84km\"/\"openshift-service-ca.crt\""
Sep 10 21:35:06.880116 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:06.880091    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp"]
Sep 10 21:35:06.928979 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:06.928938    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c414429f-d115-4020-9d36-aea2981b4a11-proc\") pod \"perf-node-gather-daemonset-gdxhp\" (UID: \"c414429f-d115-4020-9d36-aea2981b4a11\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp"
Sep 10 21:35:06.928979 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:06.928983    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c414429f-d115-4020-9d36-aea2981b4a11-podres\") pod \"perf-node-gather-daemonset-gdxhp\" (UID: \"c414429f-d115-4020-9d36-aea2981b4a11\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp"
Sep 10 21:35:06.929177 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:06.929055    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s8xw\" (UniqueName: \"kubernetes.io/projected/c414429f-d115-4020-9d36-aea2981b4a11-kube-api-access-9s8xw\") pod \"perf-node-gather-daemonset-gdxhp\" (UID: \"c414429f-d115-4020-9d36-aea2981b4a11\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp"
Sep 10 21:35:06.929177 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:06.929097    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c414429f-d115-4020-9d36-aea2981b4a11-sys\") pod \"perf-node-gather-daemonset-gdxhp\" (UID: \"c414429f-d115-4020-9d36-aea2981b4a11\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp"
Sep 10 21:35:06.929249 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:06.929215    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c414429f-d115-4020-9d36-aea2981b4a11-lib-modules\") pod \"perf-node-gather-daemonset-gdxhp\" (UID: \"c414429f-d115-4020-9d36-aea2981b4a11\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp"
Sep 10 21:35:07.029938 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:07.029890    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c414429f-d115-4020-9d36-aea2981b4a11-proc\") pod \"perf-node-gather-daemonset-gdxhp\" (UID: \"c414429f-d115-4020-9d36-aea2981b4a11\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp"
Sep 10 21:35:07.029938 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:07.029936    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c414429f-d115-4020-9d36-aea2981b4a11-podres\") pod \"perf-node-gather-daemonset-gdxhp\" (UID: \"c414429f-d115-4020-9d36-aea2981b4a11\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp"
Sep 10 21:35:07.030156 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:07.029973    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s8xw\" (UniqueName: \"kubernetes.io/projected/c414429f-d115-4020-9d36-aea2981b4a11-kube-api-access-9s8xw\") pod \"perf-node-gather-daemonset-gdxhp\" (UID: \"c414429f-d115-4020-9d36-aea2981b4a11\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp"
Sep 10 21:35:07.030156 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:07.030001    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c414429f-d115-4020-9d36-aea2981b4a11-sys\") pod \"perf-node-gather-daemonset-gdxhp\" (UID: \"c414429f-d115-4020-9d36-aea2981b4a11\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp"
Sep 10 21:35:07.030156 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:07.030025    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c414429f-d115-4020-9d36-aea2981b4a11-proc\") pod \"perf-node-gather-daemonset-gdxhp\" (UID: \"c414429f-d115-4020-9d36-aea2981b4a11\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp"
Sep 10 21:35:07.030156 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:07.030111    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c414429f-d115-4020-9d36-aea2981b4a11-sys\") pod \"perf-node-gather-daemonset-gdxhp\" (UID: \"c414429f-d115-4020-9d36-aea2981b4a11\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp"
Sep 10 21:35:07.030156 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:07.030120    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c414429f-d115-4020-9d36-aea2981b4a11-lib-modules\") pod \"perf-node-gather-daemonset-gdxhp\" (UID: \"c414429f-d115-4020-9d36-aea2981b4a11\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp"
Sep 10 21:35:07.030156 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:07.030144    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c414429f-d115-4020-9d36-aea2981b4a11-podres\") pod \"perf-node-gather-daemonset-gdxhp\" (UID: \"c414429f-d115-4020-9d36-aea2981b4a11\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp"
Sep 10 21:35:07.030364 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:07.030255    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c414429f-d115-4020-9d36-aea2981b4a11-lib-modules\") pod \"perf-node-gather-daemonset-gdxhp\" (UID: \"c414429f-d115-4020-9d36-aea2981b4a11\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp"
Sep 10 21:35:07.038037 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:07.038003    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s8xw\" (UniqueName: \"kubernetes.io/projected/c414429f-d115-4020-9d36-aea2981b4a11-kube-api-access-9s8xw\") pod \"perf-node-gather-daemonset-gdxhp\" (UID: \"c414429f-d115-4020-9d36-aea2981b4a11\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp"
Sep 10 21:35:07.183627 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:07.183591    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp"
Sep 10 21:35:07.456954 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:07.456902    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp"]
Sep 10 21:35:07.458245 ip-10-0-135-205 kubenswrapper[2584]: W0910 21:35:07.458217    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc414429f_d115_4020_9d36_aea2981b4a11.slice/crio-55d4c0e90c34716c48e315d4bf811f755bac9e8f6178592e11400564615d653a WatchSource:0}: Error finding container 55d4c0e90c34716c48e315d4bf811f755bac9e8f6178592e11400564615d653a: Status 404 returned error can't find the container with id 55d4c0e90c34716c48e315d4bf811f755bac9e8f6178592e11400564615d653a
Sep 10 21:35:07.700984 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:07.700946    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp" event={"ID":"c414429f-d115-4020-9d36-aea2981b4a11","Type":"ContainerStarted","Data":"bd56353664117c74243dd06d84cb3f11f7a35c08d5f353e6476fd3bf8ef2b145"}
Sep 10 21:35:07.701155 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:07.701001    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp" event={"ID":"c414429f-d115-4020-9d36-aea2981b4a11","Type":"ContainerStarted","Data":"55d4c0e90c34716c48e315d4bf811f755bac9e8f6178592e11400564615d653a"}
Sep 10 21:35:07.701155 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:07.701086    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp"
Sep 10 21:35:07.720301 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:07.720204    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp" podStartSLOduration=1.720189339 podStartE2EDuration="1.720189339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 21:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:35:07.718476484 +0000 UTC m=+2167.383501283" watchObservedRunningTime="2026-09-10 21:35:07.720189339 +0000 UTC m=+2167.385214139"
Sep 10 21:35:08.369714 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:08.369684    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dzgxv_3dfe9291-e76b-4d25-91bf-4c3d78d62841/dns/0.log"
Sep 10 21:35:08.389565 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:08.389541    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dzgxv_3dfe9291-e76b-4d25-91bf-4c3d78d62841/kube-rbac-proxy/0.log"
Sep 10 21:35:08.519844 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:08.519819    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wprl7_6aa58f1a-77a8-4479-a7a8-a68b6a4673cc/dns-node-resolver/0.log"
Sep 10 21:35:09.065862 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:09.065831    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5ddb5757f4-cbdk5_c9d0687c-a6af-4745-9cbb-b745d52ef591/registry/0.log"
Sep 10 21:35:09.134687 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:09.134658    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xswnp_8032963a-4ac1-4d7b-a4de-e9bac1e2c9a1/node-ca/0.log"
Sep 10 21:35:10.077268 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:10.077241    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c994s2_19306155-ec71-4c18-9e12-a6afe565b642/istio-proxy/0.log"
Sep 10 21:35:10.332490 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:10.332383    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54c6c477f9-4bk2f_491006ce-4acf-49b3-9a19-210ed7756177/kube-auth-proxy/0.log"
Sep 10 21:35:10.382474 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:10.382448    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-wqb75_c45383bb-f8d6-4467-8aa4-af2e1a9e18fc/istio-proxy/0.log"
Sep 10 21:35:10.390458 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:10.390417    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-wqb75_c45383bb-f8d6-4467-8aa4-af2e1a9e18fc/istio-proxy/1.log"
Sep 10 21:35:11.385317 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:11.385290    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ql42f_e3464d4e-8d40-4389-9be1-49529d8636ae/serve-healthcheck-canary/0.log"
Sep 10 21:35:11.848599 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:11.848520    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-hpt25_367d0cac-af3c-4dce-acc3-ce2e5f85fce1/insights-operator/0.log"
Sep 10 21:35:11.849916 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:11.849897    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-hpt25_367d0cac-af3c-4dce-acc3-ce2e5f85fce1/insights-operator/1.log"
Sep 10 21:35:11.950127 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:11.950100    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f5r9p_6b961536-58d3-41eb-aa81-d1edb5dd1d0e/kube-rbac-proxy/0.log"
Sep 10 21:35:11.969171 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:11.969146    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f5r9p_6b961536-58d3-41eb-aa81-d1edb5dd1d0e/exporter/0.log"
Sep 10 21:35:11.988115 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:11.988091    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f5r9p_6b961536-58d3-41eb-aa81-d1edb5dd1d0e/extractor/0.log"
Sep 10 21:35:13.715786 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:13.715761    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-n84km/perf-node-gather-daemonset-gdxhp"
Sep 10 21:35:14.443406 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:14.443372    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-x66cz_ffde7537-c081-431e-a26a-f10f66c20b60/manager/0.log"
Sep 10 21:35:14.557320 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:14.557286    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d67fdbbcb-m5jrb_cace0196-0854-40eb-b897-917323c7b8ec/manager/0.log"
Sep 10 21:35:14.573403 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:14.573377    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d67fdbbcb-m5jrb_cace0196-0854-40eb-b897-917323c7b8ec/copy-manifests/0.log"
Sep 10 21:35:15.966800 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:15.966724    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-548fd67646-ldmh4_bc305b9b-0ab4-4760-a348-e17f66bd10b4/manager/0.log"
Sep 10 21:35:16.550243 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:16.550215    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c994s2_19306155-ec71-4c18-9e12-a6afe565b642/istio-proxy/0.log"
Sep 10 21:35:16.762733 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:16.762643    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54c6c477f9-4bk2f_491006ce-4acf-49b3-9a19-210ed7756177/kube-auth-proxy/0.log"
Sep 10 21:35:16.794687 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:16.794655    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-wqb75_c45383bb-f8d6-4467-8aa4-af2e1a9e18fc/istio-proxy/0.log"
Sep 10 21:35:16.805812 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:16.805769    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-wqb75_c45383bb-f8d6-4467-8aa4-af2e1a9e18fc/istio-proxy/1.log"
Sep 10 21:35:18.348777 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:18.348747    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c994s2_19306155-ec71-4c18-9e12-a6afe565b642/istio-proxy/0.log"
Sep 10 21:35:18.561976 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:18.561895    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54c6c477f9-4bk2f_491006ce-4acf-49b3-9a19-210ed7756177/kube-auth-proxy/0.log"
Sep 10 21:35:18.599484 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:18.599451    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-wqb75_c45383bb-f8d6-4467-8aa4-af2e1a9e18fc/istio-proxy/0.log"
Sep 10 21:35:18.608914 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:18.608887    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-wqb75_c45383bb-f8d6-4467-8aa4-af2e1a9e18fc/istio-proxy/1.log"
Sep 10 21:35:21.190047 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:21.190020    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-ptzj9_68115039-7bd0-49f4-a4d7-eca804f130f1/migrator/0.log"
Sep 10 21:35:21.210737 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:21.210713    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-ptzj9_68115039-7bd0-49f4-a4d7-eca804f130f1/graceful-termination/0.log"
Sep 10 21:35:22.691891 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:22.691864    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6cwt_841ad25d-3c20-4383-a1c2-31b401b6a6d1/kube-multus-additional-cni-plugins/0.log"
Sep 10 21:35:22.714218 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:22.714193    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6cwt_841ad25d-3c20-4383-a1c2-31b401b6a6d1/egress-router-binary-copy/0.log"
Sep 10 21:35:22.733469 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:22.733445    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6cwt_841ad25d-3c20-4383-a1c2-31b401b6a6d1/cni-plugins/0.log"
Sep 10 21:35:22.752644 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:22.752618    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6cwt_841ad25d-3c20-4383-a1c2-31b401b6a6d1/bond-cni-plugin/0.log"
Sep 10 21:35:22.771355 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:22.771322    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6cwt_841ad25d-3c20-4383-a1c2-31b401b6a6d1/routeoverride-cni/0.log"
Sep 10 21:35:22.790408 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:22.790387    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6cwt_841ad25d-3c20-4383-a1c2-31b401b6a6d1/whereabouts-cni-bincopy/0.log"
Sep 10 21:35:22.810245 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:22.810221    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6cwt_841ad25d-3c20-4383-a1c2-31b401b6a6d1/whereabouts-cni/0.log"
Sep 10 21:35:23.025203 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:23.025121    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-nrl66_445e9438-de21-4264-86b9-3fad26c5ebb3/kube-multus/0.log"
Sep 10 21:35:23.172704 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:23.172671    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jskzm_b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e/network-metrics-daemon/0.log"
Sep 10 21:35:23.188433 ip-10-0-135-205 kubenswrapper[2584]: I0910 21:35:23.188407    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jskzm_b981d7c0-ffc5-4b52-a4f5-dc2ee420c32e/kube-rbac-proxy/0.log"