Sep 15 20:55:58.535060 ip-10-0-142-207 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 15 20:55:58.535069 ip-10-0-142-207 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 15 20:55:58.535076 ip-10-0-142-207 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 15 20:55:58.535228 ip-10-0-142-207 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 15 20:56:08.569284 ip-10-0-142-207 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 15 20:56:08.569301 ip-10-0-142-207 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 696e2c05d8e1451ba4e3a2f3ea0b8582 --
Sep 15 20:58:01.402408 ip-10-0-142-207 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 20:58:01.753473 ip-10-0-142-207 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 20:58:01.753473 ip-10-0-142-207 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 20:58:01.753473 ip-10-0-142-207 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 20:58:01.753473 ip-10-0-142-207 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 20:58:01.753473 ip-10-0-142-207 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 20:58:01.755561 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.755467    2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 20:58:01.759253 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759233    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 20:58:01.759253 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759254    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 20:58:01.759321 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759257    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 20:58:01.759321 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759261    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 20:58:01.759321 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759263    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 20:58:01.759321 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759267    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 20:58:01.759321 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759271    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 20:58:01.759321 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759274    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 20:58:01.759321 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759277    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 20:58:01.759321 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759279    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 20:58:01.759321 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759282    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 20:58:01.759321 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759285    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 20:58:01.759321 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759288    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 20:58:01.759321 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759290    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 20:58:01.759321 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759293    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 20:58:01.759321 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759296    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 20:58:01.759321 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759306    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 20:58:01.759321 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759308    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 20:58:01.759321 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759312    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 20:58:01.759321 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759314    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 20:58:01.759321 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759317    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 20:58:01.759321 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759320    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 20:58:01.759817 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759322    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 20:58:01.759817 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759326    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 20:58:01.759817 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759329    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 20:58:01.759817 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759332    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 20:58:01.759817 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759335    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 20:58:01.759817 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759337    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 20:58:01.759817 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759340    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 20:58:01.759817 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759344    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 20:58:01.759817 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759349    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 20:58:01.759817 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759351    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 20:58:01.759817 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759354    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 20:58:01.759817 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759357    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 20:58:01.759817 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759360    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 20:58:01.759817 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759363    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 20:58:01.759817 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759366    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 20:58:01.759817 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759368    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 20:58:01.759817 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759371    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 20:58:01.759817 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759374    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 20:58:01.759817 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759376    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 20:58:01.760297 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759379    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 20:58:01.760297 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759381    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 20:58:01.760297 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759384    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 20:58:01.760297 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759386    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 20:58:01.760297 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759389    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 20:58:01.760297 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759392    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 20:58:01.760297 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759396    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 20:58:01.760297 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759399    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 20:58:01.760297 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759402    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 20:58:01.760297 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759404    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 20:58:01.760297 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759407    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 20:58:01.760297 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759409    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 20:58:01.760297 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759412    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 20:58:01.760297 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759414    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 20:58:01.760297 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759417    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 20:58:01.760297 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759428    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 20:58:01.760297 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759431    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 20:58:01.760297 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759434    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 20:58:01.760297 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759436    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 20:58:01.760297 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759439    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 20:58:01.760826 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759441    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 20:58:01.760826 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759444    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 20:58:01.760826 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759447    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 20:58:01.760826 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759449    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 20:58:01.760826 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759452    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 20:58:01.760826 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759455    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 20:58:01.760826 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759457    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 20:58:01.760826 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759460    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 20:58:01.760826 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759462    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 20:58:01.760826 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759465    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 20:58:01.760826 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759467    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 20:58:01.760826 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759470    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 20:58:01.760826 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759472    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 20:58:01.760826 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759475    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 20:58:01.760826 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759477    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 20:58:01.760826 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759480    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 20:58:01.760826 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759483    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 20:58:01.760826 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759485    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 20:58:01.760826 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759488    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 20:58:01.761331 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759493    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 20:58:01.761331 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759497    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 20:58:01.761331 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759500    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 20:58:01.761331 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759503    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 20:58:01.761331 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759505    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 20:58:01.761331 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759508    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 20:58:01.761331 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759511    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 20:58:01.761331 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759516    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 20:58:01.761331 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759519    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 20:58:01.761331 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759521    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 20:58:01.761331 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759524    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 20:58:01.761331 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759527    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 20:58:01.761331 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759529    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 20:58:01.761331 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759532    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 20:58:01.761331 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759534    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 20:58:01.761331 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759538    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 20:58:01.761331 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759541    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 20:58:01.761331 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759544    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 20:58:01.761331 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.759547    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 20:58:01.761799 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760261    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 20:58:01.761799 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760268    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 20:58:01.761799 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760271    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 20:58:01.761799 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760274    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 20:58:01.761799 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760277    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 20:58:01.761799 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760280    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 20:58:01.761799 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760283    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 20:58:01.761799 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760286    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 20:58:01.761799 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760288    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 20:58:01.761799 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760291    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 20:58:01.761799 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760294    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 20:58:01.761799 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760297    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 20:58:01.761799 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760299    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 20:58:01.761799 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760302    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 20:58:01.761799 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760305    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 20:58:01.761799 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760308    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 20:58:01.761799 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760311    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 20:58:01.761799 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760314    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 20:58:01.761799 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760317    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 20:58:01.761799 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760319    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 20:58:01.761799 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760323    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 20:58:01.762340 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760326    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 20:58:01.762340 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760328    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 20:58:01.762340 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760331    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 20:58:01.762340 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760334    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 20:58:01.762340 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760338    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 20:58:01.762340 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760341    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 20:58:01.762340 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760344    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 20:58:01.762340 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760347    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 20:58:01.762340 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760349    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 20:58:01.762340 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760352    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 20:58:01.762340 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760354    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 20:58:01.762340 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760357    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 20:58:01.762340 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760359    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 20:58:01.762340 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760362    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 20:58:01.762340 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760365    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 20:58:01.762340 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760368    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 20:58:01.762340 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760370    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 20:58:01.762340 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760373    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 20:58:01.762340 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760376    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 20:58:01.762811 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760378    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 20:58:01.762811 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760381    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 20:58:01.762811 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760383    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 20:58:01.762811 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760386    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 20:58:01.762811 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760388    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 20:58:01.762811 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760391    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 20:58:01.762811 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760393    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 20:58:01.762811 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760396    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 20:58:01.762811 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760399    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 20:58:01.762811 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760402    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 20:58:01.762811 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760404    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 20:58:01.762811 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760407    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 20:58:01.762811 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760410    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 20:58:01.762811 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760413    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 20:58:01.762811 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760416    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 20:58:01.762811 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760418    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 20:58:01.762811 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760421    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 20:58:01.762811 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760423    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 20:58:01.762811 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760426    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 20:58:01.763305 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760428    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 20:58:01.763305 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760431    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 20:58:01.763305 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760433    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 20:58:01.763305 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760436    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 20:58:01.763305 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760438    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 20:58:01.763305 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760440    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 20:58:01.763305 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760443    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 20:58:01.763305 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760445    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 20:58:01.763305 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760448    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 20:58:01.763305 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760450    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 20:58:01.763305 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760453    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 20:58:01.763305 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760455    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 20:58:01.763305 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760458    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 20:58:01.763305 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760460    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 20:58:01.763305 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760463    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 20:58:01.763305 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760467    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 20:58:01.763305 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760470    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 20:58:01.763305 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760473    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 20:58:01.763305 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760476    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 20:58:01.763790 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760480    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 20:58:01.763790 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760482    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 20:58:01.763790 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760485    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 20:58:01.763790 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760490    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 20:58:01.763790 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760493    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 20:58:01.763790 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760496    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 20:58:01.763790 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760499    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 20:58:01.763790 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760502    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 20:58:01.763790 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760505    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 20:58:01.763790 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760507    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 20:58:01.763790 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760510    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 20:58:01.763790 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760513    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 20:58:01.763790 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760515    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 20:58:01.763790 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760518    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 20:58:01.763790 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760520    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 20:58:01.763790 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760523    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 20:58:01.763790 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760525    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 20:58:01.763790 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760528    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 20:58:01.763790 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760530    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 20:58:01.764276 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760533    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 20:58:01.764276 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760535    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 20:58:01.764276 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760922    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 20:58:01.764276 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760927    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 20:58:01.764276 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760930    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 20:58:01.764276 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760932    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 20:58:01.764276 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760935    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 20:58:01.764276 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760938    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 20:58:01.764276 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760942    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 20:58:01.764276 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760945    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 20:58:01.764276 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760947    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 20:58:01.764276 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760953    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 20:58:01.764276 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760957    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 20:58:01.764276 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760960    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 20:58:01.764276 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760963    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 20:58:01.764276 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760966    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 20:58:01.764276 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760970    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 20:58:01.764276 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760972    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 20:58:01.764276 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760975    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 20:58:01.764767 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760979    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 20:58:01.764767 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760981    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 20:58:01.764767 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760984    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 20:58:01.764767 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760987    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 20:58:01.764767 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760989    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 20:58:01.764767 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760992    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 20:58:01.764767 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760994    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 20:58:01.764767 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.760997    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 20:58:01.764767 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761000    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 20:58:01.764767 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761002    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 20:58:01.764767 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761005    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 20:58:01.764767 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761007    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 20:58:01.764767 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761010    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 20:58:01.764767 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761013    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 20:58:01.764767 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761016    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 20:58:01.764767 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761018    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 20:58:01.764767 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761021    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 20:58:01.764767 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761023    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 20:58:01.764767 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761026    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 20:58:01.765248 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761028    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 20:58:01.765248 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761031    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 20:58:01.765248 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761033    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 20:58:01.765248 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761038    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 20:58:01.765248 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761042    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 20:58:01.765248 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761062    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 20:58:01.765248 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761066    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 20:58:01.765248 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761069    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 20:58:01.765248 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761072    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 20:58:01.765248 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761075    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 20:58:01.765248 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761078    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 20:58:01.765248 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761081    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 20:58:01.765248 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761084    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 20:58:01.765248 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761087    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 20:58:01.765248 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761090    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 20:58:01.765248 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761093    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 20:58:01.765248 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761096    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 20:58:01.765248 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761098    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 20:58:01.765248 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761101    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 20:58:01.765763 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761103    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 20:58:01.765763 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761106    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 20:58:01.765763 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761109    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 20:58:01.765763 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761112    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 20:58:01.765763 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761115    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 20:58:01.765763 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761117    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 20:58:01.765763 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761120    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 20:58:01.765763 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761123    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 20:58:01.765763 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761125    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 20:58:01.765763 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761128    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 20:58:01.765763 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761130    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 20:58:01.765763 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761133    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 20:58:01.765763 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761136    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 20:58:01.765763 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761138    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 20:58:01.765763 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761141    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 20:58:01.765763 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761143    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 20:58:01.765763 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761147    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 20:58:01.765763 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761150    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 20:58:01.765763 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761153    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 20:58:01.765763 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761156    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 20:58:01.766275 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761159    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 20:58:01.766275 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761162    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 20:58:01.766275 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761164    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 20:58:01.766275 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761167    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 20:58:01.766275 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761170    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 20:58:01.766275 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761173    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 20:58:01.766275 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761177    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 20:58:01.766275 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761179    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 20:58:01.766275 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761182    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 20:58:01.766275 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761185    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 20:58:01.766275 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761187    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 20:58:01.766275 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761190    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 20:58:01.766275 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761193    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 20:58:01.766275 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761195    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 20:58:01.766275 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761198    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 20:58:01.766275 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761200    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 20:58:01.766275 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761203    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 20:58:01.766275 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761205    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 20:58:01.766275 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761208    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 20:58:01.766275 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761210    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 20:58:01.766783 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761213    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 20:58:01.766783 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761215    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 20:58:01.766783 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761218    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 20:58:01.766783 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761220    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 20:58:01.766783 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761304    2558 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 20:58:01.766783 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761352    2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 20:58:01.766783 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761360    2558 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 20:58:01.766783 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761364    2558 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 20:58:01.766783 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761370    2558 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 20:58:01.766783 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761373    2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 20:58:01.766783 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761379    2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 20:58:01.766783 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761384    2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 20:58:01.766783 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761387    2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 20:58:01.766783 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761390    2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 20:58:01.766783 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761393    2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 20:58:01.766783 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761397    2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 20:58:01.766783 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761400    2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 20:58:01.766783 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761403    2558 flags.go:64] FLAG: --cgroup-root=""
Sep 15 20:58:01.766783 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761407    2558 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 20:58:01.766783 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761410    2558 flags.go:64] FLAG: --client-ca-file=""
Sep 15 20:58:01.766783 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761413    2558 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 20:58:01.766783 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761417    2558 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 20:58:01.766783 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761421    2558 flags.go:64] FLAG: --cluster-domain=""
Sep 15 20:58:01.766783 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761424    2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 20:58:01.767417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761427    2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 20:58:01.767417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761430    2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 20:58:01.767417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761434    2558 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 20:58:01.767417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761438    2558 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 20:58:01.767417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761441    2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 20:58:01.767417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761444    2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 20:58:01.767417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761448    2558 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 20:58:01.767417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761451    2558 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 20:58:01.767417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761454    2558 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 20:58:01.767417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761457    2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 20:58:01.767417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761460    2558 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 20:58:01.767417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761464    2558 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 20:58:01.767417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761468    2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 20:58:01.767417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761471    2558 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 20:58:01.767417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761474    2558 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 20:58:01.767417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761477    2558 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 20:58:01.767417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761480    2558 flags.go:64] FLAG: --enable-server="true"
Sep 15 20:58:01.767417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761483    2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 20:58:01.767417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761490    2558 flags.go:64] FLAG: --event-burst="100"
Sep 15 20:58:01.767417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761493    2558 flags.go:64] FLAG: --event-qps="50"
Sep 15 20:58:01.767417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761496    2558 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 20:58:01.767417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761499    2558 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 20:58:01.767417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761502    2558 flags.go:64] FLAG: --eviction-hard=""
Sep 15 20:58:01.767417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761506    2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 20:58:01.768105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761509    2558 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 20:58:01.768105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761512    2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 20:58:01.768105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761515    2558 flags.go:64] FLAG: --eviction-soft=""
Sep 15 20:58:01.768105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761519    2558 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 20:58:01.768105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761522    2558 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 20:58:01.768105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761525    2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 20:58:01.768105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761527    2558 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 20:58:01.768105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761530    2558 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 20:58:01.768105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761533    2558 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 20:58:01.768105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761536    2558 flags.go:64] FLAG: --feature-gates=""
Sep 15 20:58:01.768105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761540    2558 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 20:58:01.768105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761543    2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 20:58:01.768105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761546    2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 20:58:01.768105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761550    2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 20:58:01.768105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761553    2558 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 20:58:01.768105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761556    2558 flags.go:64] FLAG: --help="false"
Sep 15 20:58:01.768105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761559    2558 flags.go:64] FLAG: --hostname-override="ip-10-0-142-207.ec2.internal"
Sep 15 20:58:01.768105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761562    2558 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 20:58:01.768105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761565    2558 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 20:58:01.768105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761568    2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 20:58:01.768105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761571    2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 20:58:01.768105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761575    2558 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 20:58:01.768105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761578    2558 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 20:58:01.768105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761580    2558 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761584    2558 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761587    2558 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761591    2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761594    2558 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761597    2558 flags.go:64] FLAG: --kube-reserved=""
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761600    2558 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761603    2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761606    2558 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761609    2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761612    2558 flags.go:64] FLAG: --lock-file=""
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761615    2558 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761618    2558 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761622    2558 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761627    2558 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761631    2558 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761634    2558 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761637    2558 flags.go:64] FLAG: --logging-format="text"
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761639    2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761659    2558 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761662    2558 flags.go:64] FLAG: --manifest-url=""
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761666    2558 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761671    2558 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761674    2558 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761678    2558 flags.go:64] FLAG: --max-pods="110"
Sep 15 20:58:01.768731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761681    2558 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 20:58:01.769689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761684    2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 20:58:01.769689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761687    2558 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 20:58:01.769689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761691    2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 20:58:01.769689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761693    2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 20:58:01.769689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761697    2558 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 20:58:01.769689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761700    2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 20:58:01.769689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761707    2558 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 20:58:01.769689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761711    2558 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 20:58:01.769689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761714    2558 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 20:58:01.769689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761719    2558 flags.go:64] FLAG: --pod-cidr=""
Sep 15 20:58:01.769689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761722    2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 20:58:01.769689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761729    2558 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 20:58:01.769689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761732    2558 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 20:58:01.769689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761735    2558 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 20:58:01.769689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761738    2558 flags.go:64] FLAG: --port="10250"
Sep 15 20:58:01.769689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761741    2558 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 20:58:01.769689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761744    2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0902390afd34f2c84"
Sep 15 20:58:01.769689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761748    2558 flags.go:64] FLAG: --qos-reserved=""
Sep 15 20:58:01.769689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761751    2558 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 20:58:01.769689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761754    2558 flags.go:64] FLAG: --register-node="true"
Sep 15 20:58:01.769689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761758    2558 flags.go:64] FLAG: --register-with-taints=""
Sep 15 20:58:01.769689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761761    2558 flags.go:64] FLAG: --registry-burst="10"
Sep 15 20:58:01.769689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761764    2558 flags.go:64] FLAG: --registry-qps="5"
Sep 15 20:58:01.769689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761767    2558 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761770    2558 flags.go:64] FLAG: --reserved-memory=""
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761774    2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761777    2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761780    2558 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761783    2558 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761786    2558 flags.go:64] FLAG: --runonce="false"
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761789    2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761792    2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761796    2558 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761798    2558 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761802    2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761805    2558 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761808    2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761811    2558 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761814    2558 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761819    2558 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761822    2558 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761825    2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761829    2558 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761832    2558 flags.go:64] FLAG: --system-cgroups=""
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761835    2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761841    2558 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761844    2558 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761847    2558 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 20:58:01.770735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761852    2558 flags.go:64] FLAG: --tls-min-version=""
Sep 15 20:58:01.771437 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761855    2558 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 20:58:01.771437 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761857    2558 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 20:58:01.771437 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761861    2558 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 20:58:01.771437 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761864    2558 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 20:58:01.771437 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761867    2558 flags.go:64] FLAG: --v="2"
Sep 15 20:58:01.771437 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761871    2558 flags.go:64] FLAG: --version="false"
Sep 15 20:58:01.771437 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761876    2558 flags.go:64] FLAG: --vmodule=""
Sep 15 20:58:01.771437 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761879    2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 20:58:01.771437 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.761882    2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 20:58:01.771437 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.761999    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 20:58:01.771437 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762003    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 20:58:01.771437 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762007    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 20:58:01.771437 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762010    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 20:58:01.771437 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762013    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 20:58:01.771437 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762015    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 20:58:01.771437 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762018    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 20:58:01.771437 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762021    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 20:58:01.771437 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762023    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 20:58:01.771437 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762027    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 20:58:01.771437 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762031    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 20:58:01.771437 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762035    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 20:58:01.771437 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762040    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 20:58:01.771997 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762045    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 20:58:01.771997 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762062    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 20:58:01.771997 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762065    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 20:58:01.771997 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762069    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 20:58:01.771997 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762071    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 20:58:01.771997 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762074    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 20:58:01.771997 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762077    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 20:58:01.771997 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762080    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 20:58:01.771997 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762083    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 20:58:01.771997 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762085    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 20:58:01.771997 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762088    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 20:58:01.771997 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762090    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 20:58:01.771997 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762093    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 20:58:01.771997 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762096    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 20:58:01.771997 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762099    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 20:58:01.771997 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762103    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 20:58:01.771997 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762106    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 20:58:01.771997 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762108    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 20:58:01.771997 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762111    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 20:58:01.772495 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762115    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 20:58:01.772495 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762119    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 20:58:01.772495 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762122    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 20:58:01.772495 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762125    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 20:58:01.772495 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762128    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 20:58:01.772495 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762130    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 20:58:01.772495 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762133    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 20:58:01.772495 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762136    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 20:58:01.772495 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762138    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 20:58:01.772495 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762140    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 20:58:01.772495 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762143    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 20:58:01.772495 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762146    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 20:58:01.772495 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762148    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 20:58:01.772495 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762152    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 20:58:01.772495 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762155    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 20:58:01.772495 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762157    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 20:58:01.772495 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762160    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 20:58:01.772495 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762163    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 20:58:01.772495 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762165    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 20:58:01.772495 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762168    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 20:58:01.772992 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762171    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 20:58:01.772992 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762173    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 20:58:01.772992 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762176    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 20:58:01.772992 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762179    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 20:58:01.772992 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762183    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 20:58:01.772992 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762186    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 20:58:01.772992 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762189    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 20:58:01.772992 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762191    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 20:58:01.772992 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762194    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 20:58:01.772992 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762197    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 20:58:01.772992 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762200    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 20:58:01.772992 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762202    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 20:58:01.772992 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762205    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 20:58:01.772992 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762207    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 20:58:01.772992 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762210    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 20:58:01.772992 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762212    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 20:58:01.772992 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762215    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 20:58:01.772992 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762217    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 20:58:01.772992 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762220    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 20:58:01.773537 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762222    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 20:58:01.773537 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762225    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 20:58:01.773537 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762227    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 20:58:01.773537 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762230    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 20:58:01.773537 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762232    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 20:58:01.773537 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762235    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 20:58:01.773537 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762239    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 20:58:01.773537 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762242    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 20:58:01.773537 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762244    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 20:58:01.773537 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762247    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 20:58:01.773537 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762249    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 20:58:01.773537 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762252    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 20:58:01.773537 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762255    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 20:58:01.773537 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762257    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 20:58:01.773537 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762260    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 20:58:01.773537 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762262    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 20:58:01.773537 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762264    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 20:58:01.773537 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762267    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 20:58:01.773537 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762269    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 20:58:01.774078 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762272    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 20:58:01.774078 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762275    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 20:58:01.774078 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762277    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 20:58:01.774078 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762279    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 20:58:01.774078 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762282    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 20:58:01.774078 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762285    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 20:58:01.774078 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762287    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 20:58:01.774078 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762290    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 20:58:01.774078 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.762292    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 20:58:01.774078 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.762862    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 20:58:01.774078 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.771964    2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 20:58:01.774078 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.771989    2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 20:58:01.774078 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.772012    2558 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 20:58:01.774078 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.772019    2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 20:58:01.774078 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772097    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 20:58:01.774475 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772103    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 20:58:01.774475 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772106    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 20:58:01.774475 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772109    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 20:58:01.774475 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772113    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 20:58:01.774475 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772116    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 20:58:01.774475 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772120    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 20:58:01.774475 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772122    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 20:58:01.774475 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772125    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 20:58:01.774475 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772128    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 20:58:01.774475 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772131    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 20:58:01.774475 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772134    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 20:58:01.774475 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772137    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 20:58:01.774475 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772140    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 20:58:01.774475 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772143    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 20:58:01.774475 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772146    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 20:58:01.774475 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772149    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 20:58:01.774475 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772152    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 20:58:01.774475 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772154    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 20:58:01.774475 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772158    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 20:58:01.774475 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772160    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 20:58:01.774987 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772163    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 20:58:01.774987 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772166    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 20:58:01.774987 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772169    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 20:58:01.774987 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772171    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 20:58:01.774987 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772175    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 20:58:01.774987 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772178    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 20:58:01.774987 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772181    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 20:58:01.774987 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772184    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 20:58:01.774987 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772187    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 20:58:01.774987 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772190    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 20:58:01.774987 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772192    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 20:58:01.774987 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772196    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 20:58:01.774987 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772201    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 20:58:01.774987 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772203    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 20:58:01.774987 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772206    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 20:58:01.774987 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772209    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 20:58:01.774987 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772212    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 20:58:01.774987 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772214    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 20:58:01.774987 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772217    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 20:58:01.775517 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772220    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 20:58:01.775517 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772222    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 20:58:01.775517 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772225    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 20:58:01.775517 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772229    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 20:58:01.775517 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772233    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 20:58:01.775517 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772236    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 20:58:01.775517 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772239    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 20:58:01.775517 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772242    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 20:58:01.775517 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772245    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 20:58:01.775517 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772248    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 20:58:01.775517 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772250    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 20:58:01.775517 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772253    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 20:58:01.775517 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772255    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 20:58:01.775517 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772258    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 20:58:01.775517 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772261    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 20:58:01.775517 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772264    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 20:58:01.775517 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772268    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 20:58:01.775517 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772272    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 20:58:01.775517 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772275    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 20:58:01.776014 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772278    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 20:58:01.776014 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772282    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 20:58:01.776014 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772285    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 20:58:01.776014 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772288    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 20:58:01.776014 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772291    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 20:58:01.776014 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772294    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 20:58:01.776014 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772297    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 20:58:01.776014 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772299    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 20:58:01.776014 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772302    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 20:58:01.776014 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772305    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 20:58:01.776014 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772308    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 20:58:01.776014 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772310    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 20:58:01.776014 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772313    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 20:58:01.776014 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772316    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 20:58:01.776014 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772318    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 20:58:01.776014 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772321    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 20:58:01.776014 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772323    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 20:58:01.776014 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772326    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 20:58:01.776014 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772329    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 20:58:01.776549 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772332    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 20:58:01.776549 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772335    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 20:58:01.776549 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772338    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 20:58:01.776549 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772342    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 20:58:01.776549 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772344    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 20:58:01.776549 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772347    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 20:58:01.776549 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772349    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 20:58:01.776549 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772352    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 20:58:01.776549 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772355    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 20:58:01.776549 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772357    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 20:58:01.776549 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772360    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 20:58:01.776549 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772363    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 20:58:01.776549 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772366    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 20:58:01.776549 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772368    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 20:58:01.776549 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772371    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 20:58:01.776549 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772374    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 20:58:01.776549 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772377    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 20:58:01.776549 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772379    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 20:58:01.776549 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772382    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 20:58:01.776549 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772385    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 20:58:01.777100 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772388    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 20:58:01.777100 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.772393    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 20:58:01.777100 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772523    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 20:58:01.777100 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772528    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 20:58:01.777100 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772531    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 20:58:01.777100 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772535    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 20:58:01.777100 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772538    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 20:58:01.777100 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772541    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 20:58:01.777100 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772544    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 20:58:01.777100 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772546    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 20:58:01.777100 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772550    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 20:58:01.777100 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772553    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 20:58:01.777100 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772556    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 20:58:01.777100 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772558    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 20:58:01.777100 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772562    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 20:58:01.777490 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772565    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 20:58:01.777490 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772568    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 20:58:01.777490 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772570    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 20:58:01.777490 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772573    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 20:58:01.777490 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772576    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 20:58:01.777490 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772578    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 20:58:01.777490 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772581    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 20:58:01.777490 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772584    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 20:58:01.777490 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772586    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 20:58:01.777490 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772589    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 20:58:01.777490 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772592    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 20:58:01.777490 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772595    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 20:58:01.777490 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772598    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 20:58:01.777490 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772601    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 20:58:01.777490 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772603    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 20:58:01.777490 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772606    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 20:58:01.777490 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772609    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 20:58:01.777490 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772611    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 20:58:01.777490 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772614    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 20:58:01.777490 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772617    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 20:58:01.778013 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772619    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 20:58:01.778013 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772622    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 20:58:01.778013 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772625    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 20:58:01.778013 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772628    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 20:58:01.778013 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772631    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 20:58:01.778013 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772636    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 20:58:01.778013 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772639    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 20:58:01.778013 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772642    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 20:58:01.778013 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772645    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 20:58:01.778013 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772648    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 20:58:01.778013 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772650    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 20:58:01.778013 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772653    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 20:58:01.778013 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772657    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 20:58:01.778013 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772660    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 20:58:01.778013 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772663    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 20:58:01.778013 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772666    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 20:58:01.778013 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772670    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 20:58:01.778013 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772673    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 20:58:01.778013 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772676    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 20:58:01.778608 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772678    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 20:58:01.778608 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772681    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 20:58:01.778608 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772683    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 20:58:01.778608 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772686    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 20:58:01.778608 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772689    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 20:58:01.778608 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772692    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 20:58:01.778608 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772694    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 20:58:01.778608 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772697    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 20:58:01.778608 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772699    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 20:58:01.778608 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772702    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 20:58:01.778608 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772705    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 20:58:01.778608 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772707    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 20:58:01.778608 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772709    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 20:58:01.778608 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772712    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 20:58:01.778608 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772715    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 20:58:01.778608 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772717    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 20:58:01.778608 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772720    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 20:58:01.778608 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772722    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 20:58:01.778608 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772725    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 20:58:01.778608 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772727    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 20:58:01.779152 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772730    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 20:58:01.779152 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772732    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 20:58:01.779152 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772735    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 20:58:01.779152 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772737    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 20:58:01.779152 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772740    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 20:58:01.779152 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772743    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 20:58:01.779152 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772746    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 20:58:01.779152 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772749    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 20:58:01.779152 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772751    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 20:58:01.779152 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772754    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 20:58:01.779152 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772757    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 20:58:01.779152 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772759    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 20:58:01.779152 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772762    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 20:58:01.779152 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772766    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 20:58:01.779152 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772769    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 20:58:01.779152 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772772    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 20:58:01.779152 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772775    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 20:58:01.779152 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772778    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 20:58:01.779152 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772781    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 20:58:01.779689 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772784    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 20:58:01.779689 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772787    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 20:58:01.779689 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772790    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 20:58:01.779689 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772793    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 20:58:01.779689 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772795    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 20:58:01.779689 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772798    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 20:58:01.779689 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772801    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 20:58:01.779689 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:01.772804    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 20:58:01.779689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.772809    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 20:58:01.779689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.772943    2558 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 20:58:01.779689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.777164    2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 20:58:01.779689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.778031    2558 server.go:1019] "Starting client certificate rotation"
Sep 15 20:58:01.779689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.778146    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 20:58:01.779689 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.778181    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 20:58:01.780710 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.780695    2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 20:58:01.781998 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.781977    2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 20:58:01.791817 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.791789    2558 log.go:25] "Validated CRI v1 runtime API"
Sep 15 20:58:01.796586 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.796565    2558 log.go:25] "Validated CRI v1 image API"
Sep 15 20:58:01.797503 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.797488    2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 20:58:01.801160 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.801133    2558 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 902a17d8-d192-4581-8d27-ffa0499f95b9:/dev/nvme0n1p4 98d269e7-f9d9-4df0-8ce7-e4bcd6252aa7:/dev/nvme0n1p3]
Sep 15 20:58:01.801244 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.801159    2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 20:58:01.806619 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.806594    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 20:58:01.807485 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.807343    2558 manager.go:216] Machine: {Timestamp:2026-09-15 20:58:01.805833008 +0000 UTC m=+0.296478729 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102008 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2af27eb3c9e76a6b10abd251cb61aa SystemUUID:ec2af27e-b3c9-e76a-6b10-abd251cb61aa BootID:696e2c05-d8e1-451b-a4e3-a2f3ea0b8582 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e1:63:d5:65 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e1:63:d5:65 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:37:68:d2:be:2e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 20:58:01.807485 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.807464    2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 20:58:01.807643 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.807561    2558 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 20:58:01.810212 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.810178    2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 20:58:01.810360 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.810214    2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-207.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 20:58:01.810925 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.810913    2558 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 20:58:01.810960 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.810927    2558 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 20:58:01.810960 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.810935    2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 20:58:01.811518 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.811505    2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 20:58:01.812641 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.812629    2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 20:58:01.816593 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.816579    2558 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 20:58:01.816720 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.816711    2558 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 20:58:01.816757 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.816743    2558 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 20:58:01.816757 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.816753    2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 20:58:01.816815 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.816769    2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 20:58:01.816815 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.816779    2558 kubelet.go:398] "Adding apiserver pod source"
Sep 15 20:58:01.816815 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.816789    2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 20:58:01.817202 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.817188    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 20:58:01.817271 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.817207    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 20:58:01.819667 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.819650    2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 20:58:01.822163 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.822148    2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 20:58:01.822208 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.822182    2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 20:58:01.823727 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.823715    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 20:58:01.823772 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.823734    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 20:58:01.823772 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.823742    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 20:58:01.823772 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.823750    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 20:58:01.823772 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.823759    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 20:58:01.823772 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.823764    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 20:58:01.823907 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.823777    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 20:58:01.823907 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.823784    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 20:58:01.823907 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.823791    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 20:58:01.823907 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.823796    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 20:58:01.823907 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.823806    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 20:58:01.823907 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.823811    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 20:58:01.824408 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.824398    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 20:58:01.824408 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.824408    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 20:58:01.826521 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.826496    2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-207.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 20:58:01.826615 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:01.826524    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-207.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 20:58:01.826615 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:01.826546    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 20:58:01.828234 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.828218    2558 server.go:1268] "Started kubelet"
Sep 15 20:58:01.828346 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.828290    2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 20:58:01.828443 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.828374    2558 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 20:58:01.828443 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.828372    2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 20:58:01.829132 ip-10-0-142-207 systemd[1]: Started Kubernetes Kubelet.
Sep 15 20:58:01.830179 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.830158    2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 20:58:01.830270 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.830185    2558 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 20:58:01.833942 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.833754    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z57kp"
Sep 15 20:58:01.834540 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:01.834508    2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 20:58:01.835697 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:01.834908    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-207.ec2.internal.18d59a1e49d75f71  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-207.ec2.internal,UID:ip-10-0-142-207.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-207.ec2.internal,},FirstTimestamp:2026-09-15 20:58:01.828188017 +0000 UTC m=+0.318833741,LastTimestamp:2026-09-15 20:58:01.828188017 +0000 UTC m=+0.318833741,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-207.ec2.internal,}"
Sep 15 20:58:01.836284 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.836268    2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 20:58:01.836364 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.836292    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 20:58:01.836408 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:01.836367    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-207.ec2.internal\" not found"
Sep 15 20:58:01.836833 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.836813    2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 20:58:01.836949 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.836840    2558 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 20:58:01.836949 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.836851    2558 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 20:58:01.836949 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.836925    2558 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 20:58:01.836949 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.836933    2558 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 20:58:01.837641 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.837626    2558 factory.go:55] Registering systemd factory
Sep 15 20:58:01.837641 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.837643    2558 factory.go:223] Registration of the systemd container factory successfully
Sep 15 20:58:01.837857 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.837845    2558 factory.go:153] Registering CRI-O factory
Sep 15 20:58:01.837891 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.837860    2558 factory.go:223] Registration of the crio container factory successfully
Sep 15 20:58:01.837926 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.837914    2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 20:58:01.837961 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.837940    2558 factory.go:103] Registering Raw factory
Sep 15 20:58:01.837961 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.837956    2558 manager.go:1196] Started watching for new ooms in manager
Sep 15 20:58:01.838321 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.838306    2558 manager.go:349] Starting recovery of all containers
Sep 15 20:58:01.839326 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.839306    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z57kp"
Sep 15 20:58:01.843392 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:01.843358    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 15 20:58:01.843507 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:01.843477    2558 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-207.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 15 20:58:01.851441 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.851409    2558 manager.go:354] Recovery completed
Sep 15 20:58:01.853684 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:01.853033    2558 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 15 20:58:01.857527 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.857497    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 20:58:01.859932 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.859900    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-207.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 20:58:01.859932 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.859934    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-207.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 20:58:01.860088 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.859948    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-207.ec2.internal" event="NodeHasSufficientPID"
Sep 15 20:58:01.860647 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.860626    2558 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 20:58:01.860747 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.860652    2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 20:58:01.860747 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.860642    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 20:58:01.860747 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.860698    2558 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 20:58:01.861478 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.861462    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-207.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 20:58:01.861557 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.861488    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-207.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 20:58:01.861557 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.861500    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-207.ec2.internal" event="NodeHasSufficientPID"
Sep 15 20:58:01.861626 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:01.861511    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-207.ec2.internal.18d59a1e4bbb8da3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-207.ec2.internal,UID:ip-10-0-142-207.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-207.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-207.ec2.internal,},FirstTimestamp:2026-09-15 20:58:01.859919267 +0000 UTC m=+0.350564992,LastTimestamp:2026-09-15 20:58:01.859919267 +0000 UTC m=+0.350564992,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-207.ec2.internal,}"
Sep 15 20:58:01.862810 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.862796    2558 policy_none.go:49] "None policy: Start"
Sep 15 20:58:01.862810 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.862812    2558 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 20:58:01.862963 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.862822    2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 20:58:01.864819 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.864806    2558 policy_none.go:47] "Start"
Sep 15 20:58:01.915153 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.904757    2558 manager.go:337] "Starting Device Plugin manager"
Sep 15 20:58:01.915153 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:01.904798    2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 20:58:01.915153 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.904811    2558 server.go:87] "Starting device plugin registration server"
Sep 15 20:58:01.915153 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.905250    2558 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 20:58:01.915153 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.905263    2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 20:58:01.915153 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.905360    2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 20:58:01.915153 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.905447    2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 20:58:01.915153 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.905454    2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 20:58:01.915153 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:01.906489    2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 20:58:01.915153 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:01.906543    2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-207.ec2.internal\" not found"
Sep 15 20:58:01.934386 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.934344    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 20:58:01.935574 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.935557    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 20:58:01.935638 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.935612    2558 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 20:58:01.935684 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.935639    2558 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 20:58:01.935729 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:01.935693    2558 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 20:58:01.937970 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:01.937946    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 20:58:02.006113 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.006002    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 20:58:02.007067 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.007034    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-207.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 20:58:02.007149 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.007084    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-207.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 20:58:02.007149 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.007097    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-207.ec2.internal" event="NodeHasSufficientPID"
Sep 15 20:58:02.007149 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.007121    2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-207.ec2.internal"
Sep 15 20:58:02.013220 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.013193    2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-207.ec2.internal"
Sep 15 20:58:02.013220 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:02.013222    2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-207.ec2.internal\": node \"ip-10-0-142-207.ec2.internal\" not found"
Sep 15 20:58:02.032915 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:02.032887    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-207.ec2.internal\" not found"
Sep 15 20:58:02.035888 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.035867    2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-207.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-207.ec2.internal"]
Sep 15 20:58:02.035949 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.035941    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 20:58:02.036913 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.036897    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-207.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 20:58:02.036996 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.036923    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-207.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 20:58:02.036996 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.036933    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-207.ec2.internal" event="NodeHasSufficientPID"
Sep 15 20:58:02.038072 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.038038    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/344eacc22c151cd63c505c6fcd3eb468-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-207.ec2.internal\" (UID: \"344eacc22c151cd63c505c6fcd3eb468\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-207.ec2.internal"
Sep 15 20:58:02.038132 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.038088    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/344eacc22c151cd63c505c6fcd3eb468-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-207.ec2.internal\" (UID: \"344eacc22c151cd63c505c6fcd3eb468\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-207.ec2.internal"
Sep 15 20:58:02.039320 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.039307    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 20:58:02.039460 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.039434    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-207.ec2.internal"
Sep 15 20:58:02.039515 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.039473    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 20:58:02.040094 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.040079    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-207.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 20:58:02.040173 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.040103    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-207.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 20:58:02.040173 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.040121    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-207.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 20:58:02.040173 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.040129    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-207.ec2.internal" event="NodeHasSufficientPID"
Sep 15 20:58:02.040173 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.040104    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-207.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 20:58:02.040297 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.040193    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-207.ec2.internal" event="NodeHasSufficientPID"
Sep 15 20:58:02.042407 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.042392    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-207.ec2.internal"
Sep 15 20:58:02.042486 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.042418    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 20:58:02.043184 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.043169    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-207.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 20:58:02.043279 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.043192    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-207.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 20:58:02.043279 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.043201    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-207.ec2.internal" event="NodeHasSufficientPID"
Sep 15 20:58:02.074006 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:02.073972    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-207.ec2.internal\" not found" node="ip-10-0-142-207.ec2.internal"
Sep 15 20:58:02.077565 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:02.077548    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-207.ec2.internal\" not found" node="ip-10-0-142-207.ec2.internal"
Sep 15 20:58:02.133271 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:02.133241    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-207.ec2.internal\" not found"
Sep 15 20:58:02.139088 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.139066    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/344eacc22c151cd63c505c6fcd3eb468-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-207.ec2.internal\" (UID: \"344eacc22c151cd63c505c6fcd3eb468\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-207.ec2.internal"
Sep 15 20:58:02.139165 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.139093    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/344eacc22c151cd63c505c6fcd3eb468-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-207.ec2.internal\" (UID: \"344eacc22c151cd63c505c6fcd3eb468\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-207.ec2.internal"
Sep 15 20:58:02.139165 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.139113    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1c3857c19d72e89622f4c04f57a3da21-config\") pod \"kube-apiserver-proxy-ip-10-0-142-207.ec2.internal\" (UID: \"1c3857c19d72e89622f4c04f57a3da21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-207.ec2.internal"
Sep 15 20:58:02.139235 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.139174    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/344eacc22c151cd63c505c6fcd3eb468-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-207.ec2.internal\" (UID: \"344eacc22c151cd63c505c6fcd3eb468\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-207.ec2.internal"
Sep 15 20:58:02.139235 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.139185    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/344eacc22c151cd63c505c6fcd3eb468-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-207.ec2.internal\" (UID: \"344eacc22c151cd63c505c6fcd3eb468\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-207.ec2.internal"
Sep 15 20:58:02.233956 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:02.233925    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-207.ec2.internal\" not found"
Sep 15 20:58:02.239347 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.239310    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1c3857c19d72e89622f4c04f57a3da21-config\") pod \"kube-apiserver-proxy-ip-10-0-142-207.ec2.internal\" (UID: \"1c3857c19d72e89622f4c04f57a3da21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-207.ec2.internal"
Sep 15 20:58:02.240936 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.239604    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1c3857c19d72e89622f4c04f57a3da21-config\") pod \"kube-apiserver-proxy-ip-10-0-142-207.ec2.internal\" (UID: \"1c3857c19d72e89622f4c04f57a3da21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-207.ec2.internal"
Sep 15 20:58:02.334689 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:02.334607    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-207.ec2.internal\" not found"
Sep 15 20:58:02.377185 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.377150    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-207.ec2.internal"
Sep 15 20:58:02.380732 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.380709    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-207.ec2.internal"
Sep 15 20:58:02.435709 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:02.435678    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-207.ec2.internal\" not found"
Sep 15 20:58:02.536248 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:02.536215    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-207.ec2.internal\" not found"
Sep 15 20:58:02.636764 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:02.636675    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-207.ec2.internal\" not found"
Sep 15 20:58:02.737153 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:02.737120    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-207.ec2.internal\" not found"
Sep 15 20:58:02.778500 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.778466    2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 20:58:02.778947 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.778647    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 20:58:02.837567 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:02.837532    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-207.ec2.internal\" not found"
Sep 15 20:58:02.848682 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.848643    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 20:53:02 +0000 UTC" deadline="2028-02-27 23:40:34.829658719 +0000 UTC"
Sep 15 20:58:02.848682 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.848677    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12722h42m31.980985192s"
Sep 15 20:58:02.902925 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.902856    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 20:58:02.937368 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.937336    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-207.ec2.internal"
Sep 15 20:58:02.948033 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.948002    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 20:58:02.949524 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.949508    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-207.ec2.internal"
Sep 15 20:58:02.955951 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.955931    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 20:58:02.985137 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.985111    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 20:58:02.987310 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:02.987280    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod344eacc22c151cd63c505c6fcd3eb468.slice/crio-8f48ad1f2c758ac124029a18c3d6f5a94bb4d837909ed72c73e721e258c21dbd WatchSource:0}: Error finding container 8f48ad1f2c758ac124029a18c3d6f5a94bb4d837909ed72c73e721e258c21dbd: Status 404 returned error can't find the container with id 8f48ad1f2c758ac124029a18c3d6f5a94bb4d837909ed72c73e721e258c21dbd
Sep 15 20:58:02.987636 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:02.987614    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3857c19d72e89622f4c04f57a3da21.slice/crio-5ce6297cd84547bd86053f75b86055d3ad3107360b568689b5da0f60d0067d45 WatchSource:0}: Error finding container 5ce6297cd84547bd86053f75b86055d3ad3107360b568689b5da0f60d0067d45: Status 404 returned error can't find the container with id 5ce6297cd84547bd86053f75b86055d3ad3107360b568689b5da0f60d0067d45
Sep 15 20:58:02.994337 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:02.993860    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 20:58:03.153591 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.153497    2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 20:58:03.818013 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.817979    2558 apiserver.go:52] "Watching apiserver"
Sep 15 20:58:03.827352 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.827203    2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 20:58:03.827628 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.827606    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-z2vkz","openshift-network-operator/iptables-alerter-wnktt","kube-system/global-pull-secret-syncer-qphq9","kube-system/konnectivity-agent-pc79j","openshift-cluster-node-tuning-operator/tuned-fl48z","openshift-dns/node-resolver-nwspf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-207.ec2.internal","openshift-multus/network-metrics-daemon-vf58t","openshift-ovn-kubernetes/ovnkube-node-6fp7g","kube-system/kube-apiserver-proxy-ip-10-0-142-207.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk","openshift-image-registry/node-ca-xkmbd","openshift-multus/multus-additional-cni-plugins-hrmmd","openshift-multus/multus-vjzlh"]
Sep 15 20:58:03.830398 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.830372    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hrmmd"
Sep 15 20:58:03.832805 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.832782    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wnktt"
Sep 15 20:58:03.833528 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.833326    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 20:58:03.833528 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.833362    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 20:58:03.833528 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.833507    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 20:58:03.833528 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.833507    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 20:58:03.833792 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.833511    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 20:58:03.833792 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.833594    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sct9r\""
Sep 15 20:58:03.834875 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.834852    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:03.834990 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:03.834932    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:03.835745 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.835425    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 20:58:03.837343 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.836208    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 20:58:03.837343 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.836416    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7xrrg\""
Sep 15 20:58:03.837343 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.836624    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 20:58:03.837343 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.836906    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 20:58:03.839087 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.839063    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pc79j"
Sep 15 20:58:03.841433 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.841409    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:03.844082 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.843643    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 20:58:03.844082 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.843969    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 20:58:03.844082 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.844003    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pnmqq\""
Sep 15 20:58:03.846483 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.844436    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nwspf"
Sep 15 20:58:03.846483 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.845845    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4ng2n\""
Sep 15 20:58:03.846483 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.846253    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 20:58:03.846701 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.846593    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:03.846757 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:03.846715    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:03.848980 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.848127    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 20:58:03.848980 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.848286    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pxpd8\""
Sep 15 20:58:03.848980 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.848379    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 20:58:03.848980 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.848771    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 20:58:03.849655 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.849330    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:03.849655 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.849372    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/49a4e9ea-347b-4653-9210-e9b142172bdd-iptables-alerter-script\") pod \"iptables-alerter-wnktt\" (UID: \"49a4e9ea-347b-4653-9210-e9b142172bdd\") " pod="openshift-network-operator/iptables-alerter-wnktt"
Sep 15 20:58:03.849655 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.849423    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/695e9f08-df31-4f11-b54e-583d69edbb05-cnibin\") pod \"multus-additional-cni-plugins-hrmmd\" (UID: \"695e9f08-df31-4f11-b54e-583d69edbb05\") " pod="openshift-multus/multus-additional-cni-plugins-hrmmd"
Sep 15 20:58:03.849655 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:03.849444    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:03.849655 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.849456    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ef60e7a1-62c7-46aa-8fd8-4043168ffa94-konnectivity-ca\") pod \"konnectivity-agent-pc79j\" (UID: \"ef60e7a1-62c7-46aa-8fd8-4043168ffa94\") " pod="kube-system/konnectivity-agent-pc79j"
Sep 15 20:58:03.849655 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.849503    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/695e9f08-df31-4f11-b54e-583d69edbb05-cni-binary-copy\") pod \"multus-additional-cni-plugins-hrmmd\" (UID: \"695e9f08-df31-4f11-b54e-583d69edbb05\") " pod="openshift-multus/multus-additional-cni-plugins-hrmmd"
Sep 15 20:58:03.849655 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.849533    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc7c2\" (UniqueName: \"kubernetes.io/projected/49a4e9ea-347b-4653-9210-e9b142172bdd-kube-api-access-zc7c2\") pod \"iptables-alerter-wnktt\" (UID: \"49a4e9ea-347b-4653-9210-e9b142172bdd\") " pod="openshift-network-operator/iptables-alerter-wnktt"
Sep 15 20:58:03.849655 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.849564    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/932cce37-6a6a-4968-8cbe-d163e1da16f0-kubelet-config\") pod \"global-pull-secret-syncer-qphq9\" (UID: \"932cce37-6a6a-4968-8cbe-d163e1da16f0\") " pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:03.849655 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.849591    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/932cce37-6a6a-4968-8cbe-d163e1da16f0-original-pull-secret\") pod \"global-pull-secret-syncer-qphq9\" (UID: \"932cce37-6a6a-4968-8cbe-d163e1da16f0\") " pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:03.849655 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.849625    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ef60e7a1-62c7-46aa-8fd8-4043168ffa94-agent-certs\") pod \"konnectivity-agent-pc79j\" (UID: \"ef60e7a1-62c7-46aa-8fd8-4043168ffa94\") " pod="kube-system/konnectivity-agent-pc79j"
Sep 15 20:58:03.849655 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.849649    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/695e9f08-df31-4f11-b54e-583d69edbb05-system-cni-dir\") pod \"multus-additional-cni-plugins-hrmmd\" (UID: \"695e9f08-df31-4f11-b54e-583d69edbb05\") " pod="openshift-multus/multus-additional-cni-plugins-hrmmd"
Sep 15 20:58:03.850207 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.849679    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/695e9f08-df31-4f11-b54e-583d69edbb05-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hrmmd\" (UID: \"695e9f08-df31-4f11-b54e-583d69edbb05\") " pod="openshift-multus/multus-additional-cni-plugins-hrmmd"
Sep 15 20:58:03.850207 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.849707    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb9wx\" (UniqueName: \"kubernetes.io/projected/695e9f08-df31-4f11-b54e-583d69edbb05-kube-api-access-tb9wx\") pod \"multus-additional-cni-plugins-hrmmd\" (UID: \"695e9f08-df31-4f11-b54e-583d69edbb05\") " pod="openshift-multus/multus-additional-cni-plugins-hrmmd"
Sep 15 20:58:03.850207 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.849739    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/695e9f08-df31-4f11-b54e-583d69edbb05-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hrmmd\" (UID: \"695e9f08-df31-4f11-b54e-583d69edbb05\") " pod="openshift-multus/multus-additional-cni-plugins-hrmmd"
Sep 15 20:58:03.850207 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.849779    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/695e9f08-df31-4f11-b54e-583d69edbb05-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hrmmd\" (UID: \"695e9f08-df31-4f11-b54e-583d69edbb05\") " pod="openshift-multus/multus-additional-cni-plugins-hrmmd"
Sep 15 20:58:03.850207 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.849820    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49a4e9ea-347b-4653-9210-e9b142172bdd-host-slash\") pod \"iptables-alerter-wnktt\" (UID: \"49a4e9ea-347b-4653-9210-e9b142172bdd\") " pod="openshift-network-operator/iptables-alerter-wnktt"
Sep 15 20:58:03.850207 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.849846    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/932cce37-6a6a-4968-8cbe-d163e1da16f0-dbus\") pod \"global-pull-secret-syncer-qphq9\" (UID: \"932cce37-6a6a-4968-8cbe-d163e1da16f0\") " pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:03.851624 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.851579    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 20:58:03.852450 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.852426    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:03.855620 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.855597    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:03.856416 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.856391    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 20:58:03.856538 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.856416    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8cgfm\""
Sep 15 20:58:03.856538 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.856432    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 20:58:03.856538 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.856490    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 20:58:03.856666 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.856567    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 20:58:03.857149 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.857126    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 20:58:03.857393 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.857155    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 20:58:03.858116 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.858099    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk"
Sep 15 20:58:03.858807 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.858788    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4x6zs\""
Sep 15 20:58:03.858902 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.858838    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 20:58:03.858958 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.858926    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 20:58:03.860585 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.860563    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xkmbd"
Sep 15 20:58:03.861230 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.861128    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z5b9k\""
Sep 15 20:58:03.862288 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.862223    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 20:58:03.862396 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.862350    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 20:58:03.862473 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.862446    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 20:58:03.862833 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.862813    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 20:58:03.863062 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.863030    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 20:58:03.863440 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.863410    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 20:58:03.863524 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.863424    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zxzzq\""
Sep 15 20:58:03.872637 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.872612    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kq5fc"
Sep 15 20:58:03.879661 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.879638    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kq5fc"
Sep 15 20:58:03.938012 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.937982    2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 20:58:03.941749 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.941682    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-207.ec2.internal" event={"ID":"344eacc22c151cd63c505c6fcd3eb468","Type":"ContainerStarted","Data":"8f48ad1f2c758ac124029a18c3d6f5a94bb4d837909ed72c73e721e258c21dbd"}
Sep 15 20:58:03.943513 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.943478    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-207.ec2.internal" event={"ID":"1c3857c19d72e89622f4c04f57a3da21","Type":"ContainerStarted","Data":"5ce6297cd84547bd86053f75b86055d3ad3107360b568689b5da0f60d0067d45"}
Sep 15 20:58:03.950382 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.950337    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-hostroot\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:03.950543 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.950405    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71567bed-fb87-4bb0-9833-a30ff9a16ef9-serviceca\") pod \"node-ca-xkmbd\" (UID: \"71567bed-fb87-4bb0-9833-a30ff9a16ef9\") " pod="openshift-image-registry/node-ca-xkmbd"
Sep 15 20:58:03.950543 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.950439    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-etc-sysconfig\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:03.950543 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.950466    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-etc-kubernetes\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:03.950543 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.950508    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49a4e9ea-347b-4653-9210-e9b142172bdd-host-slash\") pod \"iptables-alerter-wnktt\" (UID: \"49a4e9ea-347b-4653-9210-e9b142172bdd\") " pod="openshift-network-operator/iptables-alerter-wnktt"
Sep 15 20:58:03.950738 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.950539    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/932cce37-6a6a-4968-8cbe-d163e1da16f0-dbus\") pod \"global-pull-secret-syncer-qphq9\" (UID: \"932cce37-6a6a-4968-8cbe-d163e1da16f0\") " pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:03.950738 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.950579    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-var-lib-openvswitch\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:03.950738 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.950628    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjh8j\" (UniqueName: \"kubernetes.io/projected/71567bed-fb87-4bb0-9833-a30ff9a16ef9-kube-api-access-mjh8j\") pod \"node-ca-xkmbd\" (UID: \"71567bed-fb87-4bb0-9833-a30ff9a16ef9\") " pod="openshift-image-registry/node-ca-xkmbd"
Sep 15 20:58:03.950868 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.950738    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4x8k\" (UniqueName: \"kubernetes.io/projected/bab658f3-4be7-4df7-8709-a281f243c624-kube-api-access-r4x8k\") pod \"aws-ebs-csi-driver-node-cvdxk\" (UID: \"bab658f3-4be7-4df7-8709-a281f243c624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk"
Sep 15 20:58:03.950868 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.950840    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb9c01cf-5113-402e-a9ff-8f177ab7f018-hosts-file\") pod \"node-resolver-nwspf\" (UID: \"cb9c01cf-5113-402e-a9ff-8f177ab7f018\") " pod="openshift-dns/node-resolver-nwspf"
Sep 15 20:58:03.950968 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.950876    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-host-run-netns\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:03.950968 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.950886    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/932cce37-6a6a-4968-8cbe-d163e1da16f0-dbus\") pod \"global-pull-secret-syncer-qphq9\" (UID: \"932cce37-6a6a-4968-8cbe-d163e1da16f0\") " pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:03.950968 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.950909    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-run-ovn\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:03.951120 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.950983    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49a4e9ea-347b-4653-9210-e9b142172bdd-host-slash\") pod \"iptables-alerter-wnktt\" (UID: \"49a4e9ea-347b-4653-9210-e9b142172bdd\") " pod="openshift-network-operator/iptables-alerter-wnktt"
Sep 15 20:58:03.951120 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.951044    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-host\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:03.951209 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.951117    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-host-run-multus-certs\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:03.951209 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.951173    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-etc-kubernetes\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:03.951458 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.951385    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ef60e7a1-62c7-46aa-8fd8-4043168ffa94-konnectivity-ca\") pod \"konnectivity-agent-pc79j\" (UID: \"ef60e7a1-62c7-46aa-8fd8-4043168ffa94\") " pod="kube-system/konnectivity-agent-pc79j"
Sep 15 20:58:03.951573 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.951539    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc7c2\" (UniqueName: \"kubernetes.io/projected/49a4e9ea-347b-4653-9210-e9b142172bdd-kube-api-access-zc7c2\") pod \"iptables-alerter-wnktt\" (UID: \"49a4e9ea-347b-4653-9210-e9b142172bdd\") " pod="openshift-network-operator/iptables-alerter-wnktt"
Sep 15 20:58:03.952388 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.951655    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/932cce37-6a6a-4968-8cbe-d163e1da16f0-kubelet-config\") pod \"global-pull-secret-syncer-qphq9\" (UID: \"932cce37-6a6a-4968-8cbe-d163e1da16f0\") " pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:03.952388 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.951695    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/932cce37-6a6a-4968-8cbe-d163e1da16f0-original-pull-secret\") pod \"global-pull-secret-syncer-qphq9\" (UID: \"932cce37-6a6a-4968-8cbe-d163e1da16f0\") " pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:03.952388 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.951742    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/695e9f08-df31-4f11-b54e-583d69edbb05-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hrmmd\" (UID: \"695e9f08-df31-4f11-b54e-583d69edbb05\") " pod="openshift-multus/multus-additional-cni-plugins-hrmmd"
Sep 15 20:58:03.952388 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.951813    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/86c2669e-e211-437d-81e3-b8d085d150f8-ovn-node-metrics-cert\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:03.952388 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.952007    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/695e9f08-df31-4f11-b54e-583d69edbb05-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hrmmd\" (UID: \"695e9f08-df31-4f11-b54e-583d69edbb05\") " pod="openshift-multus/multus-additional-cni-plugins-hrmmd"
Sep 15 20:58:03.952388 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.952066    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3181c7fe-2852-46cd-aba2-6007201a049a-tmp\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:03.952388 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.952099    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-system-cni-dir\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:03.952388 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.952133    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4708b99-1a06-45a8-933f-0e0e0de9bde6-cni-binary-copy\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:03.952388 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.952191    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-host-run-k8s-cni-cncf-io\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:03.952388 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.952238    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-host-var-lib-cni-multus\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:03.952388 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.952277    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/695e9f08-df31-4f11-b54e-583d69edbb05-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hrmmd\" (UID: \"695e9f08-df31-4f11-b54e-583d69edbb05\") " pod="openshift-multus/multus-additional-cni-plugins-hrmmd"
Sep 15 20:58:03.952388 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.952321    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt66k\" (UniqueName: \"kubernetes.io/projected/3181c7fe-2852-46cd-aba2-6007201a049a-kube-api-access-gt66k\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:03.952388 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.952351    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-host-var-lib-cni-bin\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:03.952999 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.952559    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/932cce37-6a6a-4968-8cbe-d163e1da16f0-kubelet-config\") pod \"global-pull-secret-syncer-qphq9\" (UID: \"932cce37-6a6a-4968-8cbe-d163e1da16f0\") " pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:03.952999 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:03.952699    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 20:58:03.952999 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:03.952942    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932cce37-6a6a-4968-8cbe-d163e1da16f0-original-pull-secret podName:932cce37-6a6a-4968-8cbe-d163e1da16f0 nodeName:}" failed. No retries permitted until 2026-09-15 20:58:04.4529066 +0000 UTC m=+2.943552328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/932cce37-6a6a-4968-8cbe-d163e1da16f0-original-pull-secret") pod "global-pull-secret-syncer-qphq9" (UID: "932cce37-6a6a-4968-8cbe-d163e1da16f0") : object "kube-system"/"original-pull-secret" not registered
Sep 15 20:58:03.952999 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.952969    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/695e9f08-df31-4f11-b54e-583d69edbb05-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hrmmd\" (UID: \"695e9f08-df31-4f11-b54e-583d69edbb05\") " pod="openshift-multus/multus-additional-cni-plugins-hrmmd"
Sep 15 20:58:03.953212 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953019    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a4708b99-1a06-45a8-933f-0e0e0de9bde6-multus-daemon-config\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:03.953212 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953067    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/695e9f08-df31-4f11-b54e-583d69edbb05-cnibin\") pod \"multus-additional-cni-plugins-hrmmd\" (UID: \"695e9f08-df31-4f11-b54e-583d69edbb05\") " pod="openshift-multus/multus-additional-cni-plugins-hrmmd"
Sep 15 20:58:03.953212 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953121    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-host-cni-netd\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:03.953212 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953152    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/86c2669e-e211-437d-81e3-b8d085d150f8-env-overrides\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:03.953212 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953183    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-etc-sysctl-conf\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:03.953419 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953226    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bab658f3-4be7-4df7-8709-a281f243c624-device-dir\") pod \"aws-ebs-csi-driver-node-cvdxk\" (UID: \"bab658f3-4be7-4df7-8709-a281f243c624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk"
Sep 15 20:58:03.953419 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953247    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/695e9f08-df31-4f11-b54e-583d69edbb05-cnibin\") pod \"multus-additional-cni-plugins-hrmmd\" (UID: \"695e9f08-df31-4f11-b54e-583d69edbb05\") " pod="openshift-multus/multus-additional-cni-plugins-hrmmd"
Sep 15 20:58:03.953419 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953276    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53a126d9-fd09-4306-9547-39fad0299679-metrics-certs\") pod \"network-metrics-daemon-vf58t\" (UID: \"53a126d9-fd09-4306-9547-39fad0299679\") " pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:03.953419 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953315    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-cnibin\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:03.953419 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953362    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86g7t\" (UniqueName: \"kubernetes.io/projected/a4708b99-1a06-45a8-933f-0e0e0de9bde6-kube-api-access-86g7t\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:03.953675 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953422    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ef60e7a1-62c7-46aa-8fd8-4043168ffa94-agent-certs\") pod \"konnectivity-agent-pc79j\" (UID: \"ef60e7a1-62c7-46aa-8fd8-4043168ffa94\") " pod="kube-system/konnectivity-agent-pc79j"
Sep 15 20:58:03.953675 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953472    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/695e9f08-df31-4f11-b54e-583d69edbb05-system-cni-dir\") pod \"multus-additional-cni-plugins-hrmmd\" (UID: \"695e9f08-df31-4f11-b54e-583d69edbb05\") " pod="openshift-multus/multus-additional-cni-plugins-hrmmd"
Sep 15 20:58:03.953675 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953502    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb9c01cf-5113-402e-a9ff-8f177ab7f018-tmp-dir\") pod \"node-resolver-nwspf\" (UID: \"cb9c01cf-5113-402e-a9ff-8f177ab7f018\") " pod="openshift-dns/node-resolver-nwspf"
Sep 15 20:58:03.953675 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953530    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-run-systemd\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:03.953675 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953597    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p58k\" (UniqueName: \"kubernetes.io/projected/53a126d9-fd09-4306-9547-39fad0299679-kube-api-access-6p58k\") pod \"network-metrics-daemon-vf58t\" (UID: \"53a126d9-fd09-4306-9547-39fad0299679\") " pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:03.953675 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953635    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-etc-modprobe-d\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:03.953675 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953664    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-etc-systemd\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:03.953974 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953705    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-host-kubelet\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:03.953974 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953732    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-host-slash\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:03.953974 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953783    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-run-openvswitch\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:03.953974 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953805    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-log-socket\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:03.953974 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953847    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/86c2669e-e211-437d-81e3-b8d085d150f8-ovnkube-config\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:03.953974 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953876    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/86c2669e-e211-437d-81e3-b8d085d150f8-ovnkube-script-lib\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:03.953974 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953930    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/49a4e9ea-347b-4653-9210-e9b142172bdd-iptables-alerter-script\") pod \"iptables-alerter-wnktt\" (UID: \"49a4e9ea-347b-4653-9210-e9b142172bdd\") " pod="openshift-network-operator/iptables-alerter-wnktt"
Sep 15 20:58:03.954307 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.953976    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbnnm\" (UniqueName: \"kubernetes.io/projected/cb9c01cf-5113-402e-a9ff-8f177ab7f018-kube-api-access-kbnnm\") pod \"node-resolver-nwspf\" (UID: \"cb9c01cf-5113-402e-a9ff-8f177ab7f018\") " pod="openshift-dns/node-resolver-nwspf"
Sep 15 20:58:03.954307 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.954001    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-etc-openvswitch\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:03.954307 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.954031    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bab658f3-4be7-4df7-8709-a281f243c624-registration-dir\") pod \"aws-ebs-csi-driver-node-cvdxk\" (UID: \"bab658f3-4be7-4df7-8709-a281f243c624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk"
Sep 15 20:58:03.954307 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.954076    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3181c7fe-2852-46cd-aba2-6007201a049a-etc-tuned\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:03.954307 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.954143    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/695e9f08-df31-4f11-b54e-583d69edbb05-system-cni-dir\") pod \"multus-additional-cni-plugins-hrmmd\" (UID: \"695e9f08-df31-4f11-b54e-583d69edbb05\") " pod="openshift-multus/multus-additional-cni-plugins-hrmmd"
Sep 15 20:58:03.954307 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.954206    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-host-var-lib-kubelet\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:03.954307 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.954259    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-multus-conf-dir\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:03.954307 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.954268    2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 20:58:03.954307 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.954295    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71567bed-fb87-4bb0-9833-a30ff9a16ef9-host\") pod \"node-ca-xkmbd\" (UID: \"71567bed-fb87-4bb0-9833-a30ff9a16ef9\") " pod="openshift-image-registry/node-ca-xkmbd"
Sep 15 20:58:03.954703 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.954334    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/695e9f08-df31-4f11-b54e-583d69edbb05-cni-binary-copy\") pod \"multus-additional-cni-plugins-hrmmd\" (UID: \"695e9f08-df31-4f11-b54e-583d69edbb05\") " pod="openshift-multus/multus-additional-cni-plugins-hrmmd"
Sep 15 20:58:03.954703 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.954418    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-systemd-units\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:03.954703 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.954672    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:03.954703 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.954685    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/49a4e9ea-347b-4653-9210-e9b142172bdd-iptables-alerter-script\") pod \"iptables-alerter-wnktt\" (UID: \"49a4e9ea-347b-4653-9210-e9b142172bdd\") " pod="openshift-network-operator/iptables-alerter-wnktt"
Sep 15 20:58:03.954900 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.954716    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77nxh\" (UniqueName: \"kubernetes.io/projected/86c2669e-e211-437d-81e3-b8d085d150f8-kube-api-access-77nxh\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:03.954900 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.954743    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-etc-sysctl-d\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:03.954900 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.954774    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-run\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:03.954900 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.954812    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb9wx\" (UniqueName: \"kubernetes.io/projected/695e9f08-df31-4f11-b54e-583d69edbb05-kube-api-access-tb9wx\") pod \"multus-additional-cni-plugins-hrmmd\" (UID: \"695e9f08-df31-4f11-b54e-583d69edbb05\") " pod="openshift-multus/multus-additional-cni-plugins-hrmmd"
Sep 15 20:58:03.954900 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.954843    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-node-log\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:03.954900 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.954868    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-var-lib-kubelet\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:03.955189 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.954903    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/695e9f08-df31-4f11-b54e-583d69edbb05-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hrmmd\" (UID: \"695e9f08-df31-4f11-b54e-583d69edbb05\") " pod="openshift-multus/multus-additional-cni-plugins-hrmmd"
Sep 15 20:58:03.955189 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.954939    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bab658f3-4be7-4df7-8709-a281f243c624-etc-selinux\") pod \"aws-ebs-csi-driver-node-cvdxk\" (UID: \"bab658f3-4be7-4df7-8709-a281f243c624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk"
Sep 15 20:58:03.955189 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.954973    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-host-run-netns\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:03.955189 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.954902    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/695e9f08-df31-4f11-b54e-583d69edbb05-cni-binary-copy\") pod \"multus-additional-cni-plugins-hrmmd\" (UID: \"695e9f08-df31-4f11-b54e-583d69edbb05\") " pod="openshift-multus/multus-additional-cni-plugins-hrmmd"
Sep 15 20:58:03.955189 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.955012    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bab658f3-4be7-4df7-8709-a281f243c624-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cvdxk\" (UID: \"bab658f3-4be7-4df7-8709-a281f243c624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk"
Sep 15 20:58:03.955189 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.955087    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bab658f3-4be7-4df7-8709-a281f243c624-socket-dir\") pod \"aws-ebs-csi-driver-node-cvdxk\" (UID: \"bab658f3-4be7-4df7-8709-a281f243c624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk"
Sep 15 20:58:03.955189 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.955178    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-multus-cni-dir\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:03.955517 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.955213    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-host-run-ovn-kubernetes\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:03.955517 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.955266    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bab658f3-4be7-4df7-8709-a281f243c624-sys-fs\") pod \"aws-ebs-csi-driver-node-cvdxk\" (UID: \"bab658f3-4be7-4df7-8709-a281f243c624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk"
Sep 15 20:58:03.955517 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.955308    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-sys\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:03.955517 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.955341    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-lib-modules\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:03.955517 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.955373    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-host-cni-bin\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:03.955517 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.955497    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvj4n\" (UniqueName: \"kubernetes.io/projected/e86303f1-1359-459e-910a-31c3a4256728-kube-api-access-fvj4n\") pod \"network-check-target-z2vkz\" (UID: \"e86303f1-1359-459e-910a-31c3a4256728\") " pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:03.955773 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.955532    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-multus-socket-dir-parent\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:03.955773 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.955564    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/695e9f08-df31-4f11-b54e-583d69edbb05-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hrmmd\" (UID: \"695e9f08-df31-4f11-b54e-583d69edbb05\") " pod="openshift-multus/multus-additional-cni-plugins-hrmmd"
Sep 15 20:58:03.956673 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.955984    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ef60e7a1-62c7-46aa-8fd8-4043168ffa94-konnectivity-ca\") pod \"konnectivity-agent-pc79j\" (UID: \"ef60e7a1-62c7-46aa-8fd8-4043168ffa94\") " pod="kube-system/konnectivity-agent-pc79j"
Sep 15 20:58:03.958989 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.958955    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ef60e7a1-62c7-46aa-8fd8-4043168ffa94-agent-certs\") pod \"konnectivity-agent-pc79j\" (UID: \"ef60e7a1-62c7-46aa-8fd8-4043168ffa94\") " pod="kube-system/konnectivity-agent-pc79j"
Sep 15 20:58:03.963852 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.963781    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc7c2\" (UniqueName: \"kubernetes.io/projected/49a4e9ea-347b-4653-9210-e9b142172bdd-kube-api-access-zc7c2\") pod \"iptables-alerter-wnktt\" (UID: \"49a4e9ea-347b-4653-9210-e9b142172bdd\") " pod="openshift-network-operator/iptables-alerter-wnktt"
Sep 15 20:58:03.964599 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:03.964574    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb9wx\" (UniqueName: \"kubernetes.io/projected/695e9f08-df31-4f11-b54e-583d69edbb05-kube-api-access-tb9wx\") pod \"multus-additional-cni-plugins-hrmmd\" (UID: \"695e9f08-df31-4f11-b54e-583d69edbb05\") " pod="openshift-multus/multus-additional-cni-plugins-hrmmd"
Sep 15 20:58:04.056454 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.056391    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/86c2669e-e211-437d-81e3-b8d085d150f8-env-overrides\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.056454 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.056454    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-etc-sysctl-conf\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.056678 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.056477    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bab658f3-4be7-4df7-8709-a281f243c624-device-dir\") pod \"aws-ebs-csi-driver-node-cvdxk\" (UID: \"bab658f3-4be7-4df7-8709-a281f243c624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk"
Sep 15 20:58:04.056678 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.056500    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53a126d9-fd09-4306-9547-39fad0299679-metrics-certs\") pod \"network-metrics-daemon-vf58t\" (UID: \"53a126d9-fd09-4306-9547-39fad0299679\") " pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:04.056678 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.056527    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-cnibin\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.056678 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.056554    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86g7t\" (UniqueName: \"kubernetes.io/projected/a4708b99-1a06-45a8-933f-0e0e0de9bde6-kube-api-access-86g7t\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.056678 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.056579    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb9c01cf-5113-402e-a9ff-8f177ab7f018-tmp-dir\") pod \"node-resolver-nwspf\" (UID: \"cb9c01cf-5113-402e-a9ff-8f177ab7f018\") " pod="openshift-dns/node-resolver-nwspf"
Sep 15 20:58:04.056678 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.056617    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-etc-sysctl-conf\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.056678 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.056601    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-run-systemd\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.056678 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:04.056671    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:58:04.056979 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:04.056760    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53a126d9-fd09-4306-9547-39fad0299679-metrics-certs podName:53a126d9-fd09-4306-9547-39fad0299679 nodeName:}" failed. No retries permitted until 2026-09-15 20:58:04.556736902 +0000 UTC m=+3.047382632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53a126d9-fd09-4306-9547-39fad0299679-metrics-certs") pod "network-metrics-daemon-vf58t" (UID: "53a126d9-fd09-4306-9547-39fad0299679") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:58:04.056979 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.056763    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-cnibin\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.056979 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.056669    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p58k\" (UniqueName: \"kubernetes.io/projected/53a126d9-fd09-4306-9547-39fad0299679-kube-api-access-6p58k\") pod \"network-metrics-daemon-vf58t\" (UID: \"53a126d9-fd09-4306-9547-39fad0299679\") " pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:04.056979 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.056809    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-etc-modprobe-d\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.056979 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.056835    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-etc-systemd\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.056979 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.056860    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-host-kubelet\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.056979 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.056884    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-host-slash\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.056979 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.056908    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-run-openvswitch\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.056979 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.056932    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-log-socket\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.056979 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.056946    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/86c2669e-e211-437d-81e3-b8d085d150f8-env-overrides\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.056979 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.056954    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/86c2669e-e211-437d-81e3-b8d085d150f8-ovnkube-config\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.057379 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.056980    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/86c2669e-e211-437d-81e3-b8d085d150f8-ovnkube-script-lib\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.057379 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057007    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbnnm\" (UniqueName: \"kubernetes.io/projected/cb9c01cf-5113-402e-a9ff-8f177ab7f018-kube-api-access-kbnnm\") pod \"node-resolver-nwspf\" (UID: \"cb9c01cf-5113-402e-a9ff-8f177ab7f018\") " pod="openshift-dns/node-resolver-nwspf"
Sep 15 20:58:04.057379 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057021    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-run-systemd\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.057379 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057031    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-etc-openvswitch\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.057379 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057091    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bab658f3-4be7-4df7-8709-a281f243c624-registration-dir\") pod \"aws-ebs-csi-driver-node-cvdxk\" (UID: \"bab658f3-4be7-4df7-8709-a281f243c624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk"
Sep 15 20:58:04.057379 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057155    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3181c7fe-2852-46cd-aba2-6007201a049a-etc-tuned\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.057379 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057172    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-run-openvswitch\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.057379 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057188    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-host-var-lib-kubelet\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.057379 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057212    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-multus-conf-dir\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.057379 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057223    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bab658f3-4be7-4df7-8709-a281f243c624-device-dir\") pod \"aws-ebs-csi-driver-node-cvdxk\" (UID: \"bab658f3-4be7-4df7-8709-a281f243c624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk"
Sep 15 20:58:04.057379 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057236    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71567bed-fb87-4bb0-9833-a30ff9a16ef9-host\") pod \"node-ca-xkmbd\" (UID: \"71567bed-fb87-4bb0-9833-a30ff9a16ef9\") " pod="openshift-image-registry/node-ca-xkmbd"
Sep 15 20:58:04.057379 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057261    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-systemd-units\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.057379 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057310    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb9c01cf-5113-402e-a9ff-8f177ab7f018-tmp-dir\") pod \"node-resolver-nwspf\" (UID: \"cb9c01cf-5113-402e-a9ff-8f177ab7f018\") " pod="openshift-dns/node-resolver-nwspf"
Sep 15 20:58:04.057379 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057306    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-etc-modprobe-d\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.057379 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057304    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.057379 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057350    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77nxh\" (UniqueName: \"kubernetes.io/projected/86c2669e-e211-437d-81e3-b8d085d150f8-kube-api-access-77nxh\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.057379 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057356    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-etc-systemd\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.058172 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057383    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-etc-sysctl-d\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.058172 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057381    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-host-kubelet\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.058172 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057418    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-host-slash\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.058172 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057428    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-log-socket\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.058172 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057448    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bab658f3-4be7-4df7-8709-a281f243c624-registration-dir\") pod \"aws-ebs-csi-driver-node-cvdxk\" (UID: \"bab658f3-4be7-4df7-8709-a281f243c624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk"
Sep 15 20:58:04.058172 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057479    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-etc-sysctl-d\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.058172 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057498    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-run\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.058172 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057514    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-node-log\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.058172 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057549    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-var-lib-kubelet\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.058172 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057565    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bab658f3-4be7-4df7-8709-a281f243c624-etc-selinux\") pod \"aws-ebs-csi-driver-node-cvdxk\" (UID: \"bab658f3-4be7-4df7-8709-a281f243c624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk"
Sep 15 20:58:04.058172 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057585    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-host-run-netns\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.058172 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057622    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-etc-openvswitch\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.058172 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057625    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bab658f3-4be7-4df7-8709-a281f243c624-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cvdxk\" (UID: \"bab658f3-4be7-4df7-8709-a281f243c624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk"
Sep 15 20:58:04.058172 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057650    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-host-run-netns\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.058172 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057656    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bab658f3-4be7-4df7-8709-a281f243c624-socket-dir\") pod \"aws-ebs-csi-driver-node-cvdxk\" (UID: \"bab658f3-4be7-4df7-8709-a281f243c624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk"
Sep 15 20:58:04.058172 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057681    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-multus-cni-dir\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.058172 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057696    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71567bed-fb87-4bb0-9833-a30ff9a16ef9-host\") pod \"node-ca-xkmbd\" (UID: \"71567bed-fb87-4bb0-9833-a30ff9a16ef9\") " pod="openshift-image-registry/node-ca-xkmbd"
Sep 15 20:58:04.058172 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057707    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-host-run-ovn-kubernetes\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.058773 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057730    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-var-lib-kubelet\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.058773 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057744    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-host-run-ovn-kubernetes\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.058773 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057750    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bab658f3-4be7-4df7-8709-a281f243c624-etc-selinux\") pod \"aws-ebs-csi-driver-node-cvdxk\" (UID: \"bab658f3-4be7-4df7-8709-a281f243c624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk"
Sep 15 20:58:04.058773 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057744    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bab658f3-4be7-4df7-8709-a281f243c624-sys-fs\") pod \"aws-ebs-csi-driver-node-cvdxk\" (UID: \"bab658f3-4be7-4df7-8709-a281f243c624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk"
Sep 15 20:58:04.058773 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057684    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-node-log\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.058773 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057788    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-sys\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.058773 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057792    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-systemd-units\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.058773 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057802    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-multus-conf-dir\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.058773 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057824    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bab658f3-4be7-4df7-8709-a281f243c624-sys-fs\") pod \"aws-ebs-csi-driver-node-cvdxk\" (UID: \"bab658f3-4be7-4df7-8709-a281f243c624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk"
Sep 15 20:58:04.058773 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057841    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bab658f3-4be7-4df7-8709-a281f243c624-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cvdxk\" (UID: \"bab658f3-4be7-4df7-8709-a281f243c624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk"
Sep 15 20:58:04.058773 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057842    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-host-var-lib-kubelet\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.058773 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057889    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-sys\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.058773 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057890    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/86c2669e-e211-437d-81e3-b8d085d150f8-ovnkube-config\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.058773 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057917    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-multus-cni-dir\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.058773 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057920    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-lib-modules\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.058773 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057911    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-run\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.058773 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057950    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-host-cni-bin\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.058773 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057954    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.059703 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057975    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bab658f3-4be7-4df7-8709-a281f243c624-socket-dir\") pod \"aws-ebs-csi-driver-node-cvdxk\" (UID: \"bab658f3-4be7-4df7-8709-a281f243c624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk"
Sep 15 20:58:04.059703 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057988    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-lib-modules\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.059703 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.057995    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-host-cni-bin\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.059703 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058001    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvj4n\" (UniqueName: \"kubernetes.io/projected/e86303f1-1359-459e-910a-31c3a4256728-kube-api-access-fvj4n\") pod \"network-check-target-z2vkz\" (UID: \"e86303f1-1359-459e-910a-31c3a4256728\") " pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:04.059703 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058029    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-multus-socket-dir-parent\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.059703 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058101    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-hostroot\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.059703 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058113    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-multus-socket-dir-parent\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.059703 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058126    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71567bed-fb87-4bb0-9833-a30ff9a16ef9-serviceca\") pod \"node-ca-xkmbd\" (UID: \"71567bed-fb87-4bb0-9833-a30ff9a16ef9\") " pod="openshift-image-registry/node-ca-xkmbd"
Sep 15 20:58:04.059703 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058151    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-etc-sysconfig\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.059703 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058145    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-hostroot\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.059703 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058175    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-etc-kubernetes\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.059703 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058218    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-var-lib-openvswitch\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.059703 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058253    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-etc-kubernetes\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.059703 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058254    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjh8j\" (UniqueName: \"kubernetes.io/projected/71567bed-fb87-4bb0-9833-a30ff9a16ef9-kube-api-access-mjh8j\") pod \"node-ca-xkmbd\" (UID: \"71567bed-fb87-4bb0-9833-a30ff9a16ef9\") " pod="openshift-image-registry/node-ca-xkmbd"
Sep 15 20:58:04.059703 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058296    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4x8k\" (UniqueName: \"kubernetes.io/projected/bab658f3-4be7-4df7-8709-a281f243c624-kube-api-access-r4x8k\") pod \"aws-ebs-csi-driver-node-cvdxk\" (UID: \"bab658f3-4be7-4df7-8709-a281f243c624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk"
Sep 15 20:58:04.059703 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058323    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb9c01cf-5113-402e-a9ff-8f177ab7f018-hosts-file\") pod \"node-resolver-nwspf\" (UID: \"cb9c01cf-5113-402e-a9ff-8f177ab7f018\") " pod="openshift-dns/node-resolver-nwspf"
Sep 15 20:58:04.059703 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058334    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-etc-sysconfig\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.059703 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058364    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-host-run-netns\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.060559 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058391    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-run-ovn\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.060559 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058413    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-host\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.060559 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058296    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-var-lib-openvswitch\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.060559 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058473    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-host-run-netns\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.060559 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058475    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-host-run-multus-certs\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.060559 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058505    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-etc-kubernetes\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.060559 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058522    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb9c01cf-5113-402e-a9ff-8f177ab7f018-hosts-file\") pod \"node-resolver-nwspf\" (UID: \"cb9c01cf-5113-402e-a9ff-8f177ab7f018\") " pod="openshift-dns/node-resolver-nwspf"
Sep 15 20:58:04.060559 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058507    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-host-run-multus-certs\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.060559 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058477    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/86c2669e-e211-437d-81e3-b8d085d150f8-ovnkube-script-lib\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.060559 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058553    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3181c7fe-2852-46cd-aba2-6007201a049a-host\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.060559 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058550    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-run-ovn\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.060559 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058564    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-etc-kubernetes\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.060559 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058633    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71567bed-fb87-4bb0-9833-a30ff9a16ef9-serviceca\") pod \"node-ca-xkmbd\" (UID: \"71567bed-fb87-4bb0-9833-a30ff9a16ef9\") " pod="openshift-image-registry/node-ca-xkmbd"
Sep 15 20:58:04.060559 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058654    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/86c2669e-e211-437d-81e3-b8d085d150f8-ovn-node-metrics-cert\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.060559 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058687    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3181c7fe-2852-46cd-aba2-6007201a049a-tmp\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.060559 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058710    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-system-cni-dir\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.060559 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058733    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4708b99-1a06-45a8-933f-0e0e0de9bde6-cni-binary-copy\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.060559 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058755    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-host-run-k8s-cni-cncf-io\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.061410 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058779    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-host-var-lib-cni-multus\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.061410 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058784    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-system-cni-dir\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.061410 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058827    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-host-var-lib-cni-multus\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.061410 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058839    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-host-run-k8s-cni-cncf-io\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.061410 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058869    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt66k\" (UniqueName: \"kubernetes.io/projected/3181c7fe-2852-46cd-aba2-6007201a049a-kube-api-access-gt66k\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.061410 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058901    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-host-var-lib-cni-bin\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.061410 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058927    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a4708b99-1a06-45a8-933f-0e0e0de9bde6-multus-daemon-config\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.061410 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.058957    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-host-cni-netd\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.061410 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.059034    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/86c2669e-e211-437d-81e3-b8d085d150f8-host-cni-netd\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.061410 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.059066    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4708b99-1a06-45a8-933f-0e0e0de9bde6-host-var-lib-cni-bin\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.061410 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.059320    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4708b99-1a06-45a8-933f-0e0e0de9bde6-cni-binary-copy\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.061410 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.059525    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a4708b99-1a06-45a8-933f-0e0e0de9bde6-multus-daemon-config\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.061410 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.060097    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3181c7fe-2852-46cd-aba2-6007201a049a-etc-tuned\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.061410 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.061035    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3181c7fe-2852-46cd-aba2-6007201a049a-tmp\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.062091 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.061559    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/86c2669e-e211-437d-81e3-b8d085d150f8-ovn-node-metrics-cert\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.064847 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:04.064820    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 20:58:04.064847 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:04.064850    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 20:58:04.065044 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:04.064865    2558 projected.go:196] Error preparing data for projected volume kube-api-access-fvj4n for pod openshift-network-diagnostics/network-check-target-z2vkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:58:04.065044 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:04.064940    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e86303f1-1359-459e-910a-31c3a4256728-kube-api-access-fvj4n podName:e86303f1-1359-459e-910a-31c3a4256728 nodeName:}" failed. No retries permitted until 2026-09-15 20:58:04.564920484 +0000 UTC m=+3.055566198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fvj4n" (UniqueName: "kubernetes.io/projected/e86303f1-1359-459e-910a-31c3a4256728-kube-api-access-fvj4n") pod "network-check-target-z2vkz" (UID: "e86303f1-1359-459e-910a-31c3a4256728") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:58:04.066231 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.066202    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p58k\" (UniqueName: \"kubernetes.io/projected/53a126d9-fd09-4306-9547-39fad0299679-kube-api-access-6p58k\") pod \"network-metrics-daemon-vf58t\" (UID: \"53a126d9-fd09-4306-9547-39fad0299679\") " pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:04.066343 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.066247    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbnnm\" (UniqueName: \"kubernetes.io/projected/cb9c01cf-5113-402e-a9ff-8f177ab7f018-kube-api-access-kbnnm\") pod \"node-resolver-nwspf\" (UID: \"cb9c01cf-5113-402e-a9ff-8f177ab7f018\") " pod="openshift-dns/node-resolver-nwspf"
Sep 15 20:58:04.066343 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.066245    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86g7t\" (UniqueName: \"kubernetes.io/projected/a4708b99-1a06-45a8-933f-0e0e0de9bde6-kube-api-access-86g7t\") pod \"multus-vjzlh\" (UID: \"a4708b99-1a06-45a8-933f-0e0e0de9bde6\") " pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.067626 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.067590    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77nxh\" (UniqueName: \"kubernetes.io/projected/86c2669e-e211-437d-81e3-b8d085d150f8-kube-api-access-77nxh\") pod \"ovnkube-node-6fp7g\" (UID: \"86c2669e-e211-437d-81e3-b8d085d150f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.067856 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.067805    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt66k\" (UniqueName: \"kubernetes.io/projected/3181c7fe-2852-46cd-aba2-6007201a049a-kube-api-access-gt66k\") pod \"tuned-fl48z\" (UID: \"3181c7fe-2852-46cd-aba2-6007201a049a\") " pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.068350 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.068300    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4x8k\" (UniqueName: \"kubernetes.io/projected/bab658f3-4be7-4df7-8709-a281f243c624-kube-api-access-r4x8k\") pod \"aws-ebs-csi-driver-node-cvdxk\" (UID: \"bab658f3-4be7-4df7-8709-a281f243c624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk"
Sep 15 20:58:04.068350 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.068316    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjh8j\" (UniqueName: \"kubernetes.io/projected/71567bed-fb87-4bb0-9833-a30ff9a16ef9-kube-api-access-mjh8j\") pod \"node-ca-xkmbd\" (UID: \"71567bed-fb87-4bb0-9833-a30ff9a16ef9\") " pod="openshift-image-registry/node-ca-xkmbd"
Sep 15 20:58:04.144691 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.144654    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hrmmd"
Sep 15 20:58:04.157203 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.157167    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wnktt"
Sep 15 20:58:04.168655 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.168629    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pc79j"
Sep 15 20:58:04.174541 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.174515    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fl48z"
Sep 15 20:58:04.183215 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.183188    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nwspf"
Sep 15 20:58:04.190941 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.190918    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:04.196647 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.196621    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vjzlh"
Sep 15 20:58:04.202391 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.202363    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk"
Sep 15 20:58:04.208114 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.208091    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xkmbd"
Sep 15 20:58:04.462072 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.461965    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/932cce37-6a6a-4968-8cbe-d163e1da16f0-original-pull-secret\") pod \"global-pull-secret-syncer-qphq9\" (UID: \"932cce37-6a6a-4968-8cbe-d163e1da16f0\") " pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:04.462246 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:04.462149    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 20:58:04.462246 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:04.462224    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932cce37-6a6a-4968-8cbe-d163e1da16f0-original-pull-secret podName:932cce37-6a6a-4968-8cbe-d163e1da16f0 nodeName:}" failed. No retries permitted until 2026-09-15 20:58:05.462207968 +0000 UTC m=+3.952853678 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/932cce37-6a6a-4968-8cbe-d163e1da16f0-original-pull-secret") pod "global-pull-secret-syncer-qphq9" (UID: "932cce37-6a6a-4968-8cbe-d163e1da16f0") : object "kube-system"/"original-pull-secret" not registered
Sep 15 20:58:04.563073 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.563032    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53a126d9-fd09-4306-9547-39fad0299679-metrics-certs\") pod \"network-metrics-daemon-vf58t\" (UID: \"53a126d9-fd09-4306-9547-39fad0299679\") " pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:04.563272 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:04.563208    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:58:04.563332 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:04.563287    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53a126d9-fd09-4306-9547-39fad0299679-metrics-certs podName:53a126d9-fd09-4306-9547-39fad0299679 nodeName:}" failed. No retries permitted until 2026-09-15 20:58:05.563264945 +0000 UTC m=+4.053910672 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53a126d9-fd09-4306-9547-39fad0299679-metrics-certs") pod "network-metrics-daemon-vf58t" (UID: "53a126d9-fd09-4306-9547-39fad0299679") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:58:04.663493 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.663442    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvj4n\" (UniqueName: \"kubernetes.io/projected/e86303f1-1359-459e-910a-31c3a4256728-kube-api-access-fvj4n\") pod \"network-check-target-z2vkz\" (UID: \"e86303f1-1359-459e-910a-31c3a4256728\") " pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:04.663708 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:04.663667    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 20:58:04.663708 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:04.663701    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 20:58:04.663818 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:04.663719    2558 projected.go:196] Error preparing data for projected volume kube-api-access-fvj4n for pod openshift-network-diagnostics/network-check-target-z2vkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:58:04.663818 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:04.663797    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e86303f1-1359-459e-910a-31c3a4256728-kube-api-access-fvj4n podName:e86303f1-1359-459e-910a-31c3a4256728 nodeName:}" failed. No retries permitted until 2026-09-15 20:58:05.663775736 +0000 UTC m=+4.154421463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fvj4n" (UniqueName: "kubernetes.io/projected/e86303f1-1359-459e-910a-31c3a4256728-kube-api-access-fvj4n") pod "network-check-target-z2vkz" (UID: "e86303f1-1359-459e-910a-31c3a4256728") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:58:04.882774 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.882676    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 20:53:04 +0000 UTC" deadline="2028-03-27 08:44:43.625511234 +0000 UTC"
Sep 15 20:58:04.882774 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:04.882714    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13403h46m38.742801564s"
Sep 15 20:58:05.031958 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:05.031930    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef60e7a1_62c7_46aa_8fd8_4043168ffa94.slice/crio-837b3814e76c46f40146bbe98cae7cc7c4eaa8f5a32ff422299c83f300a5e918 WatchSource:0}: Error finding container 837b3814e76c46f40146bbe98cae7cc7c4eaa8f5a32ff422299c83f300a5e918: Status 404 returned error can't find the container with id 837b3814e76c46f40146bbe98cae7cc7c4eaa8f5a32ff422299c83f300a5e918
Sep 15 20:58:05.032933 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:05.032906    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695e9f08_df31_4f11_b54e_583d69edbb05.slice/crio-43bcc2132274641b205f63e4de930a2b70662142d2b2baa862c0fb1755512ae0 WatchSource:0}: Error finding container 43bcc2132274641b205f63e4de930a2b70662142d2b2baa862c0fb1755512ae0: Status 404 returned error can't find the container with id 43bcc2132274641b205f63e4de930a2b70662142d2b2baa862c0fb1755512ae0
Sep 15 20:58:05.034366 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:05.034341    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a4e9ea_347b_4653_9210_e9b142172bdd.slice/crio-35dfed4a8a448fa7b5a771df528772569e56e78468885301243d89ebb677f126 WatchSource:0}: Error finding container 35dfed4a8a448fa7b5a771df528772569e56e78468885301243d89ebb677f126: Status 404 returned error can't find the container with id 35dfed4a8a448fa7b5a771df528772569e56e78468885301243d89ebb677f126
Sep 15 20:58:05.036752 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:05.036716    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3181c7fe_2852_46cd_aba2_6007201a049a.slice/crio-d1c6d9ade3d867df3b2213d45afe214c3d3a62ca4cb6dfda7531fc8024db1d8a WatchSource:0}: Error finding container d1c6d9ade3d867df3b2213d45afe214c3d3a62ca4cb6dfda7531fc8024db1d8a: Status 404 returned error can't find the container with id d1c6d9ade3d867df3b2213d45afe214c3d3a62ca4cb6dfda7531fc8024db1d8a
Sep 15 20:58:05.038507 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:05.038439    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c2669e_e211_437d_81e3_b8d085d150f8.slice/crio-10f5f9018ef4457cc1228cb98e223bb96e706a5e037074aa8b7564b3b6f275f0 WatchSource:0}: Error finding container 10f5f9018ef4457cc1228cb98e223bb96e706a5e037074aa8b7564b3b6f275f0: Status 404 returned error can't find the container with id 10f5f9018ef4457cc1228cb98e223bb96e706a5e037074aa8b7564b3b6f275f0
Sep 15 20:58:05.043085 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:05.043005    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4708b99_1a06_45a8_933f_0e0e0de9bde6.slice/crio-211a59f071a62f55e0303a3963660797fa14152bed869a221e32b1dee5c33748 WatchSource:0}: Error finding container 211a59f071a62f55e0303a3963660797fa14152bed869a221e32b1dee5c33748: Status 404 returned error can't find the container with id 211a59f071a62f55e0303a3963660797fa14152bed869a221e32b1dee5c33748
Sep 15 20:58:05.043972 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:05.043946    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab658f3_4be7_4df7_8709_a281f243c624.slice/crio-0c8b92029483e68a6203d8feac0d0a708ada08d2203244fd2840275e37915646 WatchSource:0}: Error finding container 0c8b92029483e68a6203d8feac0d0a708ada08d2203244fd2840275e37915646: Status 404 returned error can't find the container with id 0c8b92029483e68a6203d8feac0d0a708ada08d2203244fd2840275e37915646
Sep 15 20:58:05.045478 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:05.045449    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9c01cf_5113_402e_a9ff_8f177ab7f018.slice/crio-abac8ec79b294da17e70cbbaf44fd71bf437a1b7124f3666aced25c3374e3141 WatchSource:0}: Error finding container abac8ec79b294da17e70cbbaf44fd71bf437a1b7124f3666aced25c3374e3141: Status 404 returned error can't find the container with id abac8ec79b294da17e70cbbaf44fd71bf437a1b7124f3666aced25c3374e3141
Sep 15 20:58:05.046712 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:05.046216    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71567bed_fb87_4bb0_9833_a30ff9a16ef9.slice/crio-5f2f27f8c777614edba9d001c194f3a26b1a5c90714ca0308853c1933487d0bf WatchSource:0}: Error finding container 5f2f27f8c777614edba9d001c194f3a26b1a5c90714ca0308853c1933487d0bf: Status 404 returned error can't find the container with id 5f2f27f8c777614edba9d001c194f3a26b1a5c90714ca0308853c1933487d0bf
Sep 15 20:58:05.470851 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:05.470747    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/932cce37-6a6a-4968-8cbe-d163e1da16f0-original-pull-secret\") pod \"global-pull-secret-syncer-qphq9\" (UID: \"932cce37-6a6a-4968-8cbe-d163e1da16f0\") " pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:05.470998 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:05.470918    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 20:58:05.470998 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:05.470993    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932cce37-6a6a-4968-8cbe-d163e1da16f0-original-pull-secret podName:932cce37-6a6a-4968-8cbe-d163e1da16f0 nodeName:}" failed. No retries permitted until 2026-09-15 20:58:07.47097516 +0000 UTC m=+5.961620877 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/932cce37-6a6a-4968-8cbe-d163e1da16f0-original-pull-secret") pod "global-pull-secret-syncer-qphq9" (UID: "932cce37-6a6a-4968-8cbe-d163e1da16f0") : object "kube-system"/"original-pull-secret" not registered
Sep 15 20:58:05.572007 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:05.571960    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53a126d9-fd09-4306-9547-39fad0299679-metrics-certs\") pod \"network-metrics-daemon-vf58t\" (UID: \"53a126d9-fd09-4306-9547-39fad0299679\") " pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:05.572203 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:05.572161    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:58:05.572285 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:05.572254    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53a126d9-fd09-4306-9547-39fad0299679-metrics-certs podName:53a126d9-fd09-4306-9547-39fad0299679 nodeName:}" failed. No retries permitted until 2026-09-15 20:58:07.572230827 +0000 UTC m=+6.062876543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53a126d9-fd09-4306-9547-39fad0299679-metrics-certs") pod "network-metrics-daemon-vf58t" (UID: "53a126d9-fd09-4306-9547-39fad0299679") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:58:05.672397 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:05.672350    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvj4n\" (UniqueName: \"kubernetes.io/projected/e86303f1-1359-459e-910a-31c3a4256728-kube-api-access-fvj4n\") pod \"network-check-target-z2vkz\" (UID: \"e86303f1-1359-459e-910a-31c3a4256728\") " pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:05.672591 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:05.672581    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 20:58:05.672653 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:05.672600    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 20:58:05.672653 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:05.672615    2558 projected.go:196] Error preparing data for projected volume kube-api-access-fvj4n for pod openshift-network-diagnostics/network-check-target-z2vkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:58:05.672753 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:05.672677    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e86303f1-1359-459e-910a-31c3a4256728-kube-api-access-fvj4n podName:e86303f1-1359-459e-910a-31c3a4256728 nodeName:}" failed. No retries permitted until 2026-09-15 20:58:07.672657961 +0000 UTC m=+6.163303676 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fvj4n" (UniqueName: "kubernetes.io/projected/e86303f1-1359-459e-910a-31c3a4256728-kube-api-access-fvj4n") pod "network-check-target-z2vkz" (UID: "e86303f1-1359-459e-910a-31c3a4256728") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:58:05.883614 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:05.883482    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 20:53:04 +0000 UTC" deadline="2028-03-04 14:40:50.209190257 +0000 UTC"
Sep 15 20:58:05.883614 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:05.883556    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12857h42m44.325638923s"
Sep 15 20:58:05.938076 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:05.937182    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:05.938076 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:05.937326    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:05.938076 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:05.937757    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:05.938076 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:05.937850    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:05.938076 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:05.937926    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:05.938076 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:05.938014    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:05.976654 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:05.976607    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk" event={"ID":"bab658f3-4be7-4df7-8709-a281f243c624","Type":"ContainerStarted","Data":"0c8b92029483e68a6203d8feac0d0a708ada08d2203244fd2840275e37915646"}
Sep 15 20:58:05.980198 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:05.980098    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vjzlh" event={"ID":"a4708b99-1a06-45a8-933f-0e0e0de9bde6","Type":"ContainerStarted","Data":"211a59f071a62f55e0303a3963660797fa14152bed869a221e32b1dee5c33748"}
Sep 15 20:58:05.984964 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:05.984883    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fl48z" event={"ID":"3181c7fe-2852-46cd-aba2-6007201a049a","Type":"ContainerStarted","Data":"d1c6d9ade3d867df3b2213d45afe214c3d3a62ca4cb6dfda7531fc8024db1d8a"}
Sep 15 20:58:06.001735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:06.001386    2558 generic.go:356] "Generic (PLEG): container finished" podID="344eacc22c151cd63c505c6fcd3eb468" containerID="473b3af2ab1f1fe622155118b19197eb91620f3d8058f42a23eaec1cb6e3b086" exitCode=0
Sep 15 20:58:06.001924 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:06.001744    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-207.ec2.internal" event={"ID":"344eacc22c151cd63c505c6fcd3eb468","Type":"ContainerDied","Data":"473b3af2ab1f1fe622155118b19197eb91620f3d8058f42a23eaec1cb6e3b086"}
Sep 15 20:58:06.004815 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:06.004763    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nwspf" event={"ID":"cb9c01cf-5113-402e-a9ff-8f177ab7f018","Type":"ContainerStarted","Data":"abac8ec79b294da17e70cbbaf44fd71bf437a1b7124f3666aced25c3374e3141"}
Sep 15 20:58:06.013830 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:06.013753    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrmmd" event={"ID":"695e9f08-df31-4f11-b54e-583d69edbb05","Type":"ContainerStarted","Data":"43bcc2132274641b205f63e4de930a2b70662142d2b2baa862c0fb1755512ae0"}
Sep 15 20:58:06.016621 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:06.016587    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g" event={"ID":"86c2669e-e211-437d-81e3-b8d085d150f8","Type":"ContainerStarted","Data":"10f5f9018ef4457cc1228cb98e223bb96e706a5e037074aa8b7564b3b6f275f0"}
Sep 15 20:58:06.020696 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:06.020665    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wnktt" event={"ID":"49a4e9ea-347b-4653-9210-e9b142172bdd","Type":"ContainerStarted","Data":"35dfed4a8a448fa7b5a771df528772569e56e78468885301243d89ebb677f126"}
Sep 15 20:58:06.027056 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:06.027016    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pc79j" event={"ID":"ef60e7a1-62c7-46aa-8fd8-4043168ffa94","Type":"ContainerStarted","Data":"837b3814e76c46f40146bbe98cae7cc7c4eaa8f5a32ff422299c83f300a5e918"}
Sep 15 20:58:06.043759 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:06.043718    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-207.ec2.internal" event={"ID":"1c3857c19d72e89622f4c04f57a3da21","Type":"ContainerStarted","Data":"a2b2e3958959d0a1b0dd1348d23e1bd450b7b3364bc3d1388f8eca51d25e6b11"}
Sep 15 20:58:06.051958 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:06.051864    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xkmbd" event={"ID":"71567bed-fb87-4bb0-9833-a30ff9a16ef9","Type":"ContainerStarted","Data":"5f2f27f8c777614edba9d001c194f3a26b1a5c90714ca0308853c1933487d0bf"}
Sep 15 20:58:07.105345 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:07.105068    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-207.ec2.internal" event={"ID":"344eacc22c151cd63c505c6fcd3eb468","Type":"ContainerStarted","Data":"215e1bd5a6739d8c67b688b591e472270d7409ce360b8370a82252ab95a43ac5"}
Sep 15 20:58:07.121643 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:07.121578    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-207.ec2.internal" podStartSLOduration=4.121551836 podStartE2EDuration="4.121551836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:58:06.060166635 +0000 UTC m=+4.550812369" watchObservedRunningTime="2026-09-15 20:58:07.121551836 +0000 UTC m=+5.612197563"
Sep 15 20:58:07.489510 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:07.488817    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/932cce37-6a6a-4968-8cbe-d163e1da16f0-original-pull-secret\") pod \"global-pull-secret-syncer-qphq9\" (UID: \"932cce37-6a6a-4968-8cbe-d163e1da16f0\") " pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:07.489510 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:07.489014    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 20:58:07.489510 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:07.489081    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932cce37-6a6a-4968-8cbe-d163e1da16f0-original-pull-secret podName:932cce37-6a6a-4968-8cbe-d163e1da16f0 nodeName:}" failed. No retries permitted until 2026-09-15 20:58:11.489062029 +0000 UTC m=+9.979707747 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/932cce37-6a6a-4968-8cbe-d163e1da16f0-original-pull-secret") pod "global-pull-secret-syncer-qphq9" (UID: "932cce37-6a6a-4968-8cbe-d163e1da16f0") : object "kube-system"/"original-pull-secret" not registered
Sep 15 20:58:07.590751 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:07.590165    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53a126d9-fd09-4306-9547-39fad0299679-metrics-certs\") pod \"network-metrics-daemon-vf58t\" (UID: \"53a126d9-fd09-4306-9547-39fad0299679\") " pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:07.590751 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:07.590332    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:58:07.590751 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:07.590398    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53a126d9-fd09-4306-9547-39fad0299679-metrics-certs podName:53a126d9-fd09-4306-9547-39fad0299679 nodeName:}" failed. No retries permitted until 2026-09-15 20:58:11.590379605 +0000 UTC m=+10.081025332 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53a126d9-fd09-4306-9547-39fad0299679-metrics-certs") pod "network-metrics-daemon-vf58t" (UID: "53a126d9-fd09-4306-9547-39fad0299679") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:58:07.692303 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:07.691508    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvj4n\" (UniqueName: \"kubernetes.io/projected/e86303f1-1359-459e-910a-31c3a4256728-kube-api-access-fvj4n\") pod \"network-check-target-z2vkz\" (UID: \"e86303f1-1359-459e-910a-31c3a4256728\") " pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:07.692303 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:07.691749    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 20:58:07.692303 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:07.691771    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 20:58:07.692303 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:07.691785    2558 projected.go:196] Error preparing data for projected volume kube-api-access-fvj4n for pod openshift-network-diagnostics/network-check-target-z2vkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:58:07.692303 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:07.691861    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e86303f1-1359-459e-910a-31c3a4256728-kube-api-access-fvj4n podName:e86303f1-1359-459e-910a-31c3a4256728 nodeName:}" failed. No retries permitted until 2026-09-15 20:58:11.691840576 +0000 UTC m=+10.182486289 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fvj4n" (UniqueName: "kubernetes.io/projected/e86303f1-1359-459e-910a-31c3a4256728-kube-api-access-fvj4n") pod "network-check-target-z2vkz" (UID: "e86303f1-1359-459e-910a-31c3a4256728") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:58:07.938076 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:07.937235    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:07.938076 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:07.937387    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:07.938076 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:07.937813    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:07.938076 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:07.937904    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:07.938753 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:07.938533    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:07.938753 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:07.938657    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:09.936851 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:09.936766    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:09.936851 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:09.936787    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:09.937412 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:09.936916    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:09.937412 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:09.937181    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:09.937412 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:09.937265    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:09.937412 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:09.937330    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:11.526215 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:11.526171    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/932cce37-6a6a-4968-8cbe-d163e1da16f0-original-pull-secret\") pod \"global-pull-secret-syncer-qphq9\" (UID: \"932cce37-6a6a-4968-8cbe-d163e1da16f0\") " pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:11.526724 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:11.526379    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 20:58:11.526724 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:11.526434    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932cce37-6a6a-4968-8cbe-d163e1da16f0-original-pull-secret podName:932cce37-6a6a-4968-8cbe-d163e1da16f0 nodeName:}" failed. No retries permitted until 2026-09-15 20:58:19.526421184 +0000 UTC m=+18.017066909 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/932cce37-6a6a-4968-8cbe-d163e1da16f0-original-pull-secret") pod "global-pull-secret-syncer-qphq9" (UID: "932cce37-6a6a-4968-8cbe-d163e1da16f0") : object "kube-system"/"original-pull-secret" not registered
Sep 15 20:58:11.627011 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:11.626968    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53a126d9-fd09-4306-9547-39fad0299679-metrics-certs\") pod \"network-metrics-daemon-vf58t\" (UID: \"53a126d9-fd09-4306-9547-39fad0299679\") " pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:11.627224 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:11.627140    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:58:11.627290 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:11.627228    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53a126d9-fd09-4306-9547-39fad0299679-metrics-certs podName:53a126d9-fd09-4306-9547-39fad0299679 nodeName:}" failed. No retries permitted until 2026-09-15 20:58:19.627206137 +0000 UTC m=+18.117851864 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53a126d9-fd09-4306-9547-39fad0299679-metrics-certs") pod "network-metrics-daemon-vf58t" (UID: "53a126d9-fd09-4306-9547-39fad0299679") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:58:11.727982 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:11.727927    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvj4n\" (UniqueName: \"kubernetes.io/projected/e86303f1-1359-459e-910a-31c3a4256728-kube-api-access-fvj4n\") pod \"network-check-target-z2vkz\" (UID: \"e86303f1-1359-459e-910a-31c3a4256728\") " pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:11.728269 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:11.728139    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 20:58:11.728269 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:11.728173    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 20:58:11.728269 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:11.728188    2558 projected.go:196] Error preparing data for projected volume kube-api-access-fvj4n for pod openshift-network-diagnostics/network-check-target-z2vkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:58:11.728269 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:11.728260    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e86303f1-1359-459e-910a-31c3a4256728-kube-api-access-fvj4n podName:e86303f1-1359-459e-910a-31c3a4256728 nodeName:}" failed. No retries permitted until 2026-09-15 20:58:19.728239784 +0000 UTC m=+18.218885506 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fvj4n" (UniqueName: "kubernetes.io/projected/e86303f1-1359-459e-910a-31c3a4256728-kube-api-access-fvj4n") pod "network-check-target-z2vkz" (UID: "e86303f1-1359-459e-910a-31c3a4256728") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:58:11.941394 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:11.941320    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:11.942046 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:11.941718    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:11.942046 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:11.941740    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:11.942046 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:11.941776    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:11.942046 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:11.941964    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:11.942363 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:11.942227    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:13.936157 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:13.936036    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:13.936625 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:13.936036    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:13.936625 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:13.936273    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:13.936625 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:13.936427    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:13.936625 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:13.936092    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:13.936625 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:13.936525    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:15.936034 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:15.935998    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:15.936549 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:15.935998    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:15.936549 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:15.936148    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:15.936549 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:15.935998    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:15.936549 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:15.936248    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:15.936549 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:15.936339    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:17.935973 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:17.935935    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:17.936451 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:17.935935    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:17.936451 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:17.936085    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:17.936451 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:17.935946    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:17.936451 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:17.936168    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:17.936451 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:17.936273    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:19.589296 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:19.589254    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/932cce37-6a6a-4968-8cbe-d163e1da16f0-original-pull-secret\") pod \"global-pull-secret-syncer-qphq9\" (UID: \"932cce37-6a6a-4968-8cbe-d163e1da16f0\") " pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:19.589745 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:19.589428    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 20:58:19.589745 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:19.589536    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932cce37-6a6a-4968-8cbe-d163e1da16f0-original-pull-secret podName:932cce37-6a6a-4968-8cbe-d163e1da16f0 nodeName:}" failed. No retries permitted until 2026-09-15 20:58:35.589516009 +0000 UTC m=+34.080161743 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/932cce37-6a6a-4968-8cbe-d163e1da16f0-original-pull-secret") pod "global-pull-secret-syncer-qphq9" (UID: "932cce37-6a6a-4968-8cbe-d163e1da16f0") : object "kube-system"/"original-pull-secret" not registered
Sep 15 20:58:19.690177 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:19.690142    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53a126d9-fd09-4306-9547-39fad0299679-metrics-certs\") pod \"network-metrics-daemon-vf58t\" (UID: \"53a126d9-fd09-4306-9547-39fad0299679\") " pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:19.690375 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:19.690318    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:58:19.690438 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:19.690399    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53a126d9-fd09-4306-9547-39fad0299679-metrics-certs podName:53a126d9-fd09-4306-9547-39fad0299679 nodeName:}" failed. No retries permitted until 2026-09-15 20:58:35.690378588 +0000 UTC m=+34.181024315 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53a126d9-fd09-4306-9547-39fad0299679-metrics-certs") pod "network-metrics-daemon-vf58t" (UID: "53a126d9-fd09-4306-9547-39fad0299679") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:58:19.790668 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:19.790624    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvj4n\" (UniqueName: \"kubernetes.io/projected/e86303f1-1359-459e-910a-31c3a4256728-kube-api-access-fvj4n\") pod \"network-check-target-z2vkz\" (UID: \"e86303f1-1359-459e-910a-31c3a4256728\") " pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:19.790845 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:19.790808    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 20:58:19.790845 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:19.790837    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 20:58:19.790946 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:19.790851    2558 projected.go:196] Error preparing data for projected volume kube-api-access-fvj4n for pod openshift-network-diagnostics/network-check-target-z2vkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:58:19.790946 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:19.790911    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e86303f1-1359-459e-910a-31c3a4256728-kube-api-access-fvj4n podName:e86303f1-1359-459e-910a-31c3a4256728 nodeName:}" failed. No retries permitted until 2026-09-15 20:58:35.790894859 +0000 UTC m=+34.281540592 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fvj4n" (UniqueName: "kubernetes.io/projected/e86303f1-1359-459e-910a-31c3a4256728-kube-api-access-fvj4n") pod "network-check-target-z2vkz" (UID: "e86303f1-1359-459e-910a-31c3a4256728") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:58:19.936804 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:19.936722    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:19.936965 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:19.936853    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:19.936965 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:19.936864    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:19.936965 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:19.936898    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:19.937544 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:19.937481    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:19.937544 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:19.937500    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:21.937425 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:21.937375    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:21.937915 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:21.937465    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:21.937915 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:21.937496    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:21.937915 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:21.937524    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:21.937915 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:21.937574    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:21.937915 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:21.937645    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:23.937038 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:23.936784    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:23.937038 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:23.936809    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:23.937038 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:23.937032    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:23.938137 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:23.937110    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:23.938137 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:23.936889    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:23.938137 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:23.937182    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:24.145014 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:24.144980    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nwspf" event={"ID":"cb9c01cf-5113-402e-a9ff-8f177ab7f018","Type":"ContainerStarted","Data":"ead169393af75672bca34a95d7571dc483a9babaed87bab3e33418d012d18abc"}
Sep 15 20:58:24.146756 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:24.146729    2558 generic.go:356] "Generic (PLEG): container finished" podID="695e9f08-df31-4f11-b54e-583d69edbb05" containerID="6b0220a5ffc74ab5ac79b45edeaedd6fb7a51bdd3365445fa51df54c797ec269" exitCode=0
Sep 15 20:58:24.146866 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:24.146801    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrmmd" event={"ID":"695e9f08-df31-4f11-b54e-583d69edbb05","Type":"ContainerDied","Data":"6b0220a5ffc74ab5ac79b45edeaedd6fb7a51bdd3365445fa51df54c797ec269"}
Sep 15 20:58:24.152636 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:24.152609    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g" event={"ID":"86c2669e-e211-437d-81e3-b8d085d150f8","Type":"ContainerStarted","Data":"16ec17bdcb0f1921981d5626c304632162a33505c0ee7f4e62e45bced54b244e"}
Sep 15 20:58:24.152753 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:24.152692    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g" event={"ID":"86c2669e-e211-437d-81e3-b8d085d150f8","Type":"ContainerStarted","Data":"4ea5bd2fcfe0b2aed448ede7e6afd6026b387c4628fdbe00bf00ec92081912b0"}
Sep 15 20:58:24.152753 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:24.152743    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g" event={"ID":"86c2669e-e211-437d-81e3-b8d085d150f8","Type":"ContainerStarted","Data":"b952408f2ef0134bfee6f7be8be0eb544a279bdd7397abf2e9995e6c086dd8e8"}
Sep 15 20:58:24.152821 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:24.152788    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g" event={"ID":"86c2669e-e211-437d-81e3-b8d085d150f8","Type":"ContainerStarted","Data":"625fd388c22df2be23bd658b46e4db0bfb443f9a3bb85852dfae58d85ea7d233"}
Sep 15 20:58:24.152856 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:24.152834    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g" event={"ID":"86c2669e-e211-437d-81e3-b8d085d150f8","Type":"ContainerStarted","Data":"5d7444d4be9766351ca4337d257cad6afd48bd50dcafa103a0d99a12dc6b1dbd"}
Sep 15 20:58:24.154303 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:24.154269    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pc79j" event={"ID":"ef60e7a1-62c7-46aa-8fd8-4043168ffa94","Type":"ContainerStarted","Data":"5827515c26521efe19b48d4a21b582b1402c93dc43a6e4a7a14441419b1f6fae"}
Sep 15 20:58:24.155827 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:24.155805    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xkmbd" event={"ID":"71567bed-fb87-4bb0-9833-a30ff9a16ef9","Type":"ContainerStarted","Data":"850779c5639ad0e3e161f3b7ac17fe86bab99cd7f59d34530371c87e5b9c3a53"}
Sep 15 20:58:24.157272 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:24.157248    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk" event={"ID":"bab658f3-4be7-4df7-8709-a281f243c624","Type":"ContainerStarted","Data":"990596e150c4f8be568bcb09b2ea9dbae2b78d259d9dd8bd3de8093ef2fd4668"}
Sep 15 20:58:24.158962 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:24.158934    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vjzlh" event={"ID":"a4708b99-1a06-45a8-933f-0e0e0de9bde6","Type":"ContainerStarted","Data":"d38a4d44e62c82af6026264a2bcb4e0ff9a74850b17634f7ffdb9b8ed432df46"}
Sep 15 20:58:24.160409 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:24.160364    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nwspf" podStartSLOduration=3.554815564 podStartE2EDuration="22.160350533s" totalImagesPullingTime="18.605534969s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:58:24.160205497 +0000 UTC m=+22.650851256" watchObservedRunningTime="2026-09-15 20:58:24.160350533 +0000 UTC m=+22.650996266"
Sep 15 20:58:24.160529 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:24.160452    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-207.ec2.internal" podStartSLOduration=21.160444686 podStartE2EDuration="21.160444686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:58:07.122950835 +0000 UTC m=+5.613596573" watchObservedRunningTime="2026-09-15 20:58:24.160444686 +0000 UTC m=+22.651090419"
Sep 15 20:58:24.160623 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:24.160603    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fl48z" event={"ID":"3181c7fe-2852-46cd-aba2-6007201a049a","Type":"ContainerStarted","Data":"1616618a60fa19c1706177d943dc2679a825101ca86dc0468943df4fd102a652"}
Sep 15 20:58:24.176266 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:24.176221    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xkmbd" podStartSLOduration=3.5713274569999998 podStartE2EDuration="22.176204707s" totalImagesPullingTime="18.60487725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:58:24.17597743 +0000 UTC m=+22.666623173" watchObservedRunningTime="2026-09-15 20:58:24.176204707 +0000 UTC m=+22.666850440"
Sep 15 20:58:24.212123 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:24.212072    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vjzlh" podStartSLOduration=3.549355734 podStartE2EDuration="22.212037234s" totalImagesPullingTime="18.6626815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:58:24.211605833 +0000 UTC m=+22.702251564" watchObservedRunningTime="2026-09-15 20:58:24.212037234 +0000 UTC m=+22.702682965"
Sep 15 20:58:24.227300 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:24.227116    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pc79j" podStartSLOduration=3.616304806 podStartE2EDuration="22.227098998s" totalImagesPullingTime="18.610794192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:58:24.226530045 +0000 UTC m=+22.717175776" watchObservedRunningTime="2026-09-15 20:58:24.227098998 +0000 UTC m=+22.717744729"
Sep 15 20:58:24.248553 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:24.248507    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fl48z" podStartSLOduration=3.638154768 podStartE2EDuration="22.248490999s" totalImagesPullingTime="18.610336231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:58:24.248331325 +0000 UTC m=+22.738977057" watchObservedRunningTime="2026-09-15 20:58:24.248490999 +0000 UTC m=+22.739136731"
Sep 15 20:58:25.165905 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:25.165869    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g" event={"ID":"86c2669e-e211-437d-81e3-b8d085d150f8","Type":"ContainerStarted","Data":"8d94033b66ad37d56c277a8a4ea58b6fcbde4490e398c360abcafbac226dea6d"}
Sep 15 20:58:25.167486 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:25.167448    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wnktt" event={"ID":"49a4e9ea-347b-4653-9210-e9b142172bdd","Type":"ContainerStarted","Data":"3bd9ccf2cf974a59b6552b34fd92de262ba669fc7c8d086d7448ca052cc7d66e"}
Sep 15 20:58:25.181692 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:25.181641    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wnktt" podStartSLOduration=4.559371593 podStartE2EDuration="23.181622338s" totalImagesPullingTime="18.622250745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:58:25.181020393 +0000 UTC m=+23.671666125" watchObservedRunningTime="2026-09-15 20:58:25.181622338 +0000 UTC m=+23.672268069"
Sep 15 20:58:25.343877 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:25.343847    2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 20:58:25.921593 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:25.921476    2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T20:58:25.343868087Z","UUID":"c2f3c6d8-5f90-43fe-acec-bce2afa556be","Handler":null,"Name":"","Endpoint":""}
Sep 15 20:58:25.923292 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:25.923268    2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 20:58:25.923292 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:25.923306    2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 20:58:25.936416 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:25.936377    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:25.936416 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:25.936414    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:25.936654 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:25.936510    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:25.936654 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:25.936559    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:25.936758 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:25.936658    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:25.936795 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:25.936761    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:26.110339 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:26.110201    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pc79j"
Sep 15 20:58:26.111169 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:26.111145    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pc79j"
Sep 15 20:58:26.172553 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:26.172425    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk" event={"ID":"bab658f3-4be7-4df7-8709-a281f243c624","Type":"ContainerStarted","Data":"5bd54213f0354ce087fa313f8d435296785e72792839ba187471fbb6c66edf6f"}
Sep 15 20:58:26.173012 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:26.172975    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pc79j"
Sep 15 20:58:26.173398 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:26.173345    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pc79j"
Sep 15 20:58:27.179299 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:27.179260    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g" event={"ID":"86c2669e-e211-437d-81e3-b8d085d150f8","Type":"ContainerStarted","Data":"e23168af335ad81197c3bfd145053a5e33d89e5f1b32d254d51d921a401fc696"}
Sep 15 20:58:27.181603 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:27.181569    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk" event={"ID":"bab658f3-4be7-4df7-8709-a281f243c624","Type":"ContainerStarted","Data":"45d31dd3bfbe8f9139f6478aa605fc293730c72327c568a835edc51803168450"}
Sep 15 20:58:27.199813 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:27.199762    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvdxk" podStartSLOduration=4.314990016 podStartE2EDuration="25.1997456s" totalImagesPullingTime="20.884755584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:58:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:58:27.199186634 +0000 UTC m=+25.689832368" watchObservedRunningTime="2026-09-15 20:58:27.1997456 +0000 UTC m=+25.690391331"
Sep 15 20:58:27.936718 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:27.936677    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:27.936718 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:27.936708    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:27.936967 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:27.936677    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:27.936967 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:27.936800    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:27.936967 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:27.936847    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:27.936967 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:27.936927    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:29.188450 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:29.188419    2558 generic.go:356] "Generic (PLEG): container finished" podID="695e9f08-df31-4f11-b54e-583d69edbb05" containerID="77565e195947c929a87033b7fabb6325b7f13a19954d5eeddae7c13aeffd46b8" exitCode=0
Sep 15 20:58:29.189292 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:29.188509    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrmmd" event={"ID":"695e9f08-df31-4f11-b54e-583d69edbb05","Type":"ContainerDied","Data":"77565e195947c929a87033b7fabb6325b7f13a19954d5eeddae7c13aeffd46b8"}
Sep 15 20:58:29.193080 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:29.193032    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g" event={"ID":"86c2669e-e211-437d-81e3-b8d085d150f8","Type":"ContainerStarted","Data":"202b29b5930d9094b4db3aca3bc5972982dbbaa4e47c6d831fcec4f190dae602"}
Sep 15 20:58:29.193311 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:29.193283    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:29.208913 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:29.208888    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:29.233384 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:29.233335    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g" podStartSLOduration=8.567292361 podStartE2EDuration="27.233317937s" totalImagesPullingTime="18.666025576s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:58:29.233292992 +0000 UTC m=+27.723938724" watchObservedRunningTime="2026-09-15 20:58:29.233317937 +0000 UTC m=+27.723963663"
Sep 15 20:58:29.936559 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:29.936526    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:29.936559 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:29.936544    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:29.936829 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:29.936672    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:29.936829 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:29.936716    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:29.936935 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:29.936825    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:29.936935 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:29.936911    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:30.195974 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:30.195899    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:30.196401 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:30.195982    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:30.211193 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:30.211166    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:58:30.620405 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:30.620224    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z2vkz"]
Sep 15 20:58:30.620572 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:30.620522    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:30.620618 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:30.620605    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:30.623916 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:30.623888    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vf58t"]
Sep 15 20:58:30.624139 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:30.624012    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:30.624207 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:30.624143    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:30.624788 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:30.624745    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qphq9"]
Sep 15 20:58:30.624913 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:30.624860    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:30.624977 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:30.624955    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:31.200280 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:31.200239    2558 generic.go:356] "Generic (PLEG): container finished" podID="695e9f08-df31-4f11-b54e-583d69edbb05" containerID="2827919fb211b96cbd81b1e955a5465f7f129b3a245df9ce5447f8c6552304ce" exitCode=0
Sep 15 20:58:31.200749 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:31.200323    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrmmd" event={"ID":"695e9f08-df31-4f11-b54e-583d69edbb05","Type":"ContainerDied","Data":"2827919fb211b96cbd81b1e955a5465f7f129b3a245df9ce5447f8c6552304ce"}
Sep 15 20:58:31.936815 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:31.936781    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:31.937027 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:31.936888    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:31.937027 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:31.936970    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:31.937167 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:31.937085    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:31.937167 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:31.937149    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:31.937282 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:31.937167    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:33.206794 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:33.206759    2558 generic.go:356] "Generic (PLEG): container finished" podID="695e9f08-df31-4f11-b54e-583d69edbb05" containerID="99bedf3488957dd014e3fd946b7d10bee6bbd95551ba4ba89584bf1da665875c" exitCode=0
Sep 15 20:58:33.207205 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:33.206819    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrmmd" event={"ID":"695e9f08-df31-4f11-b54e-583d69edbb05","Type":"ContainerDied","Data":"99bedf3488957dd014e3fd946b7d10bee6bbd95551ba4ba89584bf1da665875c"}
Sep 15 20:58:33.936615 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:33.936379    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:33.936811 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:33.936379    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:33.936811 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:33.936728    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:33.936811 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:33.936428    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:33.936931 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:33.936808    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:33.936931 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:33.936902    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:35.609068 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:35.609020    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/932cce37-6a6a-4968-8cbe-d163e1da16f0-original-pull-secret\") pod \"global-pull-secret-syncer-qphq9\" (UID: \"932cce37-6a6a-4968-8cbe-d163e1da16f0\") " pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:35.609449 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:35.609180    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 20:58:35.609449 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:35.609250    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932cce37-6a6a-4968-8cbe-d163e1da16f0-original-pull-secret podName:932cce37-6a6a-4968-8cbe-d163e1da16f0 nodeName:}" failed. No retries permitted until 2026-09-15 20:59:07.609233421 +0000 UTC m=+66.099879146 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/932cce37-6a6a-4968-8cbe-d163e1da16f0-original-pull-secret") pod "global-pull-secret-syncer-qphq9" (UID: "932cce37-6a6a-4968-8cbe-d163e1da16f0") : object "kube-system"/"original-pull-secret" not registered
Sep 15 20:58:35.710393 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:35.710360    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53a126d9-fd09-4306-9547-39fad0299679-metrics-certs\") pod \"network-metrics-daemon-vf58t\" (UID: \"53a126d9-fd09-4306-9547-39fad0299679\") " pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:35.710583 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:35.710523    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:58:35.710630 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:35.710591    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53a126d9-fd09-4306-9547-39fad0299679-metrics-certs podName:53a126d9-fd09-4306-9547-39fad0299679 nodeName:}" failed. No retries permitted until 2026-09-15 20:59:07.71057366 +0000 UTC m=+66.201219369 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53a126d9-fd09-4306-9547-39fad0299679-metrics-certs") pod "network-metrics-daemon-vf58t" (UID: "53a126d9-fd09-4306-9547-39fad0299679") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:58:35.810788 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:35.810749    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvj4n\" (UniqueName: \"kubernetes.io/projected/e86303f1-1359-459e-910a-31c3a4256728-kube-api-access-fvj4n\") pod \"network-check-target-z2vkz\" (UID: \"e86303f1-1359-459e-910a-31c3a4256728\") " pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:35.810953 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:35.810890    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 20:58:35.810953 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:35.810905    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 20:58:35.810953 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:35.810916    2558 projected.go:196] Error preparing data for projected volume kube-api-access-fvj4n for pod openshift-network-diagnostics/network-check-target-z2vkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:58:35.811117 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:35.810978    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e86303f1-1359-459e-910a-31c3a4256728-kube-api-access-fvj4n podName:e86303f1-1359-459e-910a-31c3a4256728 nodeName:}" failed. No retries permitted until 2026-09-15 20:59:07.810962771 +0000 UTC m=+66.301608485 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fvj4n" (UniqueName: "kubernetes.io/projected/e86303f1-1359-459e-910a-31c3a4256728-kube-api-access-fvj4n") pod "network-check-target-z2vkz" (UID: "e86303f1-1359-459e-910a-31c3a4256728") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:58:35.936249 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:35.936213    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:35.936249 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:35.936238    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:35.936495 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:35.936214    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:35.936495 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:35.936355    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:35.936495 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:35.936402    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:35.936495 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:35.936464    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:37.936451 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:37.936414    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:37.936985 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:37.936419    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:37.936985 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:37.936525    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:37.936985 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:37.936419    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:37.936985 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:37.936593    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:37.936985 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:37.936663    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:38.221193 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:38.221160    2558 generic.go:356] "Generic (PLEG): container finished" podID="695e9f08-df31-4f11-b54e-583d69edbb05" containerID="3e4ccdcd30b6ddae97573c0bb5bdf11ef50c4598158009d55f9a776ac366d73b" exitCode=0
Sep 15 20:58:38.221359 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:38.221224    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrmmd" event={"ID":"695e9f08-df31-4f11-b54e-583d69edbb05","Type":"ContainerDied","Data":"3e4ccdcd30b6ddae97573c0bb5bdf11ef50c4598158009d55f9a776ac366d73b"}
Sep 15 20:58:39.228926 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:39.228892    2558 generic.go:356] "Generic (PLEG): container finished" podID="695e9f08-df31-4f11-b54e-583d69edbb05" containerID="a28f01f1e554980f94aba354d1aad65f997a18fca0aeffba9286429d5c795394" exitCode=0
Sep 15 20:58:39.229390 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:39.228946    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrmmd" event={"ID":"695e9f08-df31-4f11-b54e-583d69edbb05","Type":"ContainerDied","Data":"a28f01f1e554980f94aba354d1aad65f997a18fca0aeffba9286429d5c795394"}
Sep 15 20:58:39.935935 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:39.935897    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:39.936130 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:39.935897    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:39.936130 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:39.936007    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:39.936130 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:39.936031    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:39.936247 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:39.936153    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:39.936247 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:39.936227    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:40.235094 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:40.234995    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrmmd" event={"ID":"695e9f08-df31-4f11-b54e-583d69edbb05","Type":"ContainerStarted","Data":"b5c977f5c9f9fcdd9a6b4b121664372c81c58e4a5c46256a4cf624d75b508674"}
Sep 15 20:58:40.260240 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:40.260187    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hrmmd" podStartSLOduration=7.812403296 podStartE2EDuration="38.260169297s" totalImagesPullingTime="30.447766001s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:58:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:58:40.259878761 +0000 UTC m=+38.750524492" watchObservedRunningTime="2026-09-15 20:58:40.260169297 +0000 UTC m=+38.750815029"
Sep 15 20:58:41.937507 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:41.937469    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:41.937929 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:41.937580    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:41.937929 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:41.937654    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:41.940605 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:41.938284    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:41.940605 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:41.938397    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:41.940605 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:41.939436    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:43.936169 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:43.936132    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:43.936169 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:43.936149    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:43.936619 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:43.936149    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:43.936619 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:43.936255    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:43.936619 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:43.936391    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:43.936619 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:43.936416    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:45.935906 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:45.935869    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:45.936403 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:45.936011    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:45.936403 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:45.936008    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:45.936403 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:45.936033    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:45.936403 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:45.936145    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:45.936403 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:45.936237    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:47.936446 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:47.936408    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:47.936446 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:47.936429    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:47.936940 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:47.936421    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:47.936940 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:47.936511    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:47.936940 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:47.936586    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:47.936940 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:47.936668    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:49.936697 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:49.936658    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:49.937206 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:49.936658    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:49.937206 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:49.936811    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:49.937206 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:49.936658    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:49.937206 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:49.936845    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:49.937206 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:49.936914    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:51.937199 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:51.937163    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:51.937672 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:51.937242    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:51.937672 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:51.937243    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:51.937672 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:51.937391    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:51.937672 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:51.937455    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:51.937672 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:51.937517    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:53.936641 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:53.936609    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:53.937019 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:53.936609    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:53.937019 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:53.936732    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:53.937019 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:53.936785    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:53.937019 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:53.936806    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:53.937019 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:53.936863    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:55.936365 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:55.936326    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:55.936844 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:55.936336    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:55.936844 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:55.936426    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:55.936844 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:55.936336    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:55.936844 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:55.936517    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:55.936844 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:55.936598    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:57.936155 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:57.936117    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:57.936708 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:57.936129    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:57.936708 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:57.936219    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qphq9" podUID="932cce37-6a6a-4968-8cbe-d163e1da16f0"
Sep 15 20:58:57.936708 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:57.936244    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:57.936708 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:57.936336    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2vkz" podUID="e86303f1-1359-459e-910a-31c3a4256728"
Sep 15 20:58:57.936708 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:58:57.936381    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vf58t" podUID="53a126d9-fd09-4306-9547-39fad0299679"
Sep 15 20:58:58.330119 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.330086    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-207.ec2.internal" event="NodeReady"
Sep 15 20:58:58.330284 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.330223    2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 20:58:58.369802 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.369771    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8d6b7655-2cwn2"]
Sep 15 20:58:58.371620 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.371603    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.374731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.374708    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 15 20:58:58.374731 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.374733    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 15 20:58:58.374932 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.374775    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xpshw\""
Sep 15 20:58:58.374932 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.374809    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 15 20:58:58.381043 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.381020    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 15 20:58:58.384613 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.384591    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-m4ncl"]
Sep 15 20:58:58.386320 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.386303    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m4ncl"
Sep 15 20:58:58.390498 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.390477    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 20:58:58.390643 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.390499    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 20:58:58.391474 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.391307    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wvdjj\""
Sep 15 20:58:58.391474 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.391318    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 20:58:58.391631 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.391524    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8d6b7655-2cwn2"]
Sep 15 20:58:58.395668 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.395643    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ftppc"]
Sep 15 20:58:58.397556 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.397536    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ftppc"
Sep 15 20:58:58.400310 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.400288    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p626r\""
Sep 15 20:58:58.400417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.400296    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 20:58:58.400417 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.400388    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 20:58:58.403610 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.403444    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m4ncl"]
Sep 15 20:58:58.409501 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.409475    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ftppc"]
Sep 15 20:58:58.463476 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.463386    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3b8bea-139d-46e2-9477-22e1230ca193-config-volume\") pod \"dns-default-ftppc\" (UID: \"8f3b8bea-139d-46e2-9477-22e1230ca193\") " pod="openshift-dns/dns-default-ftppc"
Sep 15 20:58:58.463476 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.463418    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f3b8bea-139d-46e2-9477-22e1230ca193-metrics-tls\") pod \"dns-default-ftppc\" (UID: \"8f3b8bea-139d-46e2-9477-22e1230ca193\") " pod="openshift-dns/dns-default-ftppc"
Sep 15 20:58:58.463476 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.463434    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f3b8bea-139d-46e2-9477-22e1230ca193-tmp-dir\") pod \"dns-default-ftppc\" (UID: \"8f3b8bea-139d-46e2-9477-22e1230ca193\") " pod="openshift-dns/dns-default-ftppc"
Sep 15 20:58:58.463476 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.463455    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2gr7\" (UniqueName: \"kubernetes.io/projected/e9273ede-dc5b-4757-a016-61ed4a57e043-kube-api-access-t2gr7\") pod \"image-registry-8d6b7655-2cwn2\" (UID: \"e9273ede-dc5b-4757-a016-61ed4a57e043\") " pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.463476 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.463475    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e9273ede-dc5b-4757-a016-61ed4a57e043-bound-sa-token\") pod \"image-registry-8d6b7655-2cwn2\" (UID: \"e9273ede-dc5b-4757-a016-61ed4a57e043\") " pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.463858 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.463534    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e9273ede-dc5b-4757-a016-61ed4a57e043-registry-tls\") pod \"image-registry-8d6b7655-2cwn2\" (UID: \"e9273ede-dc5b-4757-a016-61ed4a57e043\") " pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.463858 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.463640    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d69d\" (UniqueName: \"kubernetes.io/projected/6dc59ea6-8d86-456d-b250-e2d0fc7a2d10-kube-api-access-4d69d\") pod \"ingress-canary-m4ncl\" (UID: \"6dc59ea6-8d86-456d-b250-e2d0fc7a2d10\") " pod="openshift-ingress-canary/ingress-canary-m4ncl"
Sep 15 20:58:58.463858 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.463679    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e9273ede-dc5b-4757-a016-61ed4a57e043-image-registry-private-configuration\") pod \"image-registry-8d6b7655-2cwn2\" (UID: \"e9273ede-dc5b-4757-a016-61ed4a57e043\") " pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.463858 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.463712    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e9273ede-dc5b-4757-a016-61ed4a57e043-ca-trust-extracted\") pod \"image-registry-8d6b7655-2cwn2\" (UID: \"e9273ede-dc5b-4757-a016-61ed4a57e043\") " pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.463858 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.463736    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9273ede-dc5b-4757-a016-61ed4a57e043-trusted-ca\") pod \"image-registry-8d6b7655-2cwn2\" (UID: \"e9273ede-dc5b-4757-a016-61ed4a57e043\") " pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.463858 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.463758    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6dc59ea6-8d86-456d-b250-e2d0fc7a2d10-cert\") pod \"ingress-canary-m4ncl\" (UID: \"6dc59ea6-8d86-456d-b250-e2d0fc7a2d10\") " pod="openshift-ingress-canary/ingress-canary-m4ncl"
Sep 15 20:58:58.463858 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.463783    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e9273ede-dc5b-4757-a016-61ed4a57e043-registry-certificates\") pod \"image-registry-8d6b7655-2cwn2\" (UID: \"e9273ede-dc5b-4757-a016-61ed4a57e043\") " pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.463858 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.463807    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e9273ede-dc5b-4757-a016-61ed4a57e043-installation-pull-secrets\") pod \"image-registry-8d6b7655-2cwn2\" (UID: \"e9273ede-dc5b-4757-a016-61ed4a57e043\") " pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.463858 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.463831    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v946l\" (UniqueName: \"kubernetes.io/projected/8f3b8bea-139d-46e2-9477-22e1230ca193-kube-api-access-v946l\") pod \"dns-default-ftppc\" (UID: \"8f3b8bea-139d-46e2-9477-22e1230ca193\") " pod="openshift-dns/dns-default-ftppc"
Sep 15 20:58:58.564411 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.564371    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d69d\" (UniqueName: \"kubernetes.io/projected/6dc59ea6-8d86-456d-b250-e2d0fc7a2d10-kube-api-access-4d69d\") pod \"ingress-canary-m4ncl\" (UID: \"6dc59ea6-8d86-456d-b250-e2d0fc7a2d10\") " pod="openshift-ingress-canary/ingress-canary-m4ncl"
Sep 15 20:58:58.564494 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.564413    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e9273ede-dc5b-4757-a016-61ed4a57e043-image-registry-private-configuration\") pod \"image-registry-8d6b7655-2cwn2\" (UID: \"e9273ede-dc5b-4757-a016-61ed4a57e043\") " pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.564494 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.564441    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e9273ede-dc5b-4757-a016-61ed4a57e043-ca-trust-extracted\") pod \"image-registry-8d6b7655-2cwn2\" (UID: \"e9273ede-dc5b-4757-a016-61ed4a57e043\") " pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.564494 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.564464    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9273ede-dc5b-4757-a016-61ed4a57e043-trusted-ca\") pod \"image-registry-8d6b7655-2cwn2\" (UID: \"e9273ede-dc5b-4757-a016-61ed4a57e043\") " pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.564646 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.564507    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6dc59ea6-8d86-456d-b250-e2d0fc7a2d10-cert\") pod \"ingress-canary-m4ncl\" (UID: \"6dc59ea6-8d86-456d-b250-e2d0fc7a2d10\") " pod="openshift-ingress-canary/ingress-canary-m4ncl"
Sep 15 20:58:58.564646 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.564555    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e9273ede-dc5b-4757-a016-61ed4a57e043-registry-certificates\") pod \"image-registry-8d6b7655-2cwn2\" (UID: \"e9273ede-dc5b-4757-a016-61ed4a57e043\") " pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.564646 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.564591    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e9273ede-dc5b-4757-a016-61ed4a57e043-installation-pull-secrets\") pod \"image-registry-8d6b7655-2cwn2\" (UID: \"e9273ede-dc5b-4757-a016-61ed4a57e043\") " pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.564646 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.564625    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v946l\" (UniqueName: \"kubernetes.io/projected/8f3b8bea-139d-46e2-9477-22e1230ca193-kube-api-access-v946l\") pod \"dns-default-ftppc\" (UID: \"8f3b8bea-139d-46e2-9477-22e1230ca193\") " pod="openshift-dns/dns-default-ftppc"
Sep 15 20:58:58.564890 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.564711    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3b8bea-139d-46e2-9477-22e1230ca193-config-volume\") pod \"dns-default-ftppc\" (UID: \"8f3b8bea-139d-46e2-9477-22e1230ca193\") " pod="openshift-dns/dns-default-ftppc"
Sep 15 20:58:58.564890 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.564744    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f3b8bea-139d-46e2-9477-22e1230ca193-metrics-tls\") pod \"dns-default-ftppc\" (UID: \"8f3b8bea-139d-46e2-9477-22e1230ca193\") " pod="openshift-dns/dns-default-ftppc"
Sep 15 20:58:58.564890 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.564778    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f3b8bea-139d-46e2-9477-22e1230ca193-tmp-dir\") pod \"dns-default-ftppc\" (UID: \"8f3b8bea-139d-46e2-9477-22e1230ca193\") " pod="openshift-dns/dns-default-ftppc"
Sep 15 20:58:58.565026 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.564925    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e9273ede-dc5b-4757-a016-61ed4a57e043-ca-trust-extracted\") pod \"image-registry-8d6b7655-2cwn2\" (UID: \"e9273ede-dc5b-4757-a016-61ed4a57e043\") " pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.565117 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.565089    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2gr7\" (UniqueName: \"kubernetes.io/projected/e9273ede-dc5b-4757-a016-61ed4a57e043-kube-api-access-t2gr7\") pod \"image-registry-8d6b7655-2cwn2\" (UID: \"e9273ede-dc5b-4757-a016-61ed4a57e043\") " pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.565183 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.565140    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e9273ede-dc5b-4757-a016-61ed4a57e043-bound-sa-token\") pod \"image-registry-8d6b7655-2cwn2\" (UID: \"e9273ede-dc5b-4757-a016-61ed4a57e043\") " pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.565235 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.565187    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e9273ede-dc5b-4757-a016-61ed4a57e043-registry-tls\") pod \"image-registry-8d6b7655-2cwn2\" (UID: \"e9273ede-dc5b-4757-a016-61ed4a57e043\") " pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.565286 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.565252    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f3b8bea-139d-46e2-9477-22e1230ca193-tmp-dir\") pod \"dns-default-ftppc\" (UID: \"8f3b8bea-139d-46e2-9477-22e1230ca193\") " pod="openshift-dns/dns-default-ftppc"
Sep 15 20:58:58.565589 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.565565    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e9273ede-dc5b-4757-a016-61ed4a57e043-registry-certificates\") pod \"image-registry-8d6b7655-2cwn2\" (UID: \"e9273ede-dc5b-4757-a016-61ed4a57e043\") " pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.565970 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.565947    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3b8bea-139d-46e2-9477-22e1230ca193-config-volume\") pod \"dns-default-ftppc\" (UID: \"8f3b8bea-139d-46e2-9477-22e1230ca193\") " pod="openshift-dns/dns-default-ftppc"
Sep 15 20:58:58.566199 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.566174    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9273ede-dc5b-4757-a016-61ed4a57e043-trusted-ca\") pod \"image-registry-8d6b7655-2cwn2\" (UID: \"e9273ede-dc5b-4757-a016-61ed4a57e043\") " pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.568979 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.568948    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f3b8bea-139d-46e2-9477-22e1230ca193-metrics-tls\") pod \"dns-default-ftppc\" (UID: \"8f3b8bea-139d-46e2-9477-22e1230ca193\") " pod="openshift-dns/dns-default-ftppc"
Sep 15 20:58:58.569106 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.569013    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e9273ede-dc5b-4757-a016-61ed4a57e043-installation-pull-secrets\") pod \"image-registry-8d6b7655-2cwn2\" (UID: \"e9273ede-dc5b-4757-a016-61ed4a57e043\") " pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.569106 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.569061    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e9273ede-dc5b-4757-a016-61ed4a57e043-image-registry-private-configuration\") pod \"image-registry-8d6b7655-2cwn2\" (UID: \"e9273ede-dc5b-4757-a016-61ed4a57e043\") " pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.569106 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.569082    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6dc59ea6-8d86-456d-b250-e2d0fc7a2d10-cert\") pod \"ingress-canary-m4ncl\" (UID: \"6dc59ea6-8d86-456d-b250-e2d0fc7a2d10\") " pod="openshift-ingress-canary/ingress-canary-m4ncl"
Sep 15 20:58:58.569215 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.569100    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e9273ede-dc5b-4757-a016-61ed4a57e043-registry-tls\") pod \"image-registry-8d6b7655-2cwn2\" (UID: \"e9273ede-dc5b-4757-a016-61ed4a57e043\") " pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.574022 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.573992    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e9273ede-dc5b-4757-a016-61ed4a57e043-bound-sa-token\") pod \"image-registry-8d6b7655-2cwn2\" (UID: \"e9273ede-dc5b-4757-a016-61ed4a57e043\") " pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.574187 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.574171    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d69d\" (UniqueName: \"kubernetes.io/projected/6dc59ea6-8d86-456d-b250-e2d0fc7a2d10-kube-api-access-4d69d\") pod \"ingress-canary-m4ncl\" (UID: \"6dc59ea6-8d86-456d-b250-e2d0fc7a2d10\") " pod="openshift-ingress-canary/ingress-canary-m4ncl"
Sep 15 20:58:58.574293 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.574275    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v946l\" (UniqueName: \"kubernetes.io/projected/8f3b8bea-139d-46e2-9477-22e1230ca193-kube-api-access-v946l\") pod \"dns-default-ftppc\" (UID: \"8f3b8bea-139d-46e2-9477-22e1230ca193\") " pod="openshift-dns/dns-default-ftppc"
Sep 15 20:58:58.574367 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.574349    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2gr7\" (UniqueName: \"kubernetes.io/projected/e9273ede-dc5b-4757-a016-61ed4a57e043-kube-api-access-t2gr7\") pod \"image-registry-8d6b7655-2cwn2\" (UID: \"e9273ede-dc5b-4757-a016-61ed4a57e043\") " pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.682777 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.682741    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:58.695548 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.695516    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m4ncl"
Sep 15 20:58:58.706676 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.706642    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ftppc"
Sep 15 20:58:58.970139 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.970015    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m4ncl"]
Sep 15 20:58:58.972661 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.972619    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8d6b7655-2cwn2"]
Sep 15 20:58:58.973472 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:58.973448    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc59ea6_8d86_456d_b250_e2d0fc7a2d10.slice/crio-c51c389f87d648568f11393c0190011dfd355cbf69e897f24690ffee5eee1738 WatchSource:0}: Error finding container c51c389f87d648568f11393c0190011dfd355cbf69e897f24690ffee5eee1738: Status 404 returned error can't find the container with id c51c389f87d648568f11393c0190011dfd355cbf69e897f24690ffee5eee1738
Sep 15 20:58:58.977045 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:58.976926    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9273ede_dc5b_4757_a016_61ed4a57e043.slice/crio-773f62ff12216a7438990e15c7fe801cc55be17d37c2be16a20ded6228c130e4 WatchSource:0}: Error finding container 773f62ff12216a7438990e15c7fe801cc55be17d37c2be16a20ded6228c130e4: Status 404 returned error can't find the container with id 773f62ff12216a7438990e15c7fe801cc55be17d37c2be16a20ded6228c130e4
Sep 15 20:58:58.988230 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:58.988195    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ftppc"]
Sep 15 20:58:58.991396 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:58:58.991359    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3b8bea_139d_46e2_9477_22e1230ca193.slice/crio-1585900d7202775f5729101078d6efc67cdd660b6b6eb7cf6172bf05c10d98bf WatchSource:0}: Error finding container 1585900d7202775f5729101078d6efc67cdd660b6b6eb7cf6172bf05c10d98bf: Status 404 returned error can't find the container with id 1585900d7202775f5729101078d6efc67cdd660b6b6eb7cf6172bf05c10d98bf
Sep 15 20:58:59.279215 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:59.279127    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ftppc" event={"ID":"8f3b8bea-139d-46e2-9477-22e1230ca193","Type":"ContainerStarted","Data":"1585900d7202775f5729101078d6efc67cdd660b6b6eb7cf6172bf05c10d98bf"}
Sep 15 20:58:59.280412 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:59.280322    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m4ncl" event={"ID":"6dc59ea6-8d86-456d-b250-e2d0fc7a2d10","Type":"ContainerStarted","Data":"c51c389f87d648568f11393c0190011dfd355cbf69e897f24690ffee5eee1738"}
Sep 15 20:58:59.281903 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:59.281878    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8d6b7655-2cwn2" event={"ID":"e9273ede-dc5b-4757-a016-61ed4a57e043","Type":"ContainerStarted","Data":"06215e5842db58e3bf9cca761954d98a8e50a1019541d9f57ceb1c7d4569cacd"}
Sep 15 20:58:59.282028 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:59.281921    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8d6b7655-2cwn2" event={"ID":"e9273ede-dc5b-4757-a016-61ed4a57e043","Type":"ContainerStarted","Data":"773f62ff12216a7438990e15c7fe801cc55be17d37c2be16a20ded6228c130e4"}
Sep 15 20:58:59.282028 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:59.281962    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:58:59.301880 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:59.301826    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8d6b7655-2cwn2" podStartSLOduration=1.301807927 podStartE2EDuration="1.301807927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:58:59.301264627 +0000 UTC m=+57.791910361" watchObservedRunningTime="2026-09-15 20:58:59.301807927 +0000 UTC m=+57.792453659"
Sep 15 20:58:59.936954 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:59.936913    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:58:59.937170 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:59.936913    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:58:59.937170 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:59.937096    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:58:59.940185 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:59.940160    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 20:58:59.941901 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:59.941574    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 20:58:59.941901 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:59.941620    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-js7d5\""
Sep 15 20:58:59.941901 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:59.941641    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r6tvw\""
Sep 15 20:58:59.941901 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:59.941621    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 20:58:59.941901 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:58:59.941699    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 20:59:02.214902 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:02.214867    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6fp7g"
Sep 15 20:59:02.292366 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:02.292331    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ftppc" event={"ID":"8f3b8bea-139d-46e2-9477-22e1230ca193","Type":"ContainerStarted","Data":"b9822b264e3eb2062d5aeeb699323f8acf2111ef7b85a88228e3ab3bf0b36f1b"}
Sep 15 20:59:02.292585 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:02.292397    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ftppc" event={"ID":"8f3b8bea-139d-46e2-9477-22e1230ca193","Type":"ContainerStarted","Data":"58a2a69e96a4c7eb7b94f748d124cf1ef993739785db45e8e97279d9619438fb"}
Sep 15 20:59:02.292585 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:02.292423    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ftppc"
Sep 15 20:59:02.294088 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:02.294043    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m4ncl" event={"ID":"6dc59ea6-8d86-456d-b250-e2d0fc7a2d10","Type":"ContainerStarted","Data":"da519c5f8f239f72d545b4c047efb38ca624252c19637206f698f95f6855fd92"}
Sep 15 20:59:02.309071 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:02.309010    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ftppc" podStartSLOduration=1.324835006 podStartE2EDuration="4.308995187s" totalImagesPullingTime="2.984160181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:59:02.308612668 +0000 UTC m=+60.799258401" watchObservedRunningTime="2026-09-15 20:59:02.308995187 +0000 UTC m=+60.799640913"
Sep 15 20:59:02.338558 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:02.338508    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-m4ncl" podStartSLOduration=1.332310099 podStartE2EDuration="4.338490482s" totalImagesPullingTime="3.006180383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:59:02.338147513 +0000 UTC m=+60.828793244" watchObservedRunningTime="2026-09-15 20:59:02.338490482 +0000 UTC m=+60.829136213"
Sep 15 20:59:07.629544 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:07.629493    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/932cce37-6a6a-4968-8cbe-d163e1da16f0-original-pull-secret\") pod \"global-pull-secret-syncer-qphq9\" (UID: \"932cce37-6a6a-4968-8cbe-d163e1da16f0\") " pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:59:07.632495 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:07.632470    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 20:59:07.642376 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:07.642347    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/932cce37-6a6a-4968-8cbe-d163e1da16f0-original-pull-secret\") pod \"global-pull-secret-syncer-qphq9\" (UID: \"932cce37-6a6a-4968-8cbe-d163e1da16f0\") " pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:59:07.730706 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:07.730663    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53a126d9-fd09-4306-9547-39fad0299679-metrics-certs\") pod \"network-metrics-daemon-vf58t\" (UID: \"53a126d9-fd09-4306-9547-39fad0299679\") " pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:59:07.734092 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:07.734045    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 20:59:07.743760 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:07.743727    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53a126d9-fd09-4306-9547-39fad0299679-metrics-certs\") pod \"network-metrics-daemon-vf58t\" (UID: \"53a126d9-fd09-4306-9547-39fad0299679\") " pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:59:07.753331 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:07.753306    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r6tvw\""
Sep 15 20:59:07.761760 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:07.761728    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vf58t"
Sep 15 20:59:07.765999 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:07.765977    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qphq9"
Sep 15 20:59:07.831347 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:07.831311    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvj4n\" (UniqueName: \"kubernetes.io/projected/e86303f1-1359-459e-910a-31c3a4256728-kube-api-access-fvj4n\") pod \"network-check-target-z2vkz\" (UID: \"e86303f1-1359-459e-910a-31c3a4256728\") " pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:59:07.834948 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:07.834916    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 20:59:07.846150 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:07.845549    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 20:59:07.856338 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:07.856287    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvj4n\" (UniqueName: \"kubernetes.io/projected/e86303f1-1359-459e-910a-31c3a4256728-kube-api-access-fvj4n\") pod \"network-check-target-z2vkz\" (UID: \"e86303f1-1359-459e-910a-31c3a4256728\") " pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:59:08.061736 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:08.061705    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-js7d5\""
Sep 15 20:59:08.066347 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:08.066319    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vf58t"]
Sep 15 20:59:08.069512 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:08.069490    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:59:08.070122 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:59:08.070099    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a126d9_fd09_4306_9547_39fad0299679.slice/crio-6de68d46691121f14559c8e6cbf38f3388b4af5b7f86df9204298eaa8580c22c WatchSource:0}: Error finding container 6de68d46691121f14559c8e6cbf38f3388b4af5b7f86df9204298eaa8580c22c: Status 404 returned error can't find the container with id 6de68d46691121f14559c8e6cbf38f3388b4af5b7f86df9204298eaa8580c22c
Sep 15 20:59:08.092032 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:08.092001    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qphq9"]
Sep 15 20:59:08.095034 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:59:08.095004    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932cce37_6a6a_4968_8cbe_d163e1da16f0.slice/crio-faa24f249208dec55b59de2ca39aa464b0d884804137ee7573d04100d5a56107 WatchSource:0}: Error finding container faa24f249208dec55b59de2ca39aa464b0d884804137ee7573d04100d5a56107: Status 404 returned error can't find the container with id faa24f249208dec55b59de2ca39aa464b0d884804137ee7573d04100d5a56107
Sep 15 20:59:08.309490 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:08.309455    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vf58t" event={"ID":"53a126d9-fd09-4306-9547-39fad0299679","Type":"ContainerStarted","Data":"6de68d46691121f14559c8e6cbf38f3388b4af5b7f86df9204298eaa8580c22c"}
Sep 15 20:59:08.310539 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:08.310516    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qphq9" event={"ID":"932cce37-6a6a-4968-8cbe-d163e1da16f0","Type":"ContainerStarted","Data":"faa24f249208dec55b59de2ca39aa464b0d884804137ee7573d04100d5a56107"}
Sep 15 20:59:08.365827 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:08.365748    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z2vkz"]
Sep 15 20:59:08.368732 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:59:08.368695    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86303f1_1359_459e_910a_31c3a4256728.slice/crio-5e110a9995432847787e5f4af8a9efa48de77e539d6fe489fa5d29ccf71bab4f WatchSource:0}: Error finding container 5e110a9995432847787e5f4af8a9efa48de77e539d6fe489fa5d29ccf71bab4f: Status 404 returned error can't find the container with id 5e110a9995432847787e5f4af8a9efa48de77e539d6fe489fa5d29ccf71bab4f
Sep 15 20:59:08.913154 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:08.913115    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6rn7r"]
Sep 15 20:59:08.937277 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:08.937204    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6rn7r"]
Sep 15 20:59:08.937440 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:08.937326    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:08.940684 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:08.940575    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 20:59:08.940684 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:08.940641    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 20:59:08.940913 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:08.940708    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 20:59:08.941123 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:08.940993    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 20:59:08.941805 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:08.941784    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-22kzx\""
Sep 15 20:59:09.041285 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:09.041245    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a2612f1-6b01-421d-b105-f4a20b6356f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:09.041478 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:09.041294    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a2612f1-6b01-421d-b105-f4a20b6356f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:09.041478 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:09.041336    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnbxn\" (UniqueName: \"kubernetes.io/projected/6a2612f1-6b01-421d-b105-f4a20b6356f7-kube-api-access-hnbxn\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:09.041478 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:09.041365    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a2612f1-6b01-421d-b105-f4a20b6356f7-data-volume\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:09.041648 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:09.041465    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6a2612f1-6b01-421d-b105-f4a20b6356f7-crio-socket\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:09.143128 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:09.142490    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6a2612f1-6b01-421d-b105-f4a20b6356f7-crio-socket\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:09.143128 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:09.142568    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a2612f1-6b01-421d-b105-f4a20b6356f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:09.143128 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:09.142597    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a2612f1-6b01-421d-b105-f4a20b6356f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:09.143128 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:09.142621    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnbxn\" (UniqueName: \"kubernetes.io/projected/6a2612f1-6b01-421d-b105-f4a20b6356f7-kube-api-access-hnbxn\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:09.143128 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:09.142651    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a2612f1-6b01-421d-b105-f4a20b6356f7-data-volume\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:09.143128 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:09.142650    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6a2612f1-6b01-421d-b105-f4a20b6356f7-crio-socket\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:09.143128 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:59:09.142760    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 20:59:09.143128 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:59:09.142798    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 20:59:09.143128 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:59:09.142863    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a2612f1-6b01-421d-b105-f4a20b6356f7-kube-rbac-proxy-cm podName:6a2612f1-6b01-421d-b105-f4a20b6356f7 nodeName:}" failed. No retries permitted until 2026-09-15 20:59:09.642840254 +0000 UTC m=+68.133485979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6a2612f1-6b01-421d-b105-f4a20b6356f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6rn7r" (UID: "6a2612f1-6b01-421d-b105-f4a20b6356f7") : configmap "kube-rbac-proxy" not found
Sep 15 20:59:09.143128 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:59:09.142989    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2612f1-6b01-421d-b105-f4a20b6356f7-insights-runtime-extractor-tls podName:6a2612f1-6b01-421d-b105-f4a20b6356f7 nodeName:}" failed. No retries permitted until 2026-09-15 20:59:09.642974924 +0000 UTC m=+68.133620640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6a2612f1-6b01-421d-b105-f4a20b6356f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6rn7r" (UID: "6a2612f1-6b01-421d-b105-f4a20b6356f7") : secret "insights-runtime-extractor-tls" not found
Sep 15 20:59:09.143804 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:09.143182    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a2612f1-6b01-421d-b105-f4a20b6356f7-data-volume\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:09.153199 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:09.153132    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnbxn\" (UniqueName: \"kubernetes.io/projected/6a2612f1-6b01-421d-b105-f4a20b6356f7-kube-api-access-hnbxn\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:09.315638 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:09.315328    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z2vkz" event={"ID":"e86303f1-1359-459e-910a-31c3a4256728","Type":"ContainerStarted","Data":"5e110a9995432847787e5f4af8a9efa48de77e539d6fe489fa5d29ccf71bab4f"}
Sep 15 20:59:09.648009 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:09.647894    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a2612f1-6b01-421d-b105-f4a20b6356f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:09.648009 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:09.647949    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a2612f1-6b01-421d-b105-f4a20b6356f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:09.648384 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:59:09.648041    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 20:59:09.648384 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:59:09.648081    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 20:59:09.648384 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:59:09.648151    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2612f1-6b01-421d-b105-f4a20b6356f7-insights-runtime-extractor-tls podName:6a2612f1-6b01-421d-b105-f4a20b6356f7 nodeName:}" failed. No retries permitted until 2026-09-15 20:59:10.648131281 +0000 UTC m=+69.138776998 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6a2612f1-6b01-421d-b105-f4a20b6356f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6rn7r" (UID: "6a2612f1-6b01-421d-b105-f4a20b6356f7") : secret "insights-runtime-extractor-tls" not found
Sep 15 20:59:09.648384 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:59:09.648173    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a2612f1-6b01-421d-b105-f4a20b6356f7-kube-rbac-proxy-cm podName:6a2612f1-6b01-421d-b105-f4a20b6356f7 nodeName:}" failed. No retries permitted until 2026-09-15 20:59:10.648162415 +0000 UTC m=+69.138808131 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6a2612f1-6b01-421d-b105-f4a20b6356f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6rn7r" (UID: "6a2612f1-6b01-421d-b105-f4a20b6356f7") : configmap "kube-rbac-proxy" not found
Sep 15 20:59:09.996384 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:09.996350    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ftppc_8f3b8bea-139d-46e2-9477-22e1230ca193/dns/0.log"
Sep 15 20:59:10.179944 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:10.179913    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ftppc_8f3b8bea-139d-46e2-9477-22e1230ca193/kube-rbac-proxy/0.log"
Sep 15 20:59:10.323204 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:10.323119    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vf58t" event={"ID":"53a126d9-fd09-4306-9547-39fad0299679","Type":"ContainerStarted","Data":"ee1d92d3f9ebad24afa6588be86d332e1962330d04bb0360bbaefdf35eee600f"}
Sep 15 20:59:10.323204 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:10.323181    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vf58t" event={"ID":"53a126d9-fd09-4306-9547-39fad0299679","Type":"ContainerStarted","Data":"475d7b5021936305b28f35fb297ba02875922491506e2e06b699921879793679"}
Sep 15 20:59:10.339065 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:10.338981    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vf58t" podStartSLOduration=67.172709677 podStartE2EDuration="1m8.338956373s" totalImagesPullingTime="1.166246696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:59:10.337907502 +0000 UTC m=+68.828553235" watchObservedRunningTime="2026-09-15 20:59:10.338956373 +0000 UTC m=+68.829602106"
Sep 15 20:59:10.655636 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:10.655541    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a2612f1-6b01-421d-b105-f4a20b6356f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:10.655636 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:10.655592    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a2612f1-6b01-421d-b105-f4a20b6356f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:10.655882 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:59:10.655689    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 20:59:10.655882 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:59:10.655729    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 20:59:10.655882 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:59:10.655772    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a2612f1-6b01-421d-b105-f4a20b6356f7-kube-rbac-proxy-cm podName:6a2612f1-6b01-421d-b105-f4a20b6356f7 nodeName:}" failed. No retries permitted until 2026-09-15 20:59:12.655751952 +0000 UTC m=+71.146397692 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6a2612f1-6b01-421d-b105-f4a20b6356f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6rn7r" (UID: "6a2612f1-6b01-421d-b105-f4a20b6356f7") : configmap "kube-rbac-proxy" not found
Sep 15 20:59:10.655882 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:59:10.655794    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2612f1-6b01-421d-b105-f4a20b6356f7-insights-runtime-extractor-tls podName:6a2612f1-6b01-421d-b105-f4a20b6356f7 nodeName:}" failed. No retries permitted until 2026-09-15 20:59:12.655783963 +0000 UTC m=+71.146429681 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6a2612f1-6b01-421d-b105-f4a20b6356f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6rn7r" (UID: "6a2612f1-6b01-421d-b105-f4a20b6356f7") : secret "insights-runtime-extractor-tls" not found
Sep 15 20:59:10.979497 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:10.979470    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nwspf_cb9c01cf-5113-402e-a9ff-8f177ab7f018/dns-node-resolver/0.log"
Sep 15 20:59:11.780004 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:11.779972    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8d6b7655-2cwn2_e9273ede-dc5b-4757-a016-61ed4a57e043/registry/0.log"
Sep 15 20:59:12.298646 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.298616    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ftppc"
Sep 15 20:59:12.380862 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.380836    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xkmbd_71567bed-fb87-4bb0-9833-a30ff9a16ef9/node-ca/0.log"
Sep 15 20:59:12.671646 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.671607    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a2612f1-6b01-421d-b105-f4a20b6356f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:12.671829 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.671656    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a2612f1-6b01-421d-b105-f4a20b6356f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:12.671829 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:59:12.671752    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 20:59:12.671829 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:59:12.671824    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 20:59:12.671959 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:59:12.671838    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a2612f1-6b01-421d-b105-f4a20b6356f7-kube-rbac-proxy-cm podName:6a2612f1-6b01-421d-b105-f4a20b6356f7 nodeName:}" failed. No retries permitted until 2026-09-15 20:59:16.671814784 +0000 UTC m=+75.162460553 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6a2612f1-6b01-421d-b105-f4a20b6356f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6rn7r" (UID: "6a2612f1-6b01-421d-b105-f4a20b6356f7") : configmap "kube-rbac-proxy" not found
Sep 15 20:59:12.671959 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:59:12.671880    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2612f1-6b01-421d-b105-f4a20b6356f7-insights-runtime-extractor-tls podName:6a2612f1-6b01-421d-b105-f4a20b6356f7 nodeName:}" failed. No retries permitted until 2026-09-15 20:59:16.671862773 +0000 UTC m=+75.162508485 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6a2612f1-6b01-421d-b105-f4a20b6356f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6rn7r" (UID: "6a2612f1-6b01-421d-b105-f4a20b6356f7") : secret "insights-runtime-extractor-tls" not found
Sep 15 20:59:12.726599 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.726561    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-jbx4k"]
Sep 15 20:59:12.729769 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.729743    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-jbx4k"
Sep 15 20:59:12.732651 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.732626    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 20:59:12.732800 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.732664    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 15 20:59:12.734161 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.734135    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 20:59:12.734275 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.734174    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 20:59:12.734275 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.734200    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 15 20:59:12.734472 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.734451    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rhpmd\""
Sep 15 20:59:12.737596 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.737572    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-jbx4k"]
Sep 15 20:59:12.772625 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.772582    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/694bdf81-364f-4d17-90e2-ca9fe991ce11-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-jbx4k\" (UID: \"694bdf81-364f-4d17-90e2-ca9fe991ce11\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jbx4k"
Sep 15 20:59:12.772823 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.772635    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj866\" (UniqueName: \"kubernetes.io/projected/694bdf81-364f-4d17-90e2-ca9fe991ce11-kube-api-access-zj866\") pod \"prometheus-operator-68bcd7f987-jbx4k\" (UID: \"694bdf81-364f-4d17-90e2-ca9fe991ce11\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jbx4k"
Sep 15 20:59:12.772823 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.772718    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/694bdf81-364f-4d17-90e2-ca9fe991ce11-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-jbx4k\" (UID: \"694bdf81-364f-4d17-90e2-ca9fe991ce11\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jbx4k"
Sep 15 20:59:12.772823 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.772759    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/694bdf81-364f-4d17-90e2-ca9fe991ce11-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-jbx4k\" (UID: \"694bdf81-364f-4d17-90e2-ca9fe991ce11\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jbx4k"
Sep 15 20:59:12.873954 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.873918    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/694bdf81-364f-4d17-90e2-ca9fe991ce11-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-jbx4k\" (UID: \"694bdf81-364f-4d17-90e2-ca9fe991ce11\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jbx4k"
Sep 15 20:59:12.873954 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.873960    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/694bdf81-364f-4d17-90e2-ca9fe991ce11-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-jbx4k\" (UID: \"694bdf81-364f-4d17-90e2-ca9fe991ce11\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jbx4k"
Sep 15 20:59:12.874542 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.874037    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/694bdf81-364f-4d17-90e2-ca9fe991ce11-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-jbx4k\" (UID: \"694bdf81-364f-4d17-90e2-ca9fe991ce11\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jbx4k"
Sep 15 20:59:12.874542 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.874088    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj866\" (UniqueName: \"kubernetes.io/projected/694bdf81-364f-4d17-90e2-ca9fe991ce11-kube-api-access-zj866\") pod \"prometheus-operator-68bcd7f987-jbx4k\" (UID: \"694bdf81-364f-4d17-90e2-ca9fe991ce11\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jbx4k"
Sep 15 20:59:12.874900 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.874866    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/694bdf81-364f-4d17-90e2-ca9fe991ce11-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-jbx4k\" (UID: \"694bdf81-364f-4d17-90e2-ca9fe991ce11\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jbx4k"
Sep 15 20:59:12.876752 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.876723    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/694bdf81-364f-4d17-90e2-ca9fe991ce11-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-jbx4k\" (UID: \"694bdf81-364f-4d17-90e2-ca9fe991ce11\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jbx4k"
Sep 15 20:59:12.876883 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.876861    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/694bdf81-364f-4d17-90e2-ca9fe991ce11-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-jbx4k\" (UID: \"694bdf81-364f-4d17-90e2-ca9fe991ce11\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jbx4k"
Sep 15 20:59:12.885090 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.885030    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj866\" (UniqueName: \"kubernetes.io/projected/694bdf81-364f-4d17-90e2-ca9fe991ce11-kube-api-access-zj866\") pod \"prometheus-operator-68bcd7f987-jbx4k\" (UID: \"694bdf81-364f-4d17-90e2-ca9fe991ce11\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jbx4k"
Sep 15 20:59:12.978965 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:12.978864    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m4ncl_6dc59ea6-8d86-456d-b250-e2d0fc7a2d10/serve-healthcheck-canary/0.log"
Sep 15 20:59:13.041830 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:13.041784    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-jbx4k"
Sep 15 20:59:13.340378 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:13.340341    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z2vkz" event={"ID":"e86303f1-1359-459e-910a-31c3a4256728","Type":"ContainerStarted","Data":"f146bb89c74e9d177d21f802684195eace1aabfafad1fde957c613280059ed35"}
Sep 15 20:59:13.340588 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:13.340405    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 20:59:13.341927 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:13.341901    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qphq9" event={"ID":"932cce37-6a6a-4968-8cbe-d163e1da16f0","Type":"ContainerStarted","Data":"1e0372193126d8ae99aa5934b19cd370e2d0da00db4d9708e23a44bf60c192f9"}
Sep 15 20:59:13.357522 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:13.357470    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z2vkz" podStartSLOduration=66.571230508 podStartE2EDuration="1m11.357453965s" totalImagesPullingTime="4.786223457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:59:13.356601712 +0000 UTC m=+71.847247458" watchObservedRunningTime="2026-09-15 20:59:13.357453965 +0000 UTC m=+71.848099696"
Sep 15 20:59:13.371380 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:13.371331    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qphq9" podStartSLOduration=66.302746778 podStartE2EDuration="1m11.371314133s" totalImagesPullingTime="5.068567355s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:59:13.370970978 +0000 UTC m=+71.861616710" watchObservedRunningTime="2026-09-15 20:59:13.371314133 +0000 UTC m=+71.861959864"
Sep 15 20:59:13.423488 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:13.423456    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-jbx4k"]
Sep 15 20:59:13.428672 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:59:13.428639    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694bdf81_364f_4d17_90e2_ca9fe991ce11.slice/crio-3ef829c5866e71131f14e57b1cde654c5c41c7657b2b9662276a06f4ef7f9f99 WatchSource:0}: Error finding container 3ef829c5866e71131f14e57b1cde654c5c41c7657b2b9662276a06f4ef7f9f99: Status 404 returned error can't find the container with id 3ef829c5866e71131f14e57b1cde654c5c41c7657b2b9662276a06f4ef7f9f99
Sep 15 20:59:14.347241 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:14.347201    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-jbx4k" event={"ID":"694bdf81-364f-4d17-90e2-ca9fe991ce11","Type":"ContainerStarted","Data":"3ef829c5866e71131f14e57b1cde654c5c41c7657b2b9662276a06f4ef7f9f99"}
Sep 15 20:59:15.351920 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:15.351805    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-jbx4k" event={"ID":"694bdf81-364f-4d17-90e2-ca9fe991ce11","Type":"ContainerStarted","Data":"3321a5438d4ea72b1065b9be6b881a3c9704515d80311a5f95d6b86e5e9b41ea"}
Sep 15 20:59:15.351920 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:15.351897    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-jbx4k" event={"ID":"694bdf81-364f-4d17-90e2-ca9fe991ce11","Type":"ContainerStarted","Data":"0f1bbb1662c75401f8e832c72baf8d34ac69ed7d8e5f00731aea40fd1394fd71"}
Sep 15 20:59:15.369335 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:15.369270    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-jbx4k" podStartSLOduration=1.857091145 podStartE2EDuration="3.369253802s" totalImagesPullingTime="1.512162657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:59:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:59:15.368940582 +0000 UTC m=+73.859586316" watchObservedRunningTime="2026-09-15 20:59:15.369253802 +0000 UTC m=+73.859899534"
Sep 15 20:59:16.704307 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:16.704246    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a2612f1-6b01-421d-b105-f4a20b6356f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:16.704307 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:16.704303    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a2612f1-6b01-421d-b105-f4a20b6356f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:16.704719 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:59:16.704370    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 20:59:16.704719 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:59:16.704420    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 20:59:16.704719 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:59:16.704438    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a2612f1-6b01-421d-b105-f4a20b6356f7-kube-rbac-proxy-cm podName:6a2612f1-6b01-421d-b105-f4a20b6356f7 nodeName:}" failed. No retries permitted until 2026-09-15 20:59:24.704422613 +0000 UTC m=+83.195068323 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6a2612f1-6b01-421d-b105-f4a20b6356f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6rn7r" (UID: "6a2612f1-6b01-421d-b105-f4a20b6356f7") : configmap "kube-rbac-proxy" not found
Sep 15 20:59:16.704719 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:59:16.704466    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2612f1-6b01-421d-b105-f4a20b6356f7-insights-runtime-extractor-tls podName:6a2612f1-6b01-421d-b105-f4a20b6356f7 nodeName:}" failed. No retries permitted until 2026-09-15 20:59:24.704454456 +0000 UTC m=+83.195100180 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6a2612f1-6b01-421d-b105-f4a20b6356f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6rn7r" (UID: "6a2612f1-6b01-421d-b105-f4a20b6356f7") : secret "insights-runtime-extractor-tls" not found
Sep 15 20:59:17.071006 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.070923    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-c86pl"]
Sep 15 20:59:17.074029 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.074009    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c86pl"
Sep 15 20:59:17.077126 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.077100    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 15 20:59:17.077263 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.077097    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 15 20:59:17.077263 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.077174    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-kzljg\""
Sep 15 20:59:17.082758 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.082732    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-c86pl"]
Sep 15 20:59:17.104107 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.104071    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-b8f5c"]
Sep 15 20:59:17.107197 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.107169    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.107374 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.107351    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97qkw\" (UniqueName: \"kubernetes.io/projected/3f77ec77-c110-445c-82d2-229051a5adc6-kube-api-access-97qkw\") pod \"openshift-state-metrics-67dcf8c979-c86pl\" (UID: \"3f77ec77-c110-445c-82d2-229051a5adc6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c86pl"
Sep 15 20:59:17.107443 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.107396    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f77ec77-c110-445c-82d2-229051a5adc6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-c86pl\" (UID: \"3f77ec77-c110-445c-82d2-229051a5adc6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c86pl"
Sep 15 20:59:17.107487 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.107446    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f77ec77-c110-445c-82d2-229051a5adc6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-c86pl\" (UID: \"3f77ec77-c110-445c-82d2-229051a5adc6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c86pl"
Sep 15 20:59:17.107534 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.107487    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f77ec77-c110-445c-82d2-229051a5adc6-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-c86pl\" (UID: \"3f77ec77-c110-445c-82d2-229051a5adc6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c86pl"
Sep 15 20:59:17.109936 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.109918    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 20:59:17.110034 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.109913    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 20:59:17.110106 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.110038    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 20:59:17.110158 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.110137    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pcj6b\""
Sep 15 20:59:17.208314 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.208275    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f77ec77-c110-445c-82d2-229051a5adc6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-c86pl\" (UID: \"3f77ec77-c110-445c-82d2-229051a5adc6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c86pl"
Sep 15 20:59:17.208314 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.208316    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcw6q\" (UniqueName: \"kubernetes.io/projected/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-kube-api-access-rcw6q\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.208566 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.208337    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f77ec77-c110-445c-82d2-229051a5adc6-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-c86pl\" (UID: \"3f77ec77-c110-445c-82d2-229051a5adc6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c86pl"
Sep 15 20:59:17.208566 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.208354    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.208566 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.208422    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-node-exporter-wtmp\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.208566 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:59:17.208440    2558 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 15 20:59:17.208566 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.208477    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-metrics-client-ca\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.208566 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:59:17.208496    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f77ec77-c110-445c-82d2-229051a5adc6-openshift-state-metrics-tls podName:3f77ec77-c110-445c-82d2-229051a5adc6 nodeName:}" failed. No retries permitted until 2026-09-15 20:59:17.708480227 +0000 UTC m=+76.199125955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/3f77ec77-c110-445c-82d2-229051a5adc6-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-c86pl" (UID: "3f77ec77-c110-445c-82d2-229051a5adc6") : secret "openshift-state-metrics-tls" not found
Sep 15 20:59:17.208566 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.208559    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-sys\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.208830 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.208588    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-root\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.208830 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.208616    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f77ec77-c110-445c-82d2-229051a5adc6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-c86pl\" (UID: \"3f77ec77-c110-445c-82d2-229051a5adc6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c86pl"
Sep 15 20:59:17.208830 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.208653    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-node-exporter-textfile\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.208830 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.208685    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-node-exporter-tls\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.208830 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.208723    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-node-exporter-accelerators-collector-config\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.208830 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.208754    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97qkw\" (UniqueName: \"kubernetes.io/projected/3f77ec77-c110-445c-82d2-229051a5adc6-kube-api-access-97qkw\") pod \"openshift-state-metrics-67dcf8c979-c86pl\" (UID: \"3f77ec77-c110-445c-82d2-229051a5adc6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c86pl"
Sep 15 20:59:17.209225 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.209206    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f77ec77-c110-445c-82d2-229051a5adc6-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-c86pl\" (UID: \"3f77ec77-c110-445c-82d2-229051a5adc6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c86pl"
Sep 15 20:59:17.210972 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.210954    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f77ec77-c110-445c-82d2-229051a5adc6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-c86pl\" (UID: \"3f77ec77-c110-445c-82d2-229051a5adc6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c86pl"
Sep 15 20:59:17.217244 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.217217    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97qkw\" (UniqueName: \"kubernetes.io/projected/3f77ec77-c110-445c-82d2-229051a5adc6-kube-api-access-97qkw\") pod \"openshift-state-metrics-67dcf8c979-c86pl\" (UID: \"3f77ec77-c110-445c-82d2-229051a5adc6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c86pl"
Sep 15 20:59:17.309768 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.309728    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-node-exporter-wtmp\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.309768 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.309774    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-metrics-client-ca\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.309970 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.309792    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-sys\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.309970 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.309808    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-root\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.309970 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.309857    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-node-exporter-textfile\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.309970 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.309874    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-node-exporter-tls\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.309970 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.309877    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-sys\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.309970 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.309897    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-node-exporter-accelerators-collector-config\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.309970 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.309960    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-node-exporter-wtmp\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.310344 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.309975    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcw6q\" (UniqueName: \"kubernetes.io/projected/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-kube-api-access-rcw6q\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.310344 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.309978    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-root\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.310344 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.310027    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.310344 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.310252    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-node-exporter-textfile\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.310586 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.310564    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-metrics-client-ca\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.310669 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.310644    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-node-exporter-accelerators-collector-config\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.312534 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.312496    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-node-exporter-tls\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.312652 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.312567    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.321799 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.321733    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcw6q\" (UniqueName: \"kubernetes.io/projected/aa5b7933-e78a-4ca1-b46e-c101a3f025c5-kube-api-access-rcw6q\") pod \"node-exporter-b8f5c\" (UID: \"aa5b7933-e78a-4ca1-b46e-c101a3f025c5\") " pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.416063 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.416019    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b8f5c"
Sep 15 20:59:17.427401 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:59:17.427372    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5b7933_e78a_4ca1_b46e_c101a3f025c5.slice/crio-37cc9ae2a22bfd67bbf772d97848d495c53bbd2dffc85eab6ce29b00b9feaae2 WatchSource:0}: Error finding container 37cc9ae2a22bfd67bbf772d97848d495c53bbd2dffc85eab6ce29b00b9feaae2: Status 404 returned error can't find the container with id 37cc9ae2a22bfd67bbf772d97848d495c53bbd2dffc85eab6ce29b00b9feaae2
Sep 15 20:59:17.714041 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.714007    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f77ec77-c110-445c-82d2-229051a5adc6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-c86pl\" (UID: \"3f77ec77-c110-445c-82d2-229051a5adc6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c86pl"
Sep 15 20:59:17.716493 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.716469    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f77ec77-c110-445c-82d2-229051a5adc6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-c86pl\" (UID: \"3f77ec77-c110-445c-82d2-229051a5adc6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c86pl"
Sep 15 20:59:17.983816 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:17.983704    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c86pl"
Sep 15 20:59:18.148334 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.148300    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 20:59:18.153254 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.153223    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.156110 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.156084    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 15 20:59:18.156317 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.156116    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 15 20:59:18.156317 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.156298    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 15 20:59:18.156521 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.156340    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8nc9l\""
Sep 15 20:59:18.156521 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.156369    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 15 20:59:18.156521 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.156476    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 15 20:59:18.156678 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.156533    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 15 20:59:18.156678 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.156579    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 15 20:59:18.156748 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.156711    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 15 20:59:18.156782 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.156772    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 15 20:59:18.170314 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.170284    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 20:59:18.218847 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.218812    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d15d6dda-588c-4b50-bd5b-0f4401619f0d-config-volume\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.219071 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.218855    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d15d6dda-588c-4b50-bd5b-0f4401619f0d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.219071 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.218939    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d15d6dda-588c-4b50-bd5b-0f4401619f0d-web-config\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.219071 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.218983    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d15d6dda-588c-4b50-bd5b-0f4401619f0d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.219071 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.219070    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d15d6dda-588c-4b50-bd5b-0f4401619f0d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.219500 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.219096    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d15d6dda-588c-4b50-bd5b-0f4401619f0d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.219500 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.219126    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d15d6dda-588c-4b50-bd5b-0f4401619f0d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.219500 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.219174    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d15d6dda-588c-4b50-bd5b-0f4401619f0d-config-out\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.219500 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.219211    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d15d6dda-588c-4b50-bd5b-0f4401619f0d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.219500 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.219319    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d15d6dda-588c-4b50-bd5b-0f4401619f0d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.219500 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.219388    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d15d6dda-588c-4b50-bd5b-0f4401619f0d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.219500 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.219414    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d15d6dda-588c-4b50-bd5b-0f4401619f0d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.219500 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.219478    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whcjc\" (UniqueName: \"kubernetes.io/projected/d15d6dda-588c-4b50-bd5b-0f4401619f0d-kube-api-access-whcjc\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.290294 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.290268    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-c86pl"]
Sep 15 20:59:18.292418 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:59:18.292392    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f77ec77_c110_445c_82d2_229051a5adc6.slice/crio-375ede4897bbe7a71d9fc3dc8e69d60a2b707972d7b0680b50db04eb1337e60d WatchSource:0}: Error finding container 375ede4897bbe7a71d9fc3dc8e69d60a2b707972d7b0680b50db04eb1337e60d: Status 404 returned error can't find the container with id 375ede4897bbe7a71d9fc3dc8e69d60a2b707972d7b0680b50db04eb1337e60d
Sep 15 20:59:18.320290 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.320246    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whcjc\" (UniqueName: \"kubernetes.io/projected/d15d6dda-588c-4b50-bd5b-0f4401619f0d-kube-api-access-whcjc\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.320479 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.320310    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d15d6dda-588c-4b50-bd5b-0f4401619f0d-config-volume\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.320479 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.320334    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d15d6dda-588c-4b50-bd5b-0f4401619f0d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.320479 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.320370    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d15d6dda-588c-4b50-bd5b-0f4401619f0d-web-config\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.320479 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.320396    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d15d6dda-588c-4b50-bd5b-0f4401619f0d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.320479 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.320419    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d15d6dda-588c-4b50-bd5b-0f4401619f0d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.320479 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.320443    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d15d6dda-588c-4b50-bd5b-0f4401619f0d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.320479 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.320473    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d15d6dda-588c-4b50-bd5b-0f4401619f0d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.320833 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.320518    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d15d6dda-588c-4b50-bd5b-0f4401619f0d-config-out\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.320833 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.320542    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d15d6dda-588c-4b50-bd5b-0f4401619f0d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.320833 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.320572    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d15d6dda-588c-4b50-bd5b-0f4401619f0d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.320833 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.320594    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d15d6dda-588c-4b50-bd5b-0f4401619f0d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.320833 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.320620    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d15d6dda-588c-4b50-bd5b-0f4401619f0d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.321092 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.320931    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d15d6dda-588c-4b50-bd5b-0f4401619f0d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.321950 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.321921    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d15d6dda-588c-4b50-bd5b-0f4401619f0d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.323859 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.323558    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d15d6dda-588c-4b50-bd5b-0f4401619f0d-config-out\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.323859 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.323814    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d15d6dda-588c-4b50-bd5b-0f4401619f0d-config-volume\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.324305 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.324250    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d15d6dda-588c-4b50-bd5b-0f4401619f0d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.324305 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.324291    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d15d6dda-588c-4b50-bd5b-0f4401619f0d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.324660 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.324638    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d15d6dda-588c-4b50-bd5b-0f4401619f0d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.324867 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.324844    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d15d6dda-588c-4b50-bd5b-0f4401619f0d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.325105 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.325082    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d15d6dda-588c-4b50-bd5b-0f4401619f0d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.325271 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.325251    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d15d6dda-588c-4b50-bd5b-0f4401619f0d-web-config\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.325962 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.325938    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d15d6dda-588c-4b50-bd5b-0f4401619f0d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.326246 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.326227    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d15d6dda-588c-4b50-bd5b-0f4401619f0d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.328856 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.328830    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whcjc\" (UniqueName: \"kubernetes.io/projected/d15d6dda-588c-4b50-bd5b-0f4401619f0d-kube-api-access-whcjc\") pod \"alertmanager-main-0\" (UID: \"d15d6dda-588c-4b50-bd5b-0f4401619f0d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.363492 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.363454    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c86pl" event={"ID":"3f77ec77-c110-445c-82d2-229051a5adc6","Type":"ContainerStarted","Data":"375ede4897bbe7a71d9fc3dc8e69d60a2b707972d7b0680b50db04eb1337e60d"}
Sep 15 20:59:18.364801 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.364773    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b8f5c" event={"ID":"aa5b7933-e78a-4ca1-b46e-c101a3f025c5","Type":"ContainerStarted","Data":"37cc9ae2a22bfd67bbf772d97848d495c53bbd2dffc85eab6ce29b00b9feaae2"}
Sep 15 20:59:18.462845 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.462804    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 20:59:18.749251 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:18.749222    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 20:59:18.752193 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:59:18.752161    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15d6dda_588c_4b50_bd5b_0f4401619f0d.slice/crio-7a010573070bba1ef8967be0473af6ae09a17c285735b15db8a42d6eaae35426 WatchSource:0}: Error finding container 7a010573070bba1ef8967be0473af6ae09a17c285735b15db8a42d6eaae35426: Status 404 returned error can't find the container with id 7a010573070bba1ef8967be0473af6ae09a17c285735b15db8a42d6eaae35426
Sep 15 20:59:19.370391 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:19.370300    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c86pl" event={"ID":"3f77ec77-c110-445c-82d2-229051a5adc6","Type":"ContainerStarted","Data":"d85cb0cadaca9fa78b72515f87040a1894adc0f6139d2e54f2845d1b5f4d59eb"}
Sep 15 20:59:19.370391 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:19.370366    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c86pl" event={"ID":"3f77ec77-c110-445c-82d2-229051a5adc6","Type":"ContainerStarted","Data":"aca2f8972c75c169d964e70190909f74efc3e31231eb1320b1939a6c24166d12"}
Sep 15 20:59:19.372268 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:19.372234    2558 generic.go:356] "Generic (PLEG): container finished" podID="aa5b7933-e78a-4ca1-b46e-c101a3f025c5" containerID="d4e0515ea1c1661c4b8b8ed52233416d8ee10b76c02809708769ee3c31e3b89d" exitCode=0
Sep 15 20:59:19.372427 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:19.372324    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b8f5c" event={"ID":"aa5b7933-e78a-4ca1-b46e-c101a3f025c5","Type":"ContainerDied","Data":"d4e0515ea1c1661c4b8b8ed52233416d8ee10b76c02809708769ee3c31e3b89d"}
Sep 15 20:59:19.374884 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:19.374856    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d15d6dda-588c-4b50-bd5b-0f4401619f0d","Type":"ContainerStarted","Data":"7a010573070bba1ef8967be0473af6ae09a17c285735b15db8a42d6eaae35426"}
Sep 15 20:59:20.289798 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:20.289760    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8d6b7655-2cwn2"
Sep 15 20:59:20.380891 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:20.380854    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c86pl" event={"ID":"3f77ec77-c110-445c-82d2-229051a5adc6","Type":"ContainerStarted","Data":"c8e901cfce55446818712b59afe2809831fb4b875afa729663bc590efb51ea5c"}
Sep 15 20:59:20.383279 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:20.383246    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b8f5c" event={"ID":"aa5b7933-e78a-4ca1-b46e-c101a3f025c5","Type":"ContainerStarted","Data":"979266483dc5cecb4381a1a8b37ebdfab1370a68a555b787b330e12f1cf7c991"}
Sep 15 20:59:20.383414 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:20.383304    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b8f5c" event={"ID":"aa5b7933-e78a-4ca1-b46e-c101a3f025c5","Type":"ContainerStarted","Data":"53f285178f8270ad8cd3dc2f1b6d6f76d3342238f44713d96a38704fc044bf15"}
Sep 15 20:59:20.385011 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:20.384983    2558 generic.go:356] "Generic (PLEG): container finished" podID="d15d6dda-588c-4b50-bd5b-0f4401619f0d" containerID="6514d826ebe69024d5254a8a5b7173cf06f91d3fc27605992e2c27f44fec69ca" exitCode=0
Sep 15 20:59:20.385161 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:20.385019    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d15d6dda-588c-4b50-bd5b-0f4401619f0d","Type":"ContainerDied","Data":"6514d826ebe69024d5254a8a5b7173cf06f91d3fc27605992e2c27f44fec69ca"}
Sep 15 20:59:20.398959 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:20.398906    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c86pl" podStartSLOduration=1.885960791 podStartE2EDuration="3.398882416s" totalImagesPullingTime="1.512921625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:59:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:59:20.397666573 +0000 UTC m=+78.888312304" watchObservedRunningTime="2026-09-15 20:59:20.398882416 +0000 UTC m=+78.889528150"
Sep 15 20:59:20.418512 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:20.418436    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-b8f5c" podStartSLOduration=1.742384803 podStartE2EDuration="3.418413874s" totalImagesPullingTime="1.609836551s" totalInitContainerRuntime="66.19252ms" isStatefulPod=true podCreationTimestamp="2026-09-15 20:59:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:59:20.416692681 +0000 UTC m=+78.907338414" watchObservedRunningTime="2026-09-15 20:59:20.418413874 +0000 UTC m=+78.909059606"
Sep 15 20:59:21.419781 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.419739    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-647b67747-469rh"]
Sep 15 20:59:21.424467 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.424441    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:21.428719 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.428694    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 15 20:59:21.428888 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.428817    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 15 20:59:21.428888 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.428873    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 15 20:59:21.428988 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.428817    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-5br8v\""
Sep 15 20:59:21.429198 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.429182    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-lp92294ch1q2\""
Sep 15 20:59:21.429266 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.429182    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 15 20:59:21.433747 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.433719    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-647b67747-469rh"]
Sep 15 20:59:21.548919 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.548879    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/12d35a75-1573-447f-a27b-b337d5069891-client-ca-bundle\") pod \"metrics-server-647b67747-469rh\" (UID: \"12d35a75-1573-447f-a27b-b337d5069891\") " pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:21.549130 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.548932    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/12d35a75-1573-447f-a27b-b337d5069891-secret-metrics-server-client-certs\") pod \"metrics-server-647b67747-469rh\" (UID: \"12d35a75-1573-447f-a27b-b337d5069891\") " pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:21.549130 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.548986    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12d35a75-1573-447f-a27b-b337d5069891-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-647b67747-469rh\" (UID: \"12d35a75-1573-447f-a27b-b337d5069891\") " pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:21.549130 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.549009    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdzv6\" (UniqueName: \"kubernetes.io/projected/12d35a75-1573-447f-a27b-b337d5069891-kube-api-access-jdzv6\") pod \"metrics-server-647b67747-469rh\" (UID: \"12d35a75-1573-447f-a27b-b337d5069891\") " pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:21.549130 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.549040    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/12d35a75-1573-447f-a27b-b337d5069891-secret-metrics-server-tls\") pod \"metrics-server-647b67747-469rh\" (UID: \"12d35a75-1573-447f-a27b-b337d5069891\") " pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:21.549130 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.549074    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/12d35a75-1573-447f-a27b-b337d5069891-audit-log\") pod \"metrics-server-647b67747-469rh\" (UID: \"12d35a75-1573-447f-a27b-b337d5069891\") " pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:21.549130 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.549114    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/12d35a75-1573-447f-a27b-b337d5069891-metrics-server-audit-profiles\") pod \"metrics-server-647b67747-469rh\" (UID: \"12d35a75-1573-447f-a27b-b337d5069891\") " pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:21.650535 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.650499    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/12d35a75-1573-447f-a27b-b337d5069891-client-ca-bundle\") pod \"metrics-server-647b67747-469rh\" (UID: \"12d35a75-1573-447f-a27b-b337d5069891\") " pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:21.650677 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.650551    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/12d35a75-1573-447f-a27b-b337d5069891-secret-metrics-server-client-certs\") pod \"metrics-server-647b67747-469rh\" (UID: \"12d35a75-1573-447f-a27b-b337d5069891\") " pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:21.650677 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.650574    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12d35a75-1573-447f-a27b-b337d5069891-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-647b67747-469rh\" (UID: \"12d35a75-1573-447f-a27b-b337d5069891\") " pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:21.650677 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.650637    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdzv6\" (UniqueName: \"kubernetes.io/projected/12d35a75-1573-447f-a27b-b337d5069891-kube-api-access-jdzv6\") pod \"metrics-server-647b67747-469rh\" (UID: \"12d35a75-1573-447f-a27b-b337d5069891\") " pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:21.650830 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.650687    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/12d35a75-1573-447f-a27b-b337d5069891-secret-metrics-server-tls\") pod \"metrics-server-647b67747-469rh\" (UID: \"12d35a75-1573-447f-a27b-b337d5069891\") " pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:21.650830 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.650715    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/12d35a75-1573-447f-a27b-b337d5069891-audit-log\") pod \"metrics-server-647b67747-469rh\" (UID: \"12d35a75-1573-447f-a27b-b337d5069891\") " pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:21.650830 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.650773    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/12d35a75-1573-447f-a27b-b337d5069891-metrics-server-audit-profiles\") pod \"metrics-server-647b67747-469rh\" (UID: \"12d35a75-1573-447f-a27b-b337d5069891\") " pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:21.651364 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.651312    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/12d35a75-1573-447f-a27b-b337d5069891-audit-log\") pod \"metrics-server-647b67747-469rh\" (UID: \"12d35a75-1573-447f-a27b-b337d5069891\") " pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:21.651503 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.651421    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12d35a75-1573-447f-a27b-b337d5069891-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-647b67747-469rh\" (UID: \"12d35a75-1573-447f-a27b-b337d5069891\") " pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:21.651803 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.651779    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/12d35a75-1573-447f-a27b-b337d5069891-metrics-server-audit-profiles\") pod \"metrics-server-647b67747-469rh\" (UID: \"12d35a75-1573-447f-a27b-b337d5069891\") " pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:21.653179 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.653144    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/12d35a75-1573-447f-a27b-b337d5069891-secret-metrics-server-client-certs\") pod \"metrics-server-647b67747-469rh\" (UID: \"12d35a75-1573-447f-a27b-b337d5069891\") " pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:21.653265 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.653235    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/12d35a75-1573-447f-a27b-b337d5069891-secret-metrics-server-tls\") pod \"metrics-server-647b67747-469rh\" (UID: \"12d35a75-1573-447f-a27b-b337d5069891\") " pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:21.653265 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.653238    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/12d35a75-1573-447f-a27b-b337d5069891-client-ca-bundle\") pod \"metrics-server-647b67747-469rh\" (UID: \"12d35a75-1573-447f-a27b-b337d5069891\") " pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:21.658919 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.658894    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdzv6\" (UniqueName: \"kubernetes.io/projected/12d35a75-1573-447f-a27b-b337d5069891-kube-api-access-jdzv6\") pod \"metrics-server-647b67747-469rh\" (UID: \"12d35a75-1573-447f-a27b-b337d5069891\") " pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:21.736970 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.736890    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:21.891094 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.891030    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-rwhsp"]
Sep 15 20:59:21.896338 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.896298    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rwhsp"
Sep 15 20:59:21.899344 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.899320    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 15 20:59:21.899476 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.899332    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-9z5ft\""
Sep 15 20:59:21.904571 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.904541    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-rwhsp"]
Sep 15 20:59:21.953020 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.952987    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16c1df55-089f-4b2e-bba5-39c25ea6bd1f-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-rwhsp\" (UID: \"16c1df55-089f-4b2e-bba5-39c25ea6bd1f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rwhsp"
Sep 15 20:59:21.953229 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:21.953168    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-672xx\" (UniqueName: \"kubernetes.io/projected/16c1df55-089f-4b2e-bba5-39c25ea6bd1f-kube-api-access-672xx\") pod \"monitoring-plugin-5d85778b9d-rwhsp\" (UID: \"16c1df55-089f-4b2e-bba5-39c25ea6bd1f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rwhsp"
Sep 15 20:59:22.053657 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:22.053501    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-672xx\" (UniqueName: \"kubernetes.io/projected/16c1df55-089f-4b2e-bba5-39c25ea6bd1f-kube-api-access-672xx\") pod \"monitoring-plugin-5d85778b9d-rwhsp\" (UID: \"16c1df55-089f-4b2e-bba5-39c25ea6bd1f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rwhsp"
Sep 15 20:59:22.053657 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:22.053581    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16c1df55-089f-4b2e-bba5-39c25ea6bd1f-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-rwhsp\" (UID: \"16c1df55-089f-4b2e-bba5-39c25ea6bd1f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rwhsp"
Sep 15 20:59:22.053810 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:59:22.053750    2558 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 15 20:59:22.053868 ip-10-0-142-207 kubenswrapper[2558]: E0915 20:59:22.053812    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16c1df55-089f-4b2e-bba5-39c25ea6bd1f-monitoring-plugin-cert podName:16c1df55-089f-4b2e-bba5-39c25ea6bd1f nodeName:}" failed. No retries permitted until 2026-09-15 20:59:22.553790694 +0000 UTC m=+81.044436406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/16c1df55-089f-4b2e-bba5-39c25ea6bd1f-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-rwhsp" (UID: "16c1df55-089f-4b2e-bba5-39c25ea6bd1f") : secret "monitoring-plugin-cert" not found
Sep 15 20:59:22.063326 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:22.063296    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-672xx\" (UniqueName: \"kubernetes.io/projected/16c1df55-089f-4b2e-bba5-39c25ea6bd1f-kube-api-access-672xx\") pod \"monitoring-plugin-5d85778b9d-rwhsp\" (UID: \"16c1df55-089f-4b2e-bba5-39c25ea6bd1f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rwhsp"
Sep 15 20:59:22.251147 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:22.251040    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-647b67747-469rh"]
Sep 15 20:59:22.256261 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:59:22.256228    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d35a75_1573_447f_a27b_b337d5069891.slice/crio-ac85e584298a47cdce982226bd88b5a6e4e6f4f492efa4d77ce103f36f6476b6 WatchSource:0}: Error finding container ac85e584298a47cdce982226bd88b5a6e4e6f4f492efa4d77ce103f36f6476b6: Status 404 returned error can't find the container with id ac85e584298a47cdce982226bd88b5a6e4e6f4f492efa4d77ce103f36f6476b6
Sep 15 20:59:22.394061 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:22.394005    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-647b67747-469rh" event={"ID":"12d35a75-1573-447f-a27b-b337d5069891","Type":"ContainerStarted","Data":"ac85e584298a47cdce982226bd88b5a6e4e6f4f492efa4d77ce103f36f6476b6"}
Sep 15 20:59:22.397295 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:22.397265    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d15d6dda-588c-4b50-bd5b-0f4401619f0d","Type":"ContainerStarted","Data":"bc9abbebf6f52717005d3b78831c40296b8ee7a8ca470d3a4e81ee0cb726b21e"}
Sep 15 20:59:22.397416 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:22.397325    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d15d6dda-588c-4b50-bd5b-0f4401619f0d","Type":"ContainerStarted","Data":"6a47f843547ea254774cfa2cc9cb698293a1d70868f64c2912bef9e14c1c54da"}
Sep 15 20:59:22.397416 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:22.397352    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d15d6dda-588c-4b50-bd5b-0f4401619f0d","Type":"ContainerStarted","Data":"0d056032583bb5a191ce6e54fcf9e118c9f3c45b896f5bc388c880592648ff20"}
Sep 15 20:59:22.397416 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:22.397377    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d15d6dda-588c-4b50-bd5b-0f4401619f0d","Type":"ContainerStarted","Data":"eb72d315ccfe484c9a9151af07de01c3a897bf01a9a16501f54c3da01a3eacd7"}
Sep 15 20:59:22.397416 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:22.397401    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d15d6dda-588c-4b50-bd5b-0f4401619f0d","Type":"ContainerStarted","Data":"c31dd8f1d90e2a839cb4700806082341759ce333c888b58bdb1191c8f1f11bd7"}
Sep 15 20:59:22.556693 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:22.556600    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16c1df55-089f-4b2e-bba5-39c25ea6bd1f-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-rwhsp\" (UID: \"16c1df55-089f-4b2e-bba5-39c25ea6bd1f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rwhsp"
Sep 15 20:59:22.559134 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:22.559108    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16c1df55-089f-4b2e-bba5-39c25ea6bd1f-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-rwhsp\" (UID: \"16c1df55-089f-4b2e-bba5-39c25ea6bd1f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rwhsp"
Sep 15 20:59:22.808763 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:22.808680    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rwhsp"
Sep 15 20:59:23.099416 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:23.099281    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-rwhsp"]
Sep 15 20:59:23.405144 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:23.405031    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rwhsp" event={"ID":"16c1df55-089f-4b2e-bba5-39c25ea6bd1f","Type":"ContainerStarted","Data":"42ad1fba491e431a3b8af47e542159c7cf97392c4d17ad69b330e410fa670740"}
Sep 15 20:59:24.413694 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:24.413559    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d15d6dda-588c-4b50-bd5b-0f4401619f0d","Type":"ContainerStarted","Data":"2d5eb6ebb7733be5cc315e6d066d95825df226fcae6126a85d4f360d60a97e32"}
Sep 15 20:59:24.415556 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:24.415518    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-647b67747-469rh" event={"ID":"12d35a75-1573-447f-a27b-b337d5069891","Type":"ContainerStarted","Data":"bb1003ce08a521e4f8fbd5aaeed85f010302e3549f27a28bb7d72485609341d5"}
Sep 15 20:59:24.442608 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:24.442512    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.731827815 podStartE2EDuration="6.442493604s" totalImagesPullingTime="3.691583371s" totalInitContainerRuntime="19.082418ms" isStatefulPod=false podCreationTimestamp="2026-09-15 20:59:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:59:24.440641913 +0000 UTC m=+82.931287644" watchObservedRunningTime="2026-09-15 20:59:24.442493604 +0000 UTC m=+82.933139326"
Sep 15 20:59:24.459946 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:24.459891    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-647b67747-469rh" podStartSLOduration=2.003088147 podStartE2EDuration="3.459871109s" totalImagesPullingTime="1.456782962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:59:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:59:24.458962178 +0000 UTC m=+82.949607911" watchObservedRunningTime="2026-09-15 20:59:24.459871109 +0000 UTC m=+82.950516852"
Sep 15 20:59:24.779243 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:24.779205    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a2612f1-6b01-421d-b105-f4a20b6356f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:24.779398 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:24.779248    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a2612f1-6b01-421d-b105-f4a20b6356f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:24.779813 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:24.779785    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a2612f1-6b01-421d-b105-f4a20b6356f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:24.781536 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:24.781515    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a2612f1-6b01-421d-b105-f4a20b6356f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6rn7r\" (UID: \"6a2612f1-6b01-421d-b105-f4a20b6356f7\") " pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:24.849867 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:24.849815    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6rn7r"
Sep 15 20:59:25.119032 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:25.119009    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6rn7r"]
Sep 15 20:59:25.121672 ip-10-0-142-207 kubenswrapper[2558]: W0915 20:59:25.121640    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a2612f1_6b01_421d_b105_f4a20b6356f7.slice/crio-03dbf27e8a3f24742c0c9e5921f2e26d31182c5465b6d3a4787a2db46a3207cd WatchSource:0}: Error finding container 03dbf27e8a3f24742c0c9e5921f2e26d31182c5465b6d3a4787a2db46a3207cd: Status 404 returned error can't find the container with id 03dbf27e8a3f24742c0c9e5921f2e26d31182c5465b6d3a4787a2db46a3207cd
Sep 15 20:59:25.421061 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:25.421028    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6rn7r" event={"ID":"6a2612f1-6b01-421d-b105-f4a20b6356f7","Type":"ContainerStarted","Data":"9c9acc52f92a3cd06173fd37ad77aac994fdcb45281ae2324496a91ed330c007"}
Sep 15 20:59:25.421456 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:25.421106    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6rn7r" event={"ID":"6a2612f1-6b01-421d-b105-f4a20b6356f7","Type":"ContainerStarted","Data":"03dbf27e8a3f24742c0c9e5921f2e26d31182c5465b6d3a4787a2db46a3207cd"}
Sep 15 20:59:25.422711 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:25.422689    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rwhsp" event={"ID":"16c1df55-089f-4b2e-bba5-39c25ea6bd1f","Type":"ContainerStarted","Data":"7de4d04f48e0c4700ecdda59772467b7ab39b3e6131a1c0584c2acb6828b1b09"}
Sep 15 20:59:25.439735 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:25.439687    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rwhsp" podStartSLOduration=2.866865148 podStartE2EDuration="4.439672563s" totalImagesPullingTime="1.572807415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:59:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:59:25.438663321 +0000 UTC m=+83.929309055" watchObservedRunningTime="2026-09-15 20:59:25.439672563 +0000 UTC m=+83.930318288"
Sep 15 20:59:26.428251 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:26.428210    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6rn7r" event={"ID":"6a2612f1-6b01-421d-b105-f4a20b6356f7","Type":"ContainerStarted","Data":"92d5b566647a13d9ba2fb67810029426439ebe83329128ac8fa683011c19a305"}
Sep 15 20:59:26.428633 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:26.428355    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rwhsp"
Sep 15 20:59:26.433352 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:26.433326    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rwhsp"
Sep 15 20:59:28.443016 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:28.442926    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6rn7r" event={"ID":"6a2612f1-6b01-421d-b105-f4a20b6356f7","Type":"ContainerStarted","Data":"de0a9400f1b3a63f8fba3b3ed84459c47dc51b7c5e340e4c0a524ae6de4addb4"}
Sep 15 20:59:28.465230 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:28.465175    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6rn7r" podStartSLOduration=17.517260799 podStartE2EDuration="20.46515774s" totalImagesPullingTime="2.947896941s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:59:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:59:28.463287097 +0000 UTC m=+86.953932829" watchObservedRunningTime="2026-09-15 20:59:28.46515774 +0000 UTC m=+86.955803472"
Sep 15 20:59:41.737308 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:41.737267    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:41.737729 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:41.737334    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 20:59:44.350074 ip-10-0-142-207 kubenswrapper[2558]: I0915 20:59:44.350028    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z2vkz"
Sep 15 21:00:01.743421 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:00:01.743389    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 21:00:01.747452 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:00:01.747423    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-647b67747-469rh"
Sep 15 21:01:19.725358 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:19.725280    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s"]
Sep 15 21:01:19.728775 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:19.728757    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s"
Sep 15 21:01:19.734084 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:19.734042    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 21:01:19.734241 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:19.734139    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6r6f2\""
Sep 15 21:01:19.735908 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:19.735890    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 21:01:19.745702 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:19.745671    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s"]
Sep 15 21:01:19.850914 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:19.850871    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f056854a-298f-4b46-a47f-04fa78274165-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s\" (UID: \"f056854a-298f-4b46-a47f-04fa78274165\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s"
Sep 15 21:01:19.850914 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:19.850916    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f056854a-298f-4b46-a47f-04fa78274165-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s\" (UID: \"f056854a-298f-4b46-a47f-04fa78274165\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s"
Sep 15 21:01:19.851172 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:19.850991    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jc4l\" (UniqueName: \"kubernetes.io/projected/f056854a-298f-4b46-a47f-04fa78274165-kube-api-access-9jc4l\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s\" (UID: \"f056854a-298f-4b46-a47f-04fa78274165\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s"
Sep 15 21:01:19.951765 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:19.951723    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f056854a-298f-4b46-a47f-04fa78274165-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s\" (UID: \"f056854a-298f-4b46-a47f-04fa78274165\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s"
Sep 15 21:01:19.951765 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:19.951766    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f056854a-298f-4b46-a47f-04fa78274165-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s\" (UID: \"f056854a-298f-4b46-a47f-04fa78274165\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s"
Sep 15 21:01:19.952097 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:19.951814    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jc4l\" (UniqueName: \"kubernetes.io/projected/f056854a-298f-4b46-a47f-04fa78274165-kube-api-access-9jc4l\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s\" (UID: \"f056854a-298f-4b46-a47f-04fa78274165\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s"
Sep 15 21:01:19.952170 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:19.952097    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f056854a-298f-4b46-a47f-04fa78274165-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s\" (UID: \"f056854a-298f-4b46-a47f-04fa78274165\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s"
Sep 15 21:01:19.952170 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:19.952134    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f056854a-298f-4b46-a47f-04fa78274165-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s\" (UID: \"f056854a-298f-4b46-a47f-04fa78274165\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s"
Sep 15 21:01:19.960738 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:19.960705    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jc4l\" (UniqueName: \"kubernetes.io/projected/f056854a-298f-4b46-a47f-04fa78274165-kube-api-access-9jc4l\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s\" (UID: \"f056854a-298f-4b46-a47f-04fa78274165\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s"
Sep 15 21:01:20.038045 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:20.037953    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s"
Sep 15 21:01:20.312466 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:20.312440    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s"]
Sep 15 21:01:20.314792 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:01:20.314763    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf056854a_298f_4b46_a47f_04fa78274165.slice/crio-543c9d19000a7e932a3e96b3be5827e35ce1e068b47d0777219137b42bd51e6d WatchSource:0}: Error finding container 543c9d19000a7e932a3e96b3be5827e35ce1e068b47d0777219137b42bd51e6d: Status 404 returned error can't find the container with id 543c9d19000a7e932a3e96b3be5827e35ce1e068b47d0777219137b42bd51e6d
Sep 15 21:01:20.833383 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:20.833351    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s" event={"ID":"f056854a-298f-4b46-a47f-04fa78274165","Type":"ContainerStarted","Data":"543c9d19000a7e932a3e96b3be5827e35ce1e068b47d0777219137b42bd51e6d"}
Sep 15 21:01:27.863214 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:27.863125    2558 generic.go:356] "Generic (PLEG): container finished" podID="f056854a-298f-4b46-a47f-04fa78274165" containerID="9e3bdbd663039cb126c590bac1f2cb70a432c8bc9983186a107d466880cd33a2" exitCode=0
Sep 15 21:01:27.863584 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:27.863211    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s" event={"ID":"f056854a-298f-4b46-a47f-04fa78274165","Type":"ContainerDied","Data":"9e3bdbd663039cb126c590bac1f2cb70a432c8bc9983186a107d466880cd33a2"}
Sep 15 21:01:30.876448 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:30.876408    2558 generic.go:356] "Generic (PLEG): container finished" podID="f056854a-298f-4b46-a47f-04fa78274165" containerID="996203ae00f69e992f9e2cbc0e5946cc0fbd153ceb5a738f2130220dc007e774" exitCode=0
Sep 15 21:01:30.876825 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:30.876496    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s" event={"ID":"f056854a-298f-4b46-a47f-04fa78274165","Type":"ContainerDied","Data":"996203ae00f69e992f9e2cbc0e5946cc0fbd153ceb5a738f2130220dc007e774"}
Sep 15 21:01:37.905715 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:37.905678    2558 generic.go:356] "Generic (PLEG): container finished" podID="f056854a-298f-4b46-a47f-04fa78274165" containerID="10a5c1be2def7d37dc822d9eb3c12c813b1160b3072ae14bb9834428159f0cd3" exitCode=0
Sep 15 21:01:37.906187 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:37.905754    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s" event={"ID":"f056854a-298f-4b46-a47f-04fa78274165","Type":"ContainerDied","Data":"10a5c1be2def7d37dc822d9eb3c12c813b1160b3072ae14bb9834428159f0cd3"}
Sep 15 21:01:39.036006 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:39.035975    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s"
Sep 15 21:01:39.114059 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:39.114021    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f056854a-298f-4b46-a47f-04fa78274165-util\") pod \"f056854a-298f-4b46-a47f-04fa78274165\" (UID: \"f056854a-298f-4b46-a47f-04fa78274165\") "
Sep 15 21:01:39.114267 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:39.114109    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f056854a-298f-4b46-a47f-04fa78274165-bundle\") pod \"f056854a-298f-4b46-a47f-04fa78274165\" (UID: \"f056854a-298f-4b46-a47f-04fa78274165\") "
Sep 15 21:01:39.114267 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:39.114143    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jc4l\" (UniqueName: \"kubernetes.io/projected/f056854a-298f-4b46-a47f-04fa78274165-kube-api-access-9jc4l\") pod \"f056854a-298f-4b46-a47f-04fa78274165\" (UID: \"f056854a-298f-4b46-a47f-04fa78274165\") "
Sep 15 21:01:39.114796 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:39.114771    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f056854a-298f-4b46-a47f-04fa78274165-bundle" (OuterVolumeSpecName: "bundle") pod "f056854a-298f-4b46-a47f-04fa78274165" (UID: "f056854a-298f-4b46-a47f-04fa78274165"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:01:39.116302 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:39.116278    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f056854a-298f-4b46-a47f-04fa78274165-kube-api-access-9jc4l" (OuterVolumeSpecName: "kube-api-access-9jc4l") pod "f056854a-298f-4b46-a47f-04fa78274165" (UID: "f056854a-298f-4b46-a47f-04fa78274165"). InnerVolumeSpecName "kube-api-access-9jc4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:01:39.119969 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:39.119938    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f056854a-298f-4b46-a47f-04fa78274165-util" (OuterVolumeSpecName: "util") pod "f056854a-298f-4b46-a47f-04fa78274165" (UID: "f056854a-298f-4b46-a47f-04fa78274165"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:01:39.215164 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:39.215122    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f056854a-298f-4b46-a47f-04fa78274165-bundle\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:01:39.215164 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:39.215155    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jc4l\" (UniqueName: \"kubernetes.io/projected/f056854a-298f-4b46-a47f-04fa78274165-kube-api-access-9jc4l\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:01:39.215164 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:39.215167    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f056854a-298f-4b46-a47f-04fa78274165-util\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:01:39.914111 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:39.914081    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s"
Sep 15 21:01:39.914111 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:39.914085    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sck6s" event={"ID":"f056854a-298f-4b46-a47f-04fa78274165","Type":"ContainerDied","Data":"543c9d19000a7e932a3e96b3be5827e35ce1e068b47d0777219137b42bd51e6d"}
Sep 15 21:01:39.914369 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:39.914142    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="543c9d19000a7e932a3e96b3be5827e35ce1e068b47d0777219137b42bd51e6d"
Sep 15 21:01:48.054837 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:48.054804    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v5rzv"]
Sep 15 21:01:48.055241 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:48.055076    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f056854a-298f-4b46-a47f-04fa78274165" containerName="util"
Sep 15 21:01:48.055241 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:48.055088    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f056854a-298f-4b46-a47f-04fa78274165" containerName="util"
Sep 15 21:01:48.055241 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:48.055096    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f056854a-298f-4b46-a47f-04fa78274165" containerName="extract"
Sep 15 21:01:48.055241 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:48.055102    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f056854a-298f-4b46-a47f-04fa78274165" containerName="extract"
Sep 15 21:01:48.055241 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:48.055110    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f056854a-298f-4b46-a47f-04fa78274165" containerName="pull"
Sep 15 21:01:48.055241 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:48.055117    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f056854a-298f-4b46-a47f-04fa78274165" containerName="pull"
Sep 15 21:01:48.055241 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:48.055167    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f056854a-298f-4b46-a47f-04fa78274165" containerName="extract"
Sep 15 21:01:48.058837 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:48.058816    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v5rzv"
Sep 15 21:01:48.063190 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:48.063164    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:01:48.063299 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:48.063192    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 15 21:01:48.063338 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:48.063294    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-t4zzt\""
Sep 15 21:01:48.074216 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:48.074186    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v5rzv"]
Sep 15 21:01:48.187727 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:48.187685    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a052fa08-6e83-4086-87c9-75ae252f30f6-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-v5rzv\" (UID: \"a052fa08-6e83-4086-87c9-75ae252f30f6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v5rzv"
Sep 15 21:01:48.187727 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:48.187732    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9pbf\" (UniqueName: \"kubernetes.io/projected/a052fa08-6e83-4086-87c9-75ae252f30f6-kube-api-access-c9pbf\") pod \"cert-manager-operator-controller-manager-55c444bf86-v5rzv\" (UID: \"a052fa08-6e83-4086-87c9-75ae252f30f6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v5rzv"
Sep 15 21:01:48.289007 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:48.288973    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a052fa08-6e83-4086-87c9-75ae252f30f6-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-v5rzv\" (UID: \"a052fa08-6e83-4086-87c9-75ae252f30f6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v5rzv"
Sep 15 21:01:48.289007 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:48.289012    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9pbf\" (UniqueName: \"kubernetes.io/projected/a052fa08-6e83-4086-87c9-75ae252f30f6-kube-api-access-c9pbf\") pod \"cert-manager-operator-controller-manager-55c444bf86-v5rzv\" (UID: \"a052fa08-6e83-4086-87c9-75ae252f30f6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v5rzv"
Sep 15 21:01:48.289366 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:48.289348    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a052fa08-6e83-4086-87c9-75ae252f30f6-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-v5rzv\" (UID: \"a052fa08-6e83-4086-87c9-75ae252f30f6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v5rzv"
Sep 15 21:01:48.300782 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:48.300749    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9pbf\" (UniqueName: \"kubernetes.io/projected/a052fa08-6e83-4086-87c9-75ae252f30f6-kube-api-access-c9pbf\") pod \"cert-manager-operator-controller-manager-55c444bf86-v5rzv\" (UID: \"a052fa08-6e83-4086-87c9-75ae252f30f6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v5rzv"
Sep 15 21:01:48.367761 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:48.367659    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v5rzv"
Sep 15 21:01:48.647487 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:48.647460    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v5rzv"]
Sep 15 21:01:48.650240 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:01:48.650212    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda052fa08_6e83_4086_87c9_75ae252f30f6.slice/crio-392db302add7526f6c1c7f27a2fefb7bac354dab2bf77ea0fa4ff3617d38d32b WatchSource:0}: Error finding container 392db302add7526f6c1c7f27a2fefb7bac354dab2bf77ea0fa4ff3617d38d32b: Status 404 returned error can't find the container with id 392db302add7526f6c1c7f27a2fefb7bac354dab2bf77ea0fa4ff3617d38d32b
Sep 15 21:01:48.946753 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:48.946718    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v5rzv" event={"ID":"a052fa08-6e83-4086-87c9-75ae252f30f6","Type":"ContainerStarted","Data":"392db302add7526f6c1c7f27a2fefb7bac354dab2bf77ea0fa4ff3617d38d32b"}
Sep 15 21:01:50.957326 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:50.957284    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v5rzv" event={"ID":"a052fa08-6e83-4086-87c9-75ae252f30f6","Type":"ContainerStarted","Data":"9d7992ceafc47dd455d7aa6d989be6de1b62d88793883369b07ae9f819635bf4"}
Sep 15 21:01:50.980109 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:50.980034    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v5rzv" podStartSLOduration=0.866296079 podStartE2EDuration="2.980014515s" totalImagesPullingTime="2.113718436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:01:50.978887495 +0000 UTC m=+229.469533227" watchObservedRunningTime="2026-09-15 21:01:50.980014515 +0000 UTC m=+229.470660248"
Sep 15 21:01:51.860152 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:51.860116    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-d79vm"]
Sep 15 21:01:51.862274 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:51.862257    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-d79vm"
Sep 15 21:01:51.865278 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:51.865255    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 15 21:01:51.866603 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:51.866582    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-f9g6c\""
Sep 15 21:01:51.866603 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:51.866597    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 15 21:01:51.873482 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:51.873456    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-d79vm"]
Sep 15 21:01:51.919277 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:51.919237    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jz9z\" (UniqueName: \"kubernetes.io/projected/11694616-5c49-4b61-93f5-d97ebab2388c-kube-api-access-5jz9z\") pod \"cert-manager-webhook-6486ffd997-d79vm\" (UID: \"11694616-5c49-4b61-93f5-d97ebab2388c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-d79vm"
Sep 15 21:01:51.919459 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:51.919314    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11694616-5c49-4b61-93f5-d97ebab2388c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-d79vm\" (UID: \"11694616-5c49-4b61-93f5-d97ebab2388c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-d79vm"
Sep 15 21:01:52.019814 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.019776    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jz9z\" (UniqueName: \"kubernetes.io/projected/11694616-5c49-4b61-93f5-d97ebab2388c-kube-api-access-5jz9z\") pod \"cert-manager-webhook-6486ffd997-d79vm\" (UID: \"11694616-5c49-4b61-93f5-d97ebab2388c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-d79vm"
Sep 15 21:01:52.020319 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.019856    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11694616-5c49-4b61-93f5-d97ebab2388c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-d79vm\" (UID: \"11694616-5c49-4b61-93f5-d97ebab2388c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-d79vm"
Sep 15 21:01:52.028604 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.028574    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11694616-5c49-4b61-93f5-d97ebab2388c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-d79vm\" (UID: \"11694616-5c49-4b61-93f5-d97ebab2388c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-d79vm"
Sep 15 21:01:52.028725 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.028644    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jz9z\" (UniqueName: \"kubernetes.io/projected/11694616-5c49-4b61-93f5-d97ebab2388c-kube-api-access-5jz9z\") pod \"cert-manager-webhook-6486ffd997-d79vm\" (UID: \"11694616-5c49-4b61-93f5-d97ebab2388c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-d79vm"
Sep 15 21:01:52.182722 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.182680    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-d79vm"
Sep 15 21:01:52.367107 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.367072    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh"]
Sep 15 21:01:52.369512 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.369494    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh"
Sep 15 21:01:52.373368 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.373346    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 21:01:52.373507 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.373346    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 21:01:52.373507 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.373405    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6r6f2\""
Sep 15 21:01:52.379669 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.379602    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh"]
Sep 15 21:01:52.456545 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.456468    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-d79vm"]
Sep 15 21:01:52.459476 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:01:52.459446    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11694616_5c49_4b61_93f5_d97ebab2388c.slice/crio-692436cb65d69346638ecf660b7517d71e799523f521a2232a9c68139451a1d2 WatchSource:0}: Error finding container 692436cb65d69346638ecf660b7517d71e799523f521a2232a9c68139451a1d2: Status 404 returned error can't find the container with id 692436cb65d69346638ecf660b7517d71e799523f521a2232a9c68139451a1d2
Sep 15 21:01:52.524157 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.524106    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blll6\" (UniqueName: \"kubernetes.io/projected/e74d8dfc-6783-461b-aad6-4680e1b616b2-kube-api-access-blll6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh\" (UID: \"e74d8dfc-6783-461b-aad6-4680e1b616b2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh"
Sep 15 21:01:52.524157 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.524160    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e74d8dfc-6783-461b-aad6-4680e1b616b2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh\" (UID: \"e74d8dfc-6783-461b-aad6-4680e1b616b2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh"
Sep 15 21:01:52.524372 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.524283    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e74d8dfc-6783-461b-aad6-4680e1b616b2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh\" (UID: \"e74d8dfc-6783-461b-aad6-4680e1b616b2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh"
Sep 15 21:01:52.625470 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.625417    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e74d8dfc-6783-461b-aad6-4680e1b616b2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh\" (UID: \"e74d8dfc-6783-461b-aad6-4680e1b616b2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh"
Sep 15 21:01:52.625470 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.625474    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blll6\" (UniqueName: \"kubernetes.io/projected/e74d8dfc-6783-461b-aad6-4680e1b616b2-kube-api-access-blll6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh\" (UID: \"e74d8dfc-6783-461b-aad6-4680e1b616b2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh"
Sep 15 21:01:52.625687 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.625599    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e74d8dfc-6783-461b-aad6-4680e1b616b2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh\" (UID: \"e74d8dfc-6783-461b-aad6-4680e1b616b2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh"
Sep 15 21:01:52.625874 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.625856    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e74d8dfc-6783-461b-aad6-4680e1b616b2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh\" (UID: \"e74d8dfc-6783-461b-aad6-4680e1b616b2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh"
Sep 15 21:01:52.625908 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.625895    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e74d8dfc-6783-461b-aad6-4680e1b616b2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh\" (UID: \"e74d8dfc-6783-461b-aad6-4680e1b616b2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh"
Sep 15 21:01:52.634539 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.634502    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blll6\" (UniqueName: \"kubernetes.io/projected/e74d8dfc-6783-461b-aad6-4680e1b616b2-kube-api-access-blll6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh\" (UID: \"e74d8dfc-6783-461b-aad6-4680e1b616b2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh"
Sep 15 21:01:52.679526 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.679490    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh"
Sep 15 21:01:52.949517 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.949483    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh"]
Sep 15 21:01:52.952305 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:01:52.952279    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74d8dfc_6783_461b_aad6_4680e1b616b2.slice/crio-ef8906c11595690f612c238dabc6ef31ec1611b29e3c46a30c811d9529e9220d WatchSource:0}: Error finding container ef8906c11595690f612c238dabc6ef31ec1611b29e3c46a30c811d9529e9220d: Status 404 returned error can't find the container with id ef8906c11595690f612c238dabc6ef31ec1611b29e3c46a30c811d9529e9220d
Sep 15 21:01:52.965634 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.965605    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh" event={"ID":"e74d8dfc-6783-461b-aad6-4680e1b616b2","Type":"ContainerStarted","Data":"ef8906c11595690f612c238dabc6ef31ec1611b29e3c46a30c811d9529e9220d"}
Sep 15 21:01:52.966817 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:52.966792    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-d79vm" event={"ID":"11694616-5c49-4b61-93f5-d97ebab2388c","Type":"ContainerStarted","Data":"692436cb65d69346638ecf660b7517d71e799523f521a2232a9c68139451a1d2"}
Sep 15 21:01:53.973628 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:53.973586    2558 generic.go:356] "Generic (PLEG): container finished" podID="e74d8dfc-6783-461b-aad6-4680e1b616b2" containerID="16ce2b17db0c81d6a624b2ad189057a30f1a3811c65a9a43c5a0bbc495749ba3" exitCode=0
Sep 15 21:01:53.974109 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:53.973704    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh" event={"ID":"e74d8dfc-6783-461b-aad6-4680e1b616b2","Type":"ContainerDied","Data":"16ce2b17db0c81d6a624b2ad189057a30f1a3811c65a9a43c5a0bbc495749ba3"}
Sep 15 21:01:55.985777 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:55.985736    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-d79vm" event={"ID":"11694616-5c49-4b61-93f5-d97ebab2388c","Type":"ContainerStarted","Data":"56b4d1138d3ae00bb70e3f1a894e72cfd15d20e89737edee8786a44cc770e342"}
Sep 15 21:01:55.986241 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:55.985902    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-d79vm"
Sep 15 21:01:56.009576 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:56.009514    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-d79vm" podStartSLOduration=2.232591775 podStartE2EDuration="5.00949642s" totalImagesPullingTime="2.776904645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:01:56.007039145 +0000 UTC m=+234.497684878" watchObservedRunningTime="2026-09-15 21:01:56.00949642 +0000 UTC m=+234.500142150"
Sep 15 21:01:56.991589 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:56.991554    2558 generic.go:356] "Generic (PLEG): container finished" podID="e74d8dfc-6783-461b-aad6-4680e1b616b2" containerID="9dd14faa5b9997b135c3d39d80c712a2e00f883bc3e841169dd30bdfc4a8ff10" exitCode=0
Sep 15 21:01:56.992101 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:56.991640    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh" event={"ID":"e74d8dfc-6783-461b-aad6-4680e1b616b2","Type":"ContainerDied","Data":"9dd14faa5b9997b135c3d39d80c712a2e00f883bc3e841169dd30bdfc4a8ff10"}
Sep 15 21:01:57.997428 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:57.997394    2558 generic.go:356] "Generic (PLEG): container finished" podID="e74d8dfc-6783-461b-aad6-4680e1b616b2" containerID="8510b1f7cc615c36ff92c1f835b7231a50504522b1cfd8f2bd4788604a29f95c" exitCode=0
Sep 15 21:01:57.997818 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:57.997452    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh" event={"ID":"e74d8dfc-6783-461b-aad6-4680e1b616b2","Type":"ContainerDied","Data":"8510b1f7cc615c36ff92c1f835b7231a50504522b1cfd8f2bd4788604a29f95c"}
Sep 15 21:01:59.132228 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.132195    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh"
Sep 15 21:01:59.287650 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.287555    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blll6\" (UniqueName: \"kubernetes.io/projected/e74d8dfc-6783-461b-aad6-4680e1b616b2-kube-api-access-blll6\") pod \"e74d8dfc-6783-461b-aad6-4680e1b616b2\" (UID: \"e74d8dfc-6783-461b-aad6-4680e1b616b2\") "
Sep 15 21:01:59.287650 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.287610    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e74d8dfc-6783-461b-aad6-4680e1b616b2-util\") pod \"e74d8dfc-6783-461b-aad6-4680e1b616b2\" (UID: \"e74d8dfc-6783-461b-aad6-4680e1b616b2\") "
Sep 15 21:01:59.287834 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.287690    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e74d8dfc-6783-461b-aad6-4680e1b616b2-bundle\") pod \"e74d8dfc-6783-461b-aad6-4680e1b616b2\" (UID: \"e74d8dfc-6783-461b-aad6-4680e1b616b2\") "
Sep 15 21:01:59.288120 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.288063    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74d8dfc-6783-461b-aad6-4680e1b616b2-bundle" (OuterVolumeSpecName: "bundle") pod "e74d8dfc-6783-461b-aad6-4680e1b616b2" (UID: "e74d8dfc-6783-461b-aad6-4680e1b616b2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:01:59.289650 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.289614    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74d8dfc-6783-461b-aad6-4680e1b616b2-kube-api-access-blll6" (OuterVolumeSpecName: "kube-api-access-blll6") pod "e74d8dfc-6783-461b-aad6-4680e1b616b2" (UID: "e74d8dfc-6783-461b-aad6-4680e1b616b2"). InnerVolumeSpecName "kube-api-access-blll6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:01:59.291879 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.291843    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74d8dfc-6783-461b-aad6-4680e1b616b2-util" (OuterVolumeSpecName: "util") pod "e74d8dfc-6783-461b-aad6-4680e1b616b2" (UID: "e74d8dfc-6783-461b-aad6-4680e1b616b2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:01:59.388867 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.388834    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e74d8dfc-6783-461b-aad6-4680e1b616b2-bundle\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:01:59.388867 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.388861    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blll6\" (UniqueName: \"kubernetes.io/projected/e74d8dfc-6783-461b-aad6-4680e1b616b2-kube-api-access-blll6\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:01:59.388867 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.388871    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e74d8dfc-6783-461b-aad6-4680e1b616b2-util\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:01:59.561001 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.560923    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-nwghd"]
Sep 15 21:01:59.561236 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.561223    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e74d8dfc-6783-461b-aad6-4680e1b616b2" containerName="util"
Sep 15 21:01:59.561278 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.561238    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e74d8dfc-6783-461b-aad6-4680e1b616b2" containerName="util"
Sep 15 21:01:59.561278 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.561252    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e74d8dfc-6783-461b-aad6-4680e1b616b2" containerName="extract"
Sep 15 21:01:59.561278 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.561257    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e74d8dfc-6783-461b-aad6-4680e1b616b2" containerName="extract"
Sep 15 21:01:59.561278 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.561271    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e74d8dfc-6783-461b-aad6-4680e1b616b2" containerName="pull"
Sep 15 21:01:59.561278 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.561276    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e74d8dfc-6783-461b-aad6-4680e1b616b2" containerName="pull"
Sep 15 21:01:59.561421 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.561323    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e74d8dfc-6783-461b-aad6-4680e1b616b2" containerName="extract"
Sep 15 21:01:59.564307 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.564286    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nwghd"
Sep 15 21:01:59.567324 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.567300    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-dtsxh\""
Sep 15 21:01:59.572401 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.572374    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-nwghd"]
Sep 15 21:01:59.692247 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.692207    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glbcv\" (UniqueName: \"kubernetes.io/projected/5a43c2d0-1438-44e0-99ff-7e91d6004962-kube-api-access-glbcv\") pod \"cert-manager-cainjector-559cdd8c87-nwghd\" (UID: \"5a43c2d0-1438-44e0-99ff-7e91d6004962\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nwghd"
Sep 15 21:01:59.692438 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.692259    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a43c2d0-1438-44e0-99ff-7e91d6004962-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-nwghd\" (UID: \"5a43c2d0-1438-44e0-99ff-7e91d6004962\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nwghd"
Sep 15 21:01:59.792819 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.792779    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glbcv\" (UniqueName: \"kubernetes.io/projected/5a43c2d0-1438-44e0-99ff-7e91d6004962-kube-api-access-glbcv\") pod \"cert-manager-cainjector-559cdd8c87-nwghd\" (UID: \"5a43c2d0-1438-44e0-99ff-7e91d6004962\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nwghd"
Sep 15 21:01:59.793003 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.792847    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a43c2d0-1438-44e0-99ff-7e91d6004962-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-nwghd\" (UID: \"5a43c2d0-1438-44e0-99ff-7e91d6004962\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nwghd"
Sep 15 21:01:59.802403 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.802370    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a43c2d0-1438-44e0-99ff-7e91d6004962-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-nwghd\" (UID: \"5a43c2d0-1438-44e0-99ff-7e91d6004962\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nwghd"
Sep 15 21:01:59.802537 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.802459    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glbcv\" (UniqueName: \"kubernetes.io/projected/5a43c2d0-1438-44e0-99ff-7e91d6004962-kube-api-access-glbcv\") pod \"cert-manager-cainjector-559cdd8c87-nwghd\" (UID: \"5a43c2d0-1438-44e0-99ff-7e91d6004962\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nwghd"
Sep 15 21:01:59.873884 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:01:59.873793    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nwghd"
Sep 15 21:02:00.008633 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:00.008601    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh"
Sep 15 21:02:00.008791 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:00.008629    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvbdh" event={"ID":"e74d8dfc-6783-461b-aad6-4680e1b616b2","Type":"ContainerDied","Data":"ef8906c11595690f612c238dabc6ef31ec1611b29e3c46a30c811d9529e9220d"}
Sep 15 21:02:00.008791 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:00.008679    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef8906c11595690f612c238dabc6ef31ec1611b29e3c46a30c811d9529e9220d"
Sep 15 21:02:00.147325 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:00.147250    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-nwghd"]
Sep 15 21:02:01.013920 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:01.013885    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nwghd" event={"ID":"5a43c2d0-1438-44e0-99ff-7e91d6004962","Type":"ContainerStarted","Data":"2d8c8ed0ec75e0e243ecd1fb341d31a7d853a25da5b1b7013c2828b3cab22601"}
Sep 15 21:02:01.014117 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:01.013928    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nwghd" event={"ID":"5a43c2d0-1438-44e0-99ff-7e91d6004962","Type":"ContainerStarted","Data":"24b93749bc80cd3c56ae7d21af8c740f9947dd6207441bdd6e09090117e91fd2"}
Sep 15 21:02:01.032965 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:01.032904    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nwghd" podStartSLOduration=2.032886004 podStartE2EDuration="2.032886004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:02:01.031827901 +0000 UTC m=+239.522473633" watchObservedRunningTime="2026-09-15 21:02:01.032886004 +0000 UTC m=+239.523531736"
Sep 15 21:02:01.994582 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:01.994553    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-d79vm"
Sep 15 21:02:10.119481 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:10.119436    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jp7c7"]
Sep 15 21:02:10.123660 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:10.123621    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jp7c7"
Sep 15 21:02:10.129502 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:10.129457    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 15 21:02:10.130122 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:10.130097    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:02:10.132418 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:10.132399    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-d5mdx\""
Sep 15 21:02:10.145031 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:10.145003    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jp7c7"]
Sep 15 21:02:10.275860 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:10.275825    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d0f3039-6a34-4ea1-92f1-672bbc5c6143-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jp7c7\" (UID: \"7d0f3039-6a34-4ea1-92f1-672bbc5c6143\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jp7c7"
Sep 15 21:02:10.275860 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:10.275866    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv4v6\" (UniqueName: \"kubernetes.io/projected/7d0f3039-6a34-4ea1-92f1-672bbc5c6143-kube-api-access-wv4v6\") pod \"openshift-lws-operator-fd8ccff4c-jp7c7\" (UID: \"7d0f3039-6a34-4ea1-92f1-672bbc5c6143\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jp7c7"
Sep 15 21:02:10.376833 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:10.376734    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d0f3039-6a34-4ea1-92f1-672bbc5c6143-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jp7c7\" (UID: \"7d0f3039-6a34-4ea1-92f1-672bbc5c6143\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jp7c7"
Sep 15 21:02:10.376833 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:10.376777    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv4v6\" (UniqueName: \"kubernetes.io/projected/7d0f3039-6a34-4ea1-92f1-672bbc5c6143-kube-api-access-wv4v6\") pod \"openshift-lws-operator-fd8ccff4c-jp7c7\" (UID: \"7d0f3039-6a34-4ea1-92f1-672bbc5c6143\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jp7c7"
Sep 15 21:02:10.377213 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:10.377190    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d0f3039-6a34-4ea1-92f1-672bbc5c6143-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jp7c7\" (UID: \"7d0f3039-6a34-4ea1-92f1-672bbc5c6143\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jp7c7"
Sep 15 21:02:10.385740 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:10.385706    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv4v6\" (UniqueName: \"kubernetes.io/projected/7d0f3039-6a34-4ea1-92f1-672bbc5c6143-kube-api-access-wv4v6\") pod \"openshift-lws-operator-fd8ccff4c-jp7c7\" (UID: \"7d0f3039-6a34-4ea1-92f1-672bbc5c6143\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jp7c7"
Sep 15 21:02:10.433163 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:10.433127    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jp7c7"
Sep 15 21:02:10.703983 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:10.703956    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jp7c7"]
Sep 15 21:02:10.706492 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:02:10.706463    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d0f3039_6a34_4ea1_92f1_672bbc5c6143.slice/crio-2b8eac36cc31d6233f239b9db5c23a9931988607533393d83e156d4c475b4829 WatchSource:0}: Error finding container 2b8eac36cc31d6233f239b9db5c23a9931988607533393d83e156d4c475b4829: Status 404 returned error can't find the container with id 2b8eac36cc31d6233f239b9db5c23a9931988607533393d83e156d4c475b4829
Sep 15 21:02:11.057915 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:11.057832    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jp7c7" event={"ID":"7d0f3039-6a34-4ea1-92f1-672bbc5c6143","Type":"ContainerStarted","Data":"2b8eac36cc31d6233f239b9db5c23a9931988607533393d83e156d4c475b4829"}
Sep 15 21:02:13.069807 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:13.069773    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jp7c7" event={"ID":"7d0f3039-6a34-4ea1-92f1-672bbc5c6143","Type":"ContainerStarted","Data":"6a8bc9198f46a9b1bc4a0e9663c6c0464f1a1b818755bc2f9b49dcb0de05b0ef"}
Sep 15 21:02:13.089312 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:13.089248    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jp7c7" podStartSLOduration=1.5824199939999999 podStartE2EDuration="3.089225844s" totalImagesPullingTime="1.50680585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:02:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:02:13.086775999 +0000 UTC m=+251.577421732" watchObservedRunningTime="2026-09-15 21:02:13.089225844 +0000 UTC m=+251.579871577"
Sep 15 21:02:18.905278 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:18.905235    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2"]
Sep 15 21:02:18.908807 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:18.908788    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2"
Sep 15 21:02:18.911925 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:18.911901    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 21:02:18.913517 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:18.913501    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6r6f2\""
Sep 15 21:02:18.913649 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:18.913502    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 21:02:18.921557 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:18.921532    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2"]
Sep 15 21:02:18.945410 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:18.945374    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3d8c0b9-f038-4928-97f4-c00d0948e7fe-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2\" (UID: \"b3d8c0b9-f038-4928-97f4-c00d0948e7fe\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2"
Sep 15 21:02:18.945586 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:18.945450    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrnqq\" (UniqueName: \"kubernetes.io/projected/b3d8c0b9-f038-4928-97f4-c00d0948e7fe-kube-api-access-rrnqq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2\" (UID: \"b3d8c0b9-f038-4928-97f4-c00d0948e7fe\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2"
Sep 15 21:02:18.945586 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:18.945556    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3d8c0b9-f038-4928-97f4-c00d0948e7fe-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2\" (UID: \"b3d8c0b9-f038-4928-97f4-c00d0948e7fe\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2"
Sep 15 21:02:19.046553 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:19.046500    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrnqq\" (UniqueName: \"kubernetes.io/projected/b3d8c0b9-f038-4928-97f4-c00d0948e7fe-kube-api-access-rrnqq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2\" (UID: \"b3d8c0b9-f038-4928-97f4-c00d0948e7fe\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2"
Sep 15 21:02:19.046757 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:19.046601    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3d8c0b9-f038-4928-97f4-c00d0948e7fe-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2\" (UID: \"b3d8c0b9-f038-4928-97f4-c00d0948e7fe\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2"
Sep 15 21:02:19.046757 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:19.046642    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3d8c0b9-f038-4928-97f4-c00d0948e7fe-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2\" (UID: \"b3d8c0b9-f038-4928-97f4-c00d0948e7fe\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2"
Sep 15 21:02:19.047043 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:19.047021    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3d8c0b9-f038-4928-97f4-c00d0948e7fe-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2\" (UID: \"b3d8c0b9-f038-4928-97f4-c00d0948e7fe\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2"
Sep 15 21:02:19.047146 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:19.047084    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3d8c0b9-f038-4928-97f4-c00d0948e7fe-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2\" (UID: \"b3d8c0b9-f038-4928-97f4-c00d0948e7fe\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2"
Sep 15 21:02:19.055309 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:19.055284    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrnqq\" (UniqueName: \"kubernetes.io/projected/b3d8c0b9-f038-4928-97f4-c00d0948e7fe-kube-api-access-rrnqq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2\" (UID: \"b3d8c0b9-f038-4928-97f4-c00d0948e7fe\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2"
Sep 15 21:02:19.218775 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:19.218741    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2"
Sep 15 21:02:19.487428 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:19.487395    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2"]
Sep 15 21:02:19.489827 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:02:19.489800    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d8c0b9_f038_4928_97f4_c00d0948e7fe.slice/crio-51ef90e81de244cad8c87522c24f8e1c2de9ed9f0e8b31ea127c3865cf23efa8 WatchSource:0}: Error finding container 51ef90e81de244cad8c87522c24f8e1c2de9ed9f0e8b31ea127c3865cf23efa8: Status 404 returned error can't find the container with id 51ef90e81de244cad8c87522c24f8e1c2de9ed9f0e8b31ea127c3865cf23efa8
Sep 15 21:02:20.104324 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:20.104287    2558 generic.go:356] "Generic (PLEG): container finished" podID="b3d8c0b9-f038-4928-97f4-c00d0948e7fe" containerID="b7c4403f556c59f4de73df507cd145e08fc572512ace9084005e66c80466df84" exitCode=0
Sep 15 21:02:20.104725 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:20.104368    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2" event={"ID":"b3d8c0b9-f038-4928-97f4-c00d0948e7fe","Type":"ContainerDied","Data":"b7c4403f556c59f4de73df507cd145e08fc572512ace9084005e66c80466df84"}
Sep 15 21:02:20.104725 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:20.104413    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2" event={"ID":"b3d8c0b9-f038-4928-97f4-c00d0948e7fe","Type":"ContainerStarted","Data":"51ef90e81de244cad8c87522c24f8e1c2de9ed9f0e8b31ea127c3865cf23efa8"}
Sep 15 21:02:21.112641 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:21.112545    2558 generic.go:356] "Generic (PLEG): container finished" podID="b3d8c0b9-f038-4928-97f4-c00d0948e7fe" containerID="421b0fa059f30fe402a28389c1fd2d005bfcd344f9e596f44f2b346cf4977aaa" exitCode=0
Sep 15 21:02:21.112641 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:21.112590    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2" event={"ID":"b3d8c0b9-f038-4928-97f4-c00d0948e7fe","Type":"ContainerDied","Data":"421b0fa059f30fe402a28389c1fd2d005bfcd344f9e596f44f2b346cf4977aaa"}
Sep 15 21:02:22.118990 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:22.118957    2558 generic.go:356] "Generic (PLEG): container finished" podID="b3d8c0b9-f038-4928-97f4-c00d0948e7fe" containerID="fee64e40b127495c432673012425291cefc5262d3a30fe5775aeb256a2802b9b" exitCode=0
Sep 15 21:02:22.119396 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:22.119012    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2" event={"ID":"b3d8c0b9-f038-4928-97f4-c00d0948e7fe","Type":"ContainerDied","Data":"fee64e40b127495c432673012425291cefc5262d3a30fe5775aeb256a2802b9b"}
Sep 15 21:02:23.249575 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:23.249548    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2"
Sep 15 21:02:23.274649 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:23.274615    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3d8c0b9-f038-4928-97f4-c00d0948e7fe-bundle\") pod \"b3d8c0b9-f038-4928-97f4-c00d0948e7fe\" (UID: \"b3d8c0b9-f038-4928-97f4-c00d0948e7fe\") "
Sep 15 21:02:23.274814 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:23.274678    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3d8c0b9-f038-4928-97f4-c00d0948e7fe-util\") pod \"b3d8c0b9-f038-4928-97f4-c00d0948e7fe\" (UID: \"b3d8c0b9-f038-4928-97f4-c00d0948e7fe\") "
Sep 15 21:02:23.274814 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:23.274711    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrnqq\" (UniqueName: \"kubernetes.io/projected/b3d8c0b9-f038-4928-97f4-c00d0948e7fe-kube-api-access-rrnqq\") pod \"b3d8c0b9-f038-4928-97f4-c00d0948e7fe\" (UID: \"b3d8c0b9-f038-4928-97f4-c00d0948e7fe\") "
Sep 15 21:02:23.275345 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:23.275317    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d8c0b9-f038-4928-97f4-c00d0948e7fe-bundle" (OuterVolumeSpecName: "bundle") pod "b3d8c0b9-f038-4928-97f4-c00d0948e7fe" (UID: "b3d8c0b9-f038-4928-97f4-c00d0948e7fe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:02:23.276891 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:23.276864    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d8c0b9-f038-4928-97f4-c00d0948e7fe-kube-api-access-rrnqq" (OuterVolumeSpecName: "kube-api-access-rrnqq") pod "b3d8c0b9-f038-4928-97f4-c00d0948e7fe" (UID: "b3d8c0b9-f038-4928-97f4-c00d0948e7fe"). InnerVolumeSpecName "kube-api-access-rrnqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:02:23.283650 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:23.283610    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d8c0b9-f038-4928-97f4-c00d0948e7fe-util" (OuterVolumeSpecName: "util") pod "b3d8c0b9-f038-4928-97f4-c00d0948e7fe" (UID: "b3d8c0b9-f038-4928-97f4-c00d0948e7fe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:02:23.375319 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:23.375229    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3d8c0b9-f038-4928-97f4-c00d0948e7fe-util\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:02:23.375319 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:23.375258    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrnqq\" (UniqueName: \"kubernetes.io/projected/b3d8c0b9-f038-4928-97f4-c00d0948e7fe-kube-api-access-rrnqq\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:02:23.375319 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:23.375271    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3d8c0b9-f038-4928-97f4-c00d0948e7fe-bundle\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:02:24.129986 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:24.129903    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2" event={"ID":"b3d8c0b9-f038-4928-97f4-c00d0948e7fe","Type":"ContainerDied","Data":"51ef90e81de244cad8c87522c24f8e1c2de9ed9f0e8b31ea127c3865cf23efa8"}
Sep 15 21:02:24.129986 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:24.129933    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64lj2"
Sep 15 21:02:24.129986 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:24.129962    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51ef90e81de244cad8c87522c24f8e1c2de9ed9f0e8b31ea127c3865cf23efa8"
Sep 15 21:02:29.109416 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:29.109381    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579"]
Sep 15 21:02:29.109787 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:29.109662    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3d8c0b9-f038-4928-97f4-c00d0948e7fe" containerName="util"
Sep 15 21:02:29.109787 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:29.109676    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3d8c0b9-f038-4928-97f4-c00d0948e7fe" containerName="util"
Sep 15 21:02:29.109787 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:29.109684    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3d8c0b9-f038-4928-97f4-c00d0948e7fe" containerName="extract"
Sep 15 21:02:29.109787 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:29.109691    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3d8c0b9-f038-4928-97f4-c00d0948e7fe" containerName="extract"
Sep 15 21:02:29.109787 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:29.109707    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3d8c0b9-f038-4928-97f4-c00d0948e7fe" containerName="pull"
Sep 15 21:02:29.109787 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:29.109714    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3d8c0b9-f038-4928-97f4-c00d0948e7fe" containerName="pull"
Sep 15 21:02:29.109787 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:29.109759    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3d8c0b9-f038-4928-97f4-c00d0948e7fe" containerName="extract"
Sep 15 21:02:29.112811 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:29.112792    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579"
Sep 15 21:02:29.113423 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:29.113392    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/81730a25-1463-4cf0-bf25-5501fc569337-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579\" (UID: \"81730a25-1463-4cf0-bf25-5501fc569337\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579"
Sep 15 21:02:29.113520 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:29.113438    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/81730a25-1463-4cf0-bf25-5501fc569337-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579\" (UID: \"81730a25-1463-4cf0-bf25-5501fc569337\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579"
Sep 15 21:02:29.113576 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:29.113533    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trmgt\" (UniqueName: \"kubernetes.io/projected/81730a25-1463-4cf0-bf25-5501fc569337-kube-api-access-trmgt\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579\" (UID: \"81730a25-1463-4cf0-bf25-5501fc569337\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579"
Sep 15 21:02:29.115900 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:29.115875    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6r6f2\""
Sep 15 21:02:29.116028 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:29.115937    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 21:02:29.116028 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:29.115937    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 21:02:29.122404 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:29.122365    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579"]
Sep 15 21:02:29.214497 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:29.214464    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/81730a25-1463-4cf0-bf25-5501fc569337-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579\" (UID: \"81730a25-1463-4cf0-bf25-5501fc569337\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579"
Sep 15 21:02:29.214497 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:29.214502    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/81730a25-1463-4cf0-bf25-5501fc569337-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579\" (UID: \"81730a25-1463-4cf0-bf25-5501fc569337\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579"
Sep 15 21:02:29.214745 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:29.214538    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trmgt\" (UniqueName: \"kubernetes.io/projected/81730a25-1463-4cf0-bf25-5501fc569337-kube-api-access-trmgt\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579\" (UID: \"81730a25-1463-4cf0-bf25-5501fc569337\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579"
Sep 15 21:02:29.214917 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:29.214894    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/81730a25-1463-4cf0-bf25-5501fc569337-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579\" (UID: \"81730a25-1463-4cf0-bf25-5501fc569337\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579"
Sep 15 21:02:29.214985 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:29.214916    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/81730a25-1463-4cf0-bf25-5501fc569337-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579\" (UID: \"81730a25-1463-4cf0-bf25-5501fc569337\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579"
Sep 15 21:02:29.223720 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:29.223691    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trmgt\" (UniqueName: \"kubernetes.io/projected/81730a25-1463-4cf0-bf25-5501fc569337-kube-api-access-trmgt\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579\" (UID: \"81730a25-1463-4cf0-bf25-5501fc569337\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579"
Sep 15 21:02:29.423517 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:29.423478    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579"
Sep 15 21:02:29.702210 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:29.702178    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579"]
Sep 15 21:02:29.704755 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:02:29.704724    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81730a25_1463_4cf0_bf25_5501fc569337.slice/crio-654de40eaa9655242d56f197d795b304092419f72e53ea6dbfe5afa870e2ca3f WatchSource:0}: Error finding container 654de40eaa9655242d56f197d795b304092419f72e53ea6dbfe5afa870e2ca3f: Status 404 returned error can't find the container with id 654de40eaa9655242d56f197d795b304092419f72e53ea6dbfe5afa870e2ca3f
Sep 15 21:02:30.157532 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:30.157432    2558 generic.go:356] "Generic (PLEG): container finished" podID="81730a25-1463-4cf0-bf25-5501fc569337" containerID="6b4b0593574d78c4a0d4f0c8eee8fcdc45134b559ab684ecee292b504b29dd45" exitCode=0
Sep 15 21:02:30.157532 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:30.157513    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579" event={"ID":"81730a25-1463-4cf0-bf25-5501fc569337","Type":"ContainerDied","Data":"6b4b0593574d78c4a0d4f0c8eee8fcdc45134b559ab684ecee292b504b29dd45"}
Sep 15 21:02:30.157969 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:30.157547    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579" event={"ID":"81730a25-1463-4cf0-bf25-5501fc569337","Type":"ContainerStarted","Data":"654de40eaa9655242d56f197d795b304092419f72e53ea6dbfe5afa870e2ca3f"}
Sep 15 21:02:31.457532 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:31.457498    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl"]
Sep 15 21:02:31.463506 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:31.462195    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl"
Sep 15 21:02:31.468029 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:31.467993    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 21:02:31.468189 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:31.468030    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 21:02:31.468189 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:31.468128    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 21:02:31.468301 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:31.468191    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 21:02:31.468301 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:31.468211    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wj45x\""
Sep 15 21:02:31.474952 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:31.474918    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl"]
Sep 15 21:02:31.532665 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:31.532632    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b3bd7fa2-5cec-425c-8cc1-428bd89ded5b-manifests\") pod \"opendatahub-operator-controller-manager-6f85789df9-5jrpl\" (UID: \"b3bd7fa2-5cec-425c-8cc1-428bd89ded5b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl"
Sep 15 21:02:31.532665 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:31.532674    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b3bd7fa2-5cec-425c-8cc1-428bd89ded5b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f85789df9-5jrpl\" (UID: \"b3bd7fa2-5cec-425c-8cc1-428bd89ded5b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl"
Sep 15 21:02:31.532894 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:31.532698    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3bd7fa2-5cec-425c-8cc1-428bd89ded5b-tmp\") pod \"opendatahub-operator-controller-manager-6f85789df9-5jrpl\" (UID: \"b3bd7fa2-5cec-425c-8cc1-428bd89ded5b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl"
Sep 15 21:02:31.532894 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:31.532713    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j74qn\" (UniqueName: \"kubernetes.io/projected/b3bd7fa2-5cec-425c-8cc1-428bd89ded5b-kube-api-access-j74qn\") pod \"opendatahub-operator-controller-manager-6f85789df9-5jrpl\" (UID: \"b3bd7fa2-5cec-425c-8cc1-428bd89ded5b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl"
Sep 15 21:02:31.532894 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:31.532737    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3bd7fa2-5cec-425c-8cc1-428bd89ded5b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f85789df9-5jrpl\" (UID: \"b3bd7fa2-5cec-425c-8cc1-428bd89ded5b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl"
Sep 15 21:02:31.633155 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:31.633074    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b3bd7fa2-5cec-425c-8cc1-428bd89ded5b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f85789df9-5jrpl\" (UID: \"b3bd7fa2-5cec-425c-8cc1-428bd89ded5b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl"
Sep 15 21:02:31.633155 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:31.633112    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3bd7fa2-5cec-425c-8cc1-428bd89ded5b-tmp\") pod \"opendatahub-operator-controller-manager-6f85789df9-5jrpl\" (UID: \"b3bd7fa2-5cec-425c-8cc1-428bd89ded5b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl"
Sep 15 21:02:31.633359 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:31.633233    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j74qn\" (UniqueName: \"kubernetes.io/projected/b3bd7fa2-5cec-425c-8cc1-428bd89ded5b-kube-api-access-j74qn\") pod \"opendatahub-operator-controller-manager-6f85789df9-5jrpl\" (UID: \"b3bd7fa2-5cec-425c-8cc1-428bd89ded5b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl"
Sep 15 21:02:31.633359 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:31.633273    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3bd7fa2-5cec-425c-8cc1-428bd89ded5b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f85789df9-5jrpl\" (UID: \"b3bd7fa2-5cec-425c-8cc1-428bd89ded5b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl"
Sep 15 21:02:31.633448 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:31.633368    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b3bd7fa2-5cec-425c-8cc1-428bd89ded5b-manifests\") pod \"opendatahub-operator-controller-manager-6f85789df9-5jrpl\" (UID: \"b3bd7fa2-5cec-425c-8cc1-428bd89ded5b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl"
Sep 15 21:02:31.633715 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:31.633694    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b3bd7fa2-5cec-425c-8cc1-428bd89ded5b-manifests\") pod \"opendatahub-operator-controller-manager-6f85789df9-5jrpl\" (UID: \"b3bd7fa2-5cec-425c-8cc1-428bd89ded5b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl"
Sep 15 21:02:31.635503 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:31.635469    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3bd7fa2-5cec-425c-8cc1-428bd89ded5b-tmp\") pod \"opendatahub-operator-controller-manager-6f85789df9-5jrpl\" (UID: \"b3bd7fa2-5cec-425c-8cc1-428bd89ded5b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl"
Sep 15 21:02:31.635686 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:31.635668    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3bd7fa2-5cec-425c-8cc1-428bd89ded5b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f85789df9-5jrpl\" (UID: \"b3bd7fa2-5cec-425c-8cc1-428bd89ded5b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl"
Sep 15 21:02:31.635738 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:31.635705    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b3bd7fa2-5cec-425c-8cc1-428bd89ded5b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f85789df9-5jrpl\" (UID: \"b3bd7fa2-5cec-425c-8cc1-428bd89ded5b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl"
Sep 15 21:02:31.642175 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:31.642154    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j74qn\" (UniqueName: \"kubernetes.io/projected/b3bd7fa2-5cec-425c-8cc1-428bd89ded5b-kube-api-access-j74qn\") pod \"opendatahub-operator-controller-manager-6f85789df9-5jrpl\" (UID: \"b3bd7fa2-5cec-425c-8cc1-428bd89ded5b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl"
Sep 15 21:02:31.804562 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:31.804521    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl"
Sep 15 21:02:32.083361 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:32.083330    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl"]
Sep 15 21:02:32.084066 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:02:32.084028    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3bd7fa2_5cec_425c_8cc1_428bd89ded5b.slice/crio-701cd74e41ad6a4672cc476ce5498c12ca9d71357912e9cff03a2fc137c0ed64 WatchSource:0}: Error finding container 701cd74e41ad6a4672cc476ce5498c12ca9d71357912e9cff03a2fc137c0ed64: Status 404 returned error can't find the container with id 701cd74e41ad6a4672cc476ce5498c12ca9d71357912e9cff03a2fc137c0ed64
Sep 15 21:02:32.168464 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:32.168431    2558 generic.go:356] "Generic (PLEG): container finished" podID="81730a25-1463-4cf0-bf25-5501fc569337" containerID="b897db1261976cbae508e2d700ef49440f0fe8dae002a5860ec99e28b75afb9f" exitCode=0
Sep 15 21:02:32.168656 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:32.168486    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579" event={"ID":"81730a25-1463-4cf0-bf25-5501fc569337","Type":"ContainerDied","Data":"b897db1261976cbae508e2d700ef49440f0fe8dae002a5860ec99e28b75afb9f"}
Sep 15 21:02:32.170149 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:32.170014    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl" event={"ID":"b3bd7fa2-5cec-425c-8cc1-428bd89ded5b","Type":"ContainerStarted","Data":"701cd74e41ad6a4672cc476ce5498c12ca9d71357912e9cff03a2fc137c0ed64"}
Sep 15 21:02:33.179424 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:33.179279    2558 generic.go:356] "Generic (PLEG): container finished" podID="81730a25-1463-4cf0-bf25-5501fc569337" containerID="de054791d2c0dab576eafbfb70ea5af65a271f21f83225b09f3c54ef88e3dd3d" exitCode=0
Sep 15 21:02:33.179424 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:33.179366    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579" event={"ID":"81730a25-1463-4cf0-bf25-5501fc569337","Type":"ContainerDied","Data":"de054791d2c0dab576eafbfb70ea5af65a271f21f83225b09f3c54ef88e3dd3d"}
Sep 15 21:02:35.375369 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:35.375337    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579"
Sep 15 21:02:35.464987 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:35.464963    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/81730a25-1463-4cf0-bf25-5501fc569337-bundle\") pod \"81730a25-1463-4cf0-bf25-5501fc569337\" (UID: \"81730a25-1463-4cf0-bf25-5501fc569337\") "
Sep 15 21:02:35.465149 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:35.465044    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trmgt\" (UniqueName: \"kubernetes.io/projected/81730a25-1463-4cf0-bf25-5501fc569337-kube-api-access-trmgt\") pod \"81730a25-1463-4cf0-bf25-5501fc569337\" (UID: \"81730a25-1463-4cf0-bf25-5501fc569337\") "
Sep 15 21:02:35.465149 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:35.465087    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/81730a25-1463-4cf0-bf25-5501fc569337-util\") pod \"81730a25-1463-4cf0-bf25-5501fc569337\" (UID: \"81730a25-1463-4cf0-bf25-5501fc569337\") "
Sep 15 21:02:35.465862 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:35.465836    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81730a25-1463-4cf0-bf25-5501fc569337-bundle" (OuterVolumeSpecName: "bundle") pod "81730a25-1463-4cf0-bf25-5501fc569337" (UID: "81730a25-1463-4cf0-bf25-5501fc569337"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:02:35.467105 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:35.467039    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81730a25-1463-4cf0-bf25-5501fc569337-kube-api-access-trmgt" (OuterVolumeSpecName: "kube-api-access-trmgt") pod "81730a25-1463-4cf0-bf25-5501fc569337" (UID: "81730a25-1463-4cf0-bf25-5501fc569337"). InnerVolumeSpecName "kube-api-access-trmgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:02:35.470177 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:35.470129    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81730a25-1463-4cf0-bf25-5501fc569337-util" (OuterVolumeSpecName: "util") pod "81730a25-1463-4cf0-bf25-5501fc569337" (UID: "81730a25-1463-4cf0-bf25-5501fc569337"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:02:35.566154 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:35.566115    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trmgt\" (UniqueName: \"kubernetes.io/projected/81730a25-1463-4cf0-bf25-5501fc569337-kube-api-access-trmgt\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:02:35.566154 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:35.566144    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/81730a25-1463-4cf0-bf25-5501fc569337-util\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:02:35.566578 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:35.566534    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/81730a25-1463-4cf0-bf25-5501fc569337-bundle\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:02:36.196550 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:36.196521    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579"
Sep 15 21:02:36.196550 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:36.196538    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049579" event={"ID":"81730a25-1463-4cf0-bf25-5501fc569337","Type":"ContainerDied","Data":"654de40eaa9655242d56f197d795b304092419f72e53ea6dbfe5afa870e2ca3f"}
Sep 15 21:02:36.196785 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:36.196588    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="654de40eaa9655242d56f197d795b304092419f72e53ea6dbfe5afa870e2ca3f"
Sep 15 21:02:36.198429 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:36.198323    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl" event={"ID":"b3bd7fa2-5cec-425c-8cc1-428bd89ded5b","Type":"ContainerStarted","Data":"800761f3747d08faee5379522de6d26b3cfd2c1f2ec263023db588e6535244a3"}
Sep 15 21:02:37.205562 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:37.205528    2558 generic.go:356] "Generic (PLEG): container finished" podID="b3bd7fa2-5cec-425c-8cc1-428bd89ded5b" containerID="800761f3747d08faee5379522de6d26b3cfd2c1f2ec263023db588e6535244a3" exitCode=0
Sep 15 21:02:37.205939 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:37.205620    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl" event={"ID":"b3bd7fa2-5cec-425c-8cc1-428bd89ded5b","Type":"ContainerDied","Data":"800761f3747d08faee5379522de6d26b3cfd2c1f2ec263023db588e6535244a3"}
Sep 15 21:02:38.213118 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:38.213085    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl" event={"ID":"b3bd7fa2-5cec-425c-8cc1-428bd89ded5b","Type":"ContainerStarted","Data":"7294e022254935d5c53997068c4cd456db864b6657a24f3c3c3a52f555446387"}
Sep 15 21:02:38.213486 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:38.213156    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl"
Sep 15 21:02:38.263428 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:38.263347    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl" podStartSLOduration=2.264994944 podStartE2EDuration="7.263328374s" totalImagesPullingTime="3.930782578s" totalInitContainerRuntime="1.067550852s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:02:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:02:38.248325226 +0000 UTC m=+276.738970958" watchObservedRunningTime="2026-09-15 21:02:38.263328374 +0000 UTC m=+276.753974105"
Sep 15 21:02:49.219965 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:49.219932    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6f85789df9-5jrpl"
Sep 15 21:02:51.374309 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.374276    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-85f7c7997-p8wzc"]
Sep 15 21:02:51.374727 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.374606    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81730a25-1463-4cf0-bf25-5501fc569337" containerName="util"
Sep 15 21:02:51.374727 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.374622    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="81730a25-1463-4cf0-bf25-5501fc569337" containerName="util"
Sep 15 21:02:51.374727 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.374655    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81730a25-1463-4cf0-bf25-5501fc569337" containerName="extract"
Sep 15 21:02:51.374727 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.374666    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="81730a25-1463-4cf0-bf25-5501fc569337" containerName="extract"
Sep 15 21:02:51.374727 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.374679    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81730a25-1463-4cf0-bf25-5501fc569337" containerName="pull"
Sep 15 21:02:51.374727 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.374687    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="81730a25-1463-4cf0-bf25-5501fc569337" containerName="pull"
Sep 15 21:02:51.374928 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.374752    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="81730a25-1463-4cf0-bf25-5501fc569337" containerName="extract"
Sep 15 21:02:51.384763 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.384732    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85f7c7997-p8wzc"
Sep 15 21:02:51.389214 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.389189    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-wskbm\""
Sep 15 21:02:51.390846 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.390813    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 15 21:02:51.391105 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.390886    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 15 21:02:51.391105 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.390819    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 21:02:51.391105 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.390819    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 21:02:51.393230 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.393205    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85f7c7997-p8wzc"]
Sep 15 21:02:51.493102 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.493035    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvzz4\" (UniqueName: \"kubernetes.io/projected/c1d7c96f-a35b-4aba-b0b8-5b8939ceeb04-kube-api-access-bvzz4\") pod \"kube-auth-proxy-85f7c7997-p8wzc\" (UID: \"c1d7c96f-a35b-4aba-b0b8-5b8939ceeb04\") " pod="openshift-ingress/kube-auth-proxy-85f7c7997-p8wzc"
Sep 15 21:02:51.493297 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.493123    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1d7c96f-a35b-4aba-b0b8-5b8939ceeb04-tls-certs\") pod \"kube-auth-proxy-85f7c7997-p8wzc\" (UID: \"c1d7c96f-a35b-4aba-b0b8-5b8939ceeb04\") " pod="openshift-ingress/kube-auth-proxy-85f7c7997-p8wzc"
Sep 15 21:02:51.493297 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.493157    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1d7c96f-a35b-4aba-b0b8-5b8939ceeb04-tmp\") pod \"kube-auth-proxy-85f7c7997-p8wzc\" (UID: \"c1d7c96f-a35b-4aba-b0b8-5b8939ceeb04\") " pod="openshift-ingress/kube-auth-proxy-85f7c7997-p8wzc"
Sep 15 21:02:51.594014 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.593973    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1d7c96f-a35b-4aba-b0b8-5b8939ceeb04-tmp\") pod \"kube-auth-proxy-85f7c7997-p8wzc\" (UID: \"c1d7c96f-a35b-4aba-b0b8-5b8939ceeb04\") " pod="openshift-ingress/kube-auth-proxy-85f7c7997-p8wzc"
Sep 15 21:02:51.594222 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.594033    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvzz4\" (UniqueName: \"kubernetes.io/projected/c1d7c96f-a35b-4aba-b0b8-5b8939ceeb04-kube-api-access-bvzz4\") pod \"kube-auth-proxy-85f7c7997-p8wzc\" (UID: \"c1d7c96f-a35b-4aba-b0b8-5b8939ceeb04\") " pod="openshift-ingress/kube-auth-proxy-85f7c7997-p8wzc"
Sep 15 21:02:51.594222 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.594100    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1d7c96f-a35b-4aba-b0b8-5b8939ceeb04-tls-certs\") pod \"kube-auth-proxy-85f7c7997-p8wzc\" (UID: \"c1d7c96f-a35b-4aba-b0b8-5b8939ceeb04\") " pod="openshift-ingress/kube-auth-proxy-85f7c7997-p8wzc"
Sep 15 21:02:51.596346 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.596324    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1d7c96f-a35b-4aba-b0b8-5b8939ceeb04-tmp\") pod \"kube-auth-proxy-85f7c7997-p8wzc\" (UID: \"c1d7c96f-a35b-4aba-b0b8-5b8939ceeb04\") " pod="openshift-ingress/kube-auth-proxy-85f7c7997-p8wzc"
Sep 15 21:02:51.596612 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.596593    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1d7c96f-a35b-4aba-b0b8-5b8939ceeb04-tls-certs\") pod \"kube-auth-proxy-85f7c7997-p8wzc\" (UID: \"c1d7c96f-a35b-4aba-b0b8-5b8939ceeb04\") " pod="openshift-ingress/kube-auth-proxy-85f7c7997-p8wzc"
Sep 15 21:02:51.613117 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.613084    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvzz4\" (UniqueName: \"kubernetes.io/projected/c1d7c96f-a35b-4aba-b0b8-5b8939ceeb04-kube-api-access-bvzz4\") pod \"kube-auth-proxy-85f7c7997-p8wzc\" (UID: \"c1d7c96f-a35b-4aba-b0b8-5b8939ceeb04\") " pod="openshift-ingress/kube-auth-proxy-85f7c7997-p8wzc"
Sep 15 21:02:51.695401 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.695352    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85f7c7997-p8wzc"
Sep 15 21:02:51.975756 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:51.975727    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85f7c7997-p8wzc"]
Sep 15 21:02:51.979947 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:02:51.979914    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d7c96f_a35b_4aba_b0b8_5b8939ceeb04.slice/crio-3e3188b91949367cf71a7f3eba19e60ba651ab33cf0b9e1fe4ad7af547b9ace3 WatchSource:0}: Error finding container 3e3188b91949367cf71a7f3eba19e60ba651ab33cf0b9e1fe4ad7af547b9ace3: Status 404 returned error can't find the container with id 3e3188b91949367cf71a7f3eba19e60ba651ab33cf0b9e1fe4ad7af547b9ace3
Sep 15 21:02:52.276490 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:52.276405    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85f7c7997-p8wzc" event={"ID":"c1d7c96f-a35b-4aba-b0b8-5b8939ceeb04","Type":"ContainerStarted","Data":"3e3188b91949367cf71a7f3eba19e60ba651ab33cf0b9e1fe4ad7af547b9ace3"}
Sep 15 21:02:52.902336 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:52.902299    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"]
Sep 15 21:02:52.906340 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:52.906309    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:02:52.909524 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:52.909340    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-sgljd\""
Sep 15 21:02:52.909524 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:52.909413    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 21:02:52.910083 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:52.909602    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 15 21:02:52.910083 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:52.909700    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 15 21:02:52.910083 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:52.909868    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 15 21:02:52.920232 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:52.920203    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"]
Sep 15 21:02:53.006852 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:53.006808    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lk9p\" (UniqueName: \"kubernetes.io/projected/44c864b2-f9f3-4d85-84fe-32138dedb781-kube-api-access-7lk9p\") pod \"istiod-openshift-gateway-7b7995d5cd-khnlq\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:02:53.007038 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:53.006862    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/44c864b2-f9f3-4d85-84fe-32138dedb781-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-khnlq\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:02:53.007038 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:53.006958    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/44c864b2-f9f3-4d85-84fe-32138dedb781-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-khnlq\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:02:53.007038 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:53.007003    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/44c864b2-f9f3-4d85-84fe-32138dedb781-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-khnlq\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:02:53.007038 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:53.007032    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c864b2-f9f3-4d85-84fe-32138dedb781-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-khnlq\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:02:53.007246 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:53.007083    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c864b2-f9f3-4d85-84fe-32138dedb781-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-khnlq\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:02:53.007246 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:53.007108    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/44c864b2-f9f3-4d85-84fe-32138dedb781-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-khnlq\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:02:53.108017 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:53.107971    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lk9p\" (UniqueName: \"kubernetes.io/projected/44c864b2-f9f3-4d85-84fe-32138dedb781-kube-api-access-7lk9p\") pod \"istiod-openshift-gateway-7b7995d5cd-khnlq\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:02:53.108215 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:53.108045    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/44c864b2-f9f3-4d85-84fe-32138dedb781-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-khnlq\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:02:53.108215 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:53.108111    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/44c864b2-f9f3-4d85-84fe-32138dedb781-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-khnlq\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:02:53.108215 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:53.108148    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/44c864b2-f9f3-4d85-84fe-32138dedb781-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-khnlq\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:02:53.108215 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:53.108175    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c864b2-f9f3-4d85-84fe-32138dedb781-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-khnlq\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:02:53.108215 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:53.108206    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c864b2-f9f3-4d85-84fe-32138dedb781-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-khnlq\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:02:53.108533 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:53.108233    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/44c864b2-f9f3-4d85-84fe-32138dedb781-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-khnlq\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:02:53.109149 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:53.109112    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/44c864b2-f9f3-4d85-84fe-32138dedb781-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-khnlq\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:02:53.111227 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:53.111197    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/44c864b2-f9f3-4d85-84fe-32138dedb781-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-khnlq\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:02:53.111484 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:53.111432    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/44c864b2-f9f3-4d85-84fe-32138dedb781-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-khnlq\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:02:53.111600 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:53.111497    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/44c864b2-f9f3-4d85-84fe-32138dedb781-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-khnlq\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:02:53.111600 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:53.111556    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c864b2-f9f3-4d85-84fe-32138dedb781-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-khnlq\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:02:53.117015 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:53.116987    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c864b2-f9f3-4d85-84fe-32138dedb781-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-khnlq\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:02:53.117330 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:53.117307    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lk9p\" (UniqueName: \"kubernetes.io/projected/44c864b2-f9f3-4d85-84fe-32138dedb781-kube-api-access-7lk9p\") pod \"istiod-openshift-gateway-7b7995d5cd-khnlq\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:02:53.221262 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:53.221226    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:02:54.827477 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:54.827447    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"]
Sep 15 21:02:54.828884 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:02:54.828860    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c864b2_f9f3_4d85_84fe_32138dedb781.slice/crio-7f901bdd1cce1b0ba9d060691ead9f7b8c17d9b4d6b9423d0a86e9dca3b65bd0 WatchSource:0}: Error finding container 7f901bdd1cce1b0ba9d060691ead9f7b8c17d9b4d6b9423d0a86e9dca3b65bd0: Status 404 returned error can't find the container with id 7f901bdd1cce1b0ba9d060691ead9f7b8c17d9b4d6b9423d0a86e9dca3b65bd0
Sep 15 21:02:55.293991 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:55.293958    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85f7c7997-p8wzc" event={"ID":"c1d7c96f-a35b-4aba-b0b8-5b8939ceeb04","Type":"ContainerStarted","Data":"60ddcac65218f9d38347a20720043e930a57611878ee6ce573000eb66f90fd96"}
Sep 15 21:02:55.295206 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:55.295180    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq" event={"ID":"44c864b2-f9f3-4d85-84fe-32138dedb781","Type":"ContainerStarted","Data":"7f901bdd1cce1b0ba9d060691ead9f7b8c17d9b4d6b9423d0a86e9dca3b65bd0"}
Sep 15 21:02:55.311593 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:55.311539    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-85f7c7997-p8wzc" podStartSLOduration=1.6752884510000001 podStartE2EDuration="4.311521925s" totalImagesPullingTime="2.636233474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:02:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:02:55.310592066 +0000 UTC m=+293.801237799" watchObservedRunningTime="2026-09-15 21:02:55.311521925 +0000 UTC m=+293.802167677"
Sep 15 21:02:57.667799 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:57.667748    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:02:57.668117 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:57.667835    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:02:58.317763 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:58.317709    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq" event={"ID":"44c864b2-f9f3-4d85-84fe-32138dedb781","Type":"ContainerStarted","Data":"2f93f595726a41fc47859682b5b020ec9b3dbbb41e8a73edb483d279ae2c4e22"}
Sep 15 21:02:58.319665 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:58.318364    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:02:58.320420 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:58.320390    2558 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-khnlq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 21:02:58.320536 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:58.320488    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq" podUID="44c864b2-f9f3-4d85-84fe-32138dedb781" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 21:02:58.345127 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:58.345038    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq" podStartSLOduration=3.508468115 podStartE2EDuration="6.345011819s" totalImagesPullingTime="2.836543704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:02:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:02:58.341808816 +0000 UTC m=+296.832454549" watchObservedRunningTime="2026-09-15 21:02:58.345011819 +0000 UTC m=+296.835657570"
Sep 15 21:02:59.322913 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:02:59.322875    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:03:01.872531 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:01.872506    2558 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 21:03:10.642904 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:10.642859    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5665dbb77c-wgxl8"]
Sep 15 21:03:10.648656 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:10.648643    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5665dbb77c-wgxl8"
Sep 15 21:03:10.653385 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:10.653353    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 15 21:03:10.653385 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:10.653373    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 15 21:03:10.653733 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:10.653710    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-qb8hx\""
Sep 15 21:03:10.653896 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:10.653713    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 15 21:03:10.656384 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:10.656361    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5665dbb77c-wgxl8"]
Sep 15 21:03:10.767189 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:10.767151    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e064fdeb-b201-4d8c-8164-a809e0d58b29-cert\") pod \"lws-controller-manager-5665dbb77c-wgxl8\" (UID: \"e064fdeb-b201-4d8c-8164-a809e0d58b29\") " pod="openshift-lws-operator/lws-controller-manager-5665dbb77c-wgxl8"
Sep 15 21:03:10.767189 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:10.767193    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67wm7\" (UniqueName: \"kubernetes.io/projected/e064fdeb-b201-4d8c-8164-a809e0d58b29-kube-api-access-67wm7\") pod \"lws-controller-manager-5665dbb77c-wgxl8\" (UID: \"e064fdeb-b201-4d8c-8164-a809e0d58b29\") " pod="openshift-lws-operator/lws-controller-manager-5665dbb77c-wgxl8"
Sep 15 21:03:10.767411 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:10.767220    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e064fdeb-b201-4d8c-8164-a809e0d58b29-metrics-cert\") pod \"lws-controller-manager-5665dbb77c-wgxl8\" (UID: \"e064fdeb-b201-4d8c-8164-a809e0d58b29\") " pod="openshift-lws-operator/lws-controller-manager-5665dbb77c-wgxl8"
Sep 15 21:03:10.767411 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:10.767286    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e064fdeb-b201-4d8c-8164-a809e0d58b29-manager-config\") pod \"lws-controller-manager-5665dbb77c-wgxl8\" (UID: \"e064fdeb-b201-4d8c-8164-a809e0d58b29\") " pod="openshift-lws-operator/lws-controller-manager-5665dbb77c-wgxl8"
Sep 15 21:03:10.867719 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:10.867669    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e064fdeb-b201-4d8c-8164-a809e0d58b29-cert\") pod \"lws-controller-manager-5665dbb77c-wgxl8\" (UID: \"e064fdeb-b201-4d8c-8164-a809e0d58b29\") " pod="openshift-lws-operator/lws-controller-manager-5665dbb77c-wgxl8"
Sep 15 21:03:10.867719 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:10.867721    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67wm7\" (UniqueName: \"kubernetes.io/projected/e064fdeb-b201-4d8c-8164-a809e0d58b29-kube-api-access-67wm7\") pod \"lws-controller-manager-5665dbb77c-wgxl8\" (UID: \"e064fdeb-b201-4d8c-8164-a809e0d58b29\") " pod="openshift-lws-operator/lws-controller-manager-5665dbb77c-wgxl8"
Sep 15 21:03:10.867977 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:10.867756    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e064fdeb-b201-4d8c-8164-a809e0d58b29-metrics-cert\") pod \"lws-controller-manager-5665dbb77c-wgxl8\" (UID: \"e064fdeb-b201-4d8c-8164-a809e0d58b29\") " pod="openshift-lws-operator/lws-controller-manager-5665dbb77c-wgxl8"
Sep 15 21:03:10.867977 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:10.867880    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e064fdeb-b201-4d8c-8164-a809e0d58b29-manager-config\") pod \"lws-controller-manager-5665dbb77c-wgxl8\" (UID: \"e064fdeb-b201-4d8c-8164-a809e0d58b29\") " pod="openshift-lws-operator/lws-controller-manager-5665dbb77c-wgxl8"
Sep 15 21:03:10.868544 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:10.868527    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e064fdeb-b201-4d8c-8164-a809e0d58b29-manager-config\") pod \"lws-controller-manager-5665dbb77c-wgxl8\" (UID: \"e064fdeb-b201-4d8c-8164-a809e0d58b29\") " pod="openshift-lws-operator/lws-controller-manager-5665dbb77c-wgxl8"
Sep 15 21:03:10.870288 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:10.870251    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e064fdeb-b201-4d8c-8164-a809e0d58b29-cert\") pod \"lws-controller-manager-5665dbb77c-wgxl8\" (UID: \"e064fdeb-b201-4d8c-8164-a809e0d58b29\") " pod="openshift-lws-operator/lws-controller-manager-5665dbb77c-wgxl8"
Sep 15 21:03:10.870382 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:10.870322    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e064fdeb-b201-4d8c-8164-a809e0d58b29-metrics-cert\") pod \"lws-controller-manager-5665dbb77c-wgxl8\" (UID: \"e064fdeb-b201-4d8c-8164-a809e0d58b29\") " pod="openshift-lws-operator/lws-controller-manager-5665dbb77c-wgxl8"
Sep 15 21:03:10.877362 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:10.877333    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67wm7\" (UniqueName: \"kubernetes.io/projected/e064fdeb-b201-4d8c-8164-a809e0d58b29-kube-api-access-67wm7\") pod \"lws-controller-manager-5665dbb77c-wgxl8\" (UID: \"e064fdeb-b201-4d8c-8164-a809e0d58b29\") " pod="openshift-lws-operator/lws-controller-manager-5665dbb77c-wgxl8"
Sep 15 21:03:10.959065 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:10.959021    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5665dbb77c-wgxl8"
Sep 15 21:03:11.253281 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:11.253255    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5665dbb77c-wgxl8"]
Sep 15 21:03:11.255330 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:03:11.255295    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode064fdeb_b201_4d8c_8164_a809e0d58b29.slice/crio-c7c8fa262c681fde141277632e18e86492d3e175afa118dec6dc04fffeb0a587 WatchSource:0}: Error finding container c7c8fa262c681fde141277632e18e86492d3e175afa118dec6dc04fffeb0a587: Status 404 returned error can't find the container with id c7c8fa262c681fde141277632e18e86492d3e175afa118dec6dc04fffeb0a587
Sep 15 21:03:11.257533 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:11.257508    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:03:11.377491 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:11.377453    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5665dbb77c-wgxl8" event={"ID":"e064fdeb-b201-4d8c-8164-a809e0d58b29","Type":"ContainerStarted","Data":"c7c8fa262c681fde141277632e18e86492d3e175afa118dec6dc04fffeb0a587"}
Sep 15 21:03:14.395067 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:14.395024    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5665dbb77c-wgxl8" event={"ID":"e064fdeb-b201-4d8c-8164-a809e0d58b29","Type":"ContainerStarted","Data":"3a0e3a9506b9bcd23b20cc3a28bd92f383a7b71822b846390ea5e8880a8be37f"}
Sep 15 21:03:14.395490 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:14.395182    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5665dbb77c-wgxl8"
Sep 15 21:03:14.417360 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:14.417303    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5665dbb77c-wgxl8" podStartSLOduration=1.9596444960000001 podStartE2EDuration="4.417286608s" totalImagesPullingTime="2.457642112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:03:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:03:14.414620155 +0000 UTC m=+312.905265879" watchObservedRunningTime="2026-09-15 21:03:14.417286608 +0000 UTC m=+312.907932340"
Sep 15 21:03:15.062748 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.062714    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"]
Sep 15 21:03:15.062949 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.062927    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq" podUID="44c864b2-f9f3-4d85-84fe-32138dedb781" containerName="discovery" containerID="cri-o://2f93f595726a41fc47859682b5b020ec9b3dbbb41e8a73edb483d279ae2c4e22" gracePeriod=30
Sep 15 21:03:15.305023 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.304995    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:03:15.401432 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.401335    2558 generic.go:356] "Generic (PLEG): container finished" podID="44c864b2-f9f3-4d85-84fe-32138dedb781" containerID="2f93f595726a41fc47859682b5b020ec9b3dbbb41e8a73edb483d279ae2c4e22" exitCode=0
Sep 15 21:03:15.401432 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.401379    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"
Sep 15 21:03:15.401432 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.401412    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq" event={"ID":"44c864b2-f9f3-4d85-84fe-32138dedb781","Type":"ContainerDied","Data":"2f93f595726a41fc47859682b5b020ec9b3dbbb41e8a73edb483d279ae2c4e22"}
Sep 15 21:03:15.401941 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.401463    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq" event={"ID":"44c864b2-f9f3-4d85-84fe-32138dedb781","Type":"ContainerDied","Data":"7f901bdd1cce1b0ba9d060691ead9f7b8c17d9b4d6b9423d0a86e9dca3b65bd0"}
Sep 15 21:03:15.401941 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.401488    2558 scope.go:117] "RemoveContainer" containerID="2f93f595726a41fc47859682b5b020ec9b3dbbb41e8a73edb483d279ae2c4e22"
Sep 15 21:03:15.408081 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.408038    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/44c864b2-f9f3-4d85-84fe-32138dedb781-istio-kubeconfig\") pod \"44c864b2-f9f3-4d85-84fe-32138dedb781\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") "
Sep 15 21:03:15.408217 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.408108    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c864b2-f9f3-4d85-84fe-32138dedb781-istio-token\") pod \"44c864b2-f9f3-4d85-84fe-32138dedb781\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") "
Sep 15 21:03:15.408217 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.408132    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/44c864b2-f9f3-4d85-84fe-32138dedb781-cacerts\") pod \"44c864b2-f9f3-4d85-84fe-32138dedb781\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") "
Sep 15 21:03:15.408217 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.408148    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/44c864b2-f9f3-4d85-84fe-32138dedb781-istio-csr-dns-cert\") pod \"44c864b2-f9f3-4d85-84fe-32138dedb781\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") "
Sep 15 21:03:15.408217 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.408167    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c864b2-f9f3-4d85-84fe-32138dedb781-local-certs\") pod \"44c864b2-f9f3-4d85-84fe-32138dedb781\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") "
Sep 15 21:03:15.408217 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.408201    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/44c864b2-f9f3-4d85-84fe-32138dedb781-istio-csr-ca-configmap\") pod \"44c864b2-f9f3-4d85-84fe-32138dedb781\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") "
Sep 15 21:03:15.408629 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.408528    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lk9p\" (UniqueName: \"kubernetes.io/projected/44c864b2-f9f3-4d85-84fe-32138dedb781-kube-api-access-7lk9p\") pod \"44c864b2-f9f3-4d85-84fe-32138dedb781\" (UID: \"44c864b2-f9f3-4d85-84fe-32138dedb781\") "
Sep 15 21:03:15.408629 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.408563    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44c864b2-f9f3-4d85-84fe-32138dedb781-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "44c864b2-f9f3-4d85-84fe-32138dedb781" (UID: "44c864b2-f9f3-4d85-84fe-32138dedb781"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:03:15.408863 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.408832    2558 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/44c864b2-f9f3-4d85-84fe-32138dedb781-istio-csr-ca-configmap\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:03:15.410755 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.410704    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c864b2-f9f3-4d85-84fe-32138dedb781-local-certs" (OuterVolumeSpecName: "local-certs") pod "44c864b2-f9f3-4d85-84fe-32138dedb781" (UID: "44c864b2-f9f3-4d85-84fe-32138dedb781"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:03:15.410755 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.410718    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44c864b2-f9f3-4d85-84fe-32138dedb781-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "44c864b2-f9f3-4d85-84fe-32138dedb781" (UID: "44c864b2-f9f3-4d85-84fe-32138dedb781"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:03:15.411002 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.410972    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44c864b2-f9f3-4d85-84fe-32138dedb781-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "44c864b2-f9f3-4d85-84fe-32138dedb781" (UID: "44c864b2-f9f3-4d85-84fe-32138dedb781"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:03:15.411137 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.411015    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44c864b2-f9f3-4d85-84fe-32138dedb781-cacerts" (OuterVolumeSpecName: "cacerts") pod "44c864b2-f9f3-4d85-84fe-32138dedb781" (UID: "44c864b2-f9f3-4d85-84fe-32138dedb781"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:03:15.411137 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.411037    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c864b2-f9f3-4d85-84fe-32138dedb781-istio-token" (OuterVolumeSpecName: "istio-token") pod "44c864b2-f9f3-4d85-84fe-32138dedb781" (UID: "44c864b2-f9f3-4d85-84fe-32138dedb781"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:03:15.411137 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.411072    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c864b2-f9f3-4d85-84fe-32138dedb781-kube-api-access-7lk9p" (OuterVolumeSpecName: "kube-api-access-7lk9p") pod "44c864b2-f9f3-4d85-84fe-32138dedb781" (UID: "44c864b2-f9f3-4d85-84fe-32138dedb781"). InnerVolumeSpecName "kube-api-access-7lk9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:03:15.411402 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.411384    2558 scope.go:117] "RemoveContainer" containerID="2f93f595726a41fc47859682b5b020ec9b3dbbb41e8a73edb483d279ae2c4e22"
Sep 15 21:03:15.411727 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:03:15.411704    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f93f595726a41fc47859682b5b020ec9b3dbbb41e8a73edb483d279ae2c4e22\": container with ID starting with 2f93f595726a41fc47859682b5b020ec9b3dbbb41e8a73edb483d279ae2c4e22 not found: ID does not exist" containerID="2f93f595726a41fc47859682b5b020ec9b3dbbb41e8a73edb483d279ae2c4e22"
Sep 15 21:03:15.411794 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.411736    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f93f595726a41fc47859682b5b020ec9b3dbbb41e8a73edb483d279ae2c4e22"} err="failed to get container status \"2f93f595726a41fc47859682b5b020ec9b3dbbb41e8a73edb483d279ae2c4e22\": rpc error: code = NotFound desc = could not find container \"2f93f595726a41fc47859682b5b020ec9b3dbbb41e8a73edb483d279ae2c4e22\": container with ID starting with 2f93f595726a41fc47859682b5b020ec9b3dbbb41e8a73edb483d279ae2c4e22 not found: ID does not exist"
Sep 15 21:03:15.510126 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.510087    2558 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/44c864b2-f9f3-4d85-84fe-32138dedb781-istio-kubeconfig\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:03:15.510126 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.510125    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c864b2-f9f3-4d85-84fe-32138dedb781-istio-token\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:03:15.510126 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.510134    2558 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/44c864b2-f9f3-4d85-84fe-32138dedb781-cacerts\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:03:15.510126 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.510142    2558 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/44c864b2-f9f3-4d85-84fe-32138dedb781-istio-csr-dns-cert\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:03:15.510455 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.510152    2558 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c864b2-f9f3-4d85-84fe-32138dedb781-local-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:03:15.510455 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.510162    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lk9p\" (UniqueName: \"kubernetes.io/projected/44c864b2-f9f3-4d85-84fe-32138dedb781-kube-api-access-7lk9p\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:03:15.725442 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.725396    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"]
Sep 15 21:03:15.729736 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.729704    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-khnlq"]
Sep 15 21:03:15.941653 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:15.941617    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44c864b2-f9f3-4d85-84fe-32138dedb781" path="/var/lib/kubelet/pods/44c864b2-f9f3-4d85-84fe-32138dedb781/volumes"
Sep 15 21:03:25.404063 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:03:25.404018    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5665dbb77c-wgxl8"
Sep 15 21:04:07.132567 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.132529    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr"]
Sep 15 21:04:07.133072 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.132970    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44c864b2-f9f3-4d85-84fe-32138dedb781" containerName="discovery"
Sep 15 21:04:07.133072 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.132988    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="44c864b2-f9f3-4d85-84fe-32138dedb781" containerName="discovery"
Sep 15 21:04:07.133219 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.133087    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="44c864b2-f9f3-4d85-84fe-32138dedb781" containerName="discovery"
Sep 15 21:04:07.137377 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.137358    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr"
Sep 15 21:04:07.140285 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.140254    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 21:04:07.140386 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.140362    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 21:04:07.140431 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.140403    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6r6f2\""
Sep 15 21:04:07.145656 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.145635    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr"]
Sep 15 21:04:07.221652 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.221624    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0461793-7189-47d1-85a6-5e768b2e91e9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr\" (UID: \"a0461793-7189-47d1-85a6-5e768b2e91e9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr"
Sep 15 21:04:07.221773 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.221704    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwztm\" (UniqueName: \"kubernetes.io/projected/a0461793-7189-47d1-85a6-5e768b2e91e9-kube-api-access-wwztm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr\" (UID: \"a0461793-7189-47d1-85a6-5e768b2e91e9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr"
Sep 15 21:04:07.221773 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.221737    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0461793-7189-47d1-85a6-5e768b2e91e9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr\" (UID: \"a0461793-7189-47d1-85a6-5e768b2e91e9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr"
Sep 15 21:04:07.244639 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.244614    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv"]
Sep 15 21:04:07.247899 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.247885    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv"
Sep 15 21:04:07.256256 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.256234    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv"]
Sep 15 21:04:07.323165 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.323133    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0461793-7189-47d1-85a6-5e768b2e91e9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr\" (UID: \"a0461793-7189-47d1-85a6-5e768b2e91e9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr"
Sep 15 21:04:07.323304 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.323212    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwztm\" (UniqueName: \"kubernetes.io/projected/a0461793-7189-47d1-85a6-5e768b2e91e9-kube-api-access-wwztm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr\" (UID: \"a0461793-7189-47d1-85a6-5e768b2e91e9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr"
Sep 15 21:04:07.323304 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.323235    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0461793-7189-47d1-85a6-5e768b2e91e9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr\" (UID: \"a0461793-7189-47d1-85a6-5e768b2e91e9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr"
Sep 15 21:04:07.323506 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.323487    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0461793-7189-47d1-85a6-5e768b2e91e9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr\" (UID: \"a0461793-7189-47d1-85a6-5e768b2e91e9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr"
Sep 15 21:04:07.323566 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.323549    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0461793-7189-47d1-85a6-5e768b2e91e9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr\" (UID: \"a0461793-7189-47d1-85a6-5e768b2e91e9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr"
Sep 15 21:04:07.331662 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.331642    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49"]
Sep 15 21:04:07.334177 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.334154    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwztm\" (UniqueName: \"kubernetes.io/projected/a0461793-7189-47d1-85a6-5e768b2e91e9-kube-api-access-wwztm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr\" (UID: \"a0461793-7189-47d1-85a6-5e768b2e91e9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr"
Sep 15 21:04:07.335020 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.334997    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49"
Sep 15 21:04:07.342876 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.342852    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49"]
Sep 15 21:04:07.424428 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.424404    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2b3a59d-d514-48dd-9dfc-724a95a0cb0e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49\" (UID: \"e2b3a59d-d514-48dd-9dfc-724a95a0cb0e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49"
Sep 15 21:04:07.424565 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.424440    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/134507c4-f77f-45ba-abd6-efc840d68fc0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv\" (UID: \"134507c4-f77f-45ba-abd6-efc840d68fc0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv"
Sep 15 21:04:07.424565 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.424478    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/134507c4-f77f-45ba-abd6-efc840d68fc0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv\" (UID: \"134507c4-f77f-45ba-abd6-efc840d68fc0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv"
Sep 15 21:04:07.424565 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.424530    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmqh2\" (UniqueName: \"kubernetes.io/projected/134507c4-f77f-45ba-abd6-efc840d68fc0-kube-api-access-nmqh2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv\" (UID: \"134507c4-f77f-45ba-abd6-efc840d68fc0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv"
Sep 15 21:04:07.424721 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.424607    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsrbx\" (UniqueName: \"kubernetes.io/projected/e2b3a59d-d514-48dd-9dfc-724a95a0cb0e-kube-api-access-nsrbx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49\" (UID: \"e2b3a59d-d514-48dd-9dfc-724a95a0cb0e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49"
Sep 15 21:04:07.424721 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.424646    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2b3a59d-d514-48dd-9dfc-724a95a0cb0e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49\" (UID: \"e2b3a59d-d514-48dd-9dfc-724a95a0cb0e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49"
Sep 15 21:04:07.441389 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.441364    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp"]
Sep 15 21:04:07.444721 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.444700    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp"
Sep 15 21:04:07.446876 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.446857    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr"
Sep 15 21:04:07.454617 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.454595    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp"]
Sep 15 21:04:07.525617 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.525590    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2b3a59d-d514-48dd-9dfc-724a95a0cb0e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49\" (UID: \"e2b3a59d-d514-48dd-9dfc-724a95a0cb0e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49"
Sep 15 21:04:07.525761 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.525631    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/134507c4-f77f-45ba-abd6-efc840d68fc0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv\" (UID: \"134507c4-f77f-45ba-abd6-efc840d68fc0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv"
Sep 15 21:04:07.525761 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.525679    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/134507c4-f77f-45ba-abd6-efc840d68fc0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv\" (UID: \"134507c4-f77f-45ba-abd6-efc840d68fc0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv"
Sep 15 21:04:07.525761 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.525722    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmqh2\" (UniqueName: \"kubernetes.io/projected/134507c4-f77f-45ba-abd6-efc840d68fc0-kube-api-access-nmqh2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv\" (UID: \"134507c4-f77f-45ba-abd6-efc840d68fc0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv"
Sep 15 21:04:07.525882 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.525766    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsrbx\" (UniqueName: \"kubernetes.io/projected/e2b3a59d-d514-48dd-9dfc-724a95a0cb0e-kube-api-access-nsrbx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49\" (UID: \"e2b3a59d-d514-48dd-9dfc-724a95a0cb0e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49"
Sep 15 21:04:07.525882 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.525793    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2b3a59d-d514-48dd-9dfc-724a95a0cb0e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49\" (UID: \"e2b3a59d-d514-48dd-9dfc-724a95a0cb0e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49"
Sep 15 21:04:07.526104 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.526081    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/134507c4-f77f-45ba-abd6-efc840d68fc0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv\" (UID: \"134507c4-f77f-45ba-abd6-efc840d68fc0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv"
Sep 15 21:04:07.526207 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.526098    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2b3a59d-d514-48dd-9dfc-724a95a0cb0e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49\" (UID: \"e2b3a59d-d514-48dd-9dfc-724a95a0cb0e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49"
Sep 15 21:04:07.526207 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.526107    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2b3a59d-d514-48dd-9dfc-724a95a0cb0e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49\" (UID: \"e2b3a59d-d514-48dd-9dfc-724a95a0cb0e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49"
Sep 15 21:04:07.526207 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.526136    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/134507c4-f77f-45ba-abd6-efc840d68fc0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv\" (UID: \"134507c4-f77f-45ba-abd6-efc840d68fc0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv"
Sep 15 21:04:07.534043 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.534020    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmqh2\" (UniqueName: \"kubernetes.io/projected/134507c4-f77f-45ba-abd6-efc840d68fc0-kube-api-access-nmqh2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv\" (UID: \"134507c4-f77f-45ba-abd6-efc840d68fc0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv"
Sep 15 21:04:07.534875 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.534855    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsrbx\" (UniqueName: \"kubernetes.io/projected/e2b3a59d-d514-48dd-9dfc-724a95a0cb0e-kube-api-access-nsrbx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49\" (UID: \"e2b3a59d-d514-48dd-9dfc-724a95a0cb0e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49"
Sep 15 21:04:07.556781 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.556760    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv"
Sep 15 21:04:07.626789 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.626762    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a77cfe3f-3f07-4074-b11a-45e7bc993ba7-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp\" (UID: \"a77cfe3f-3f07-4074-b11a-45e7bc993ba7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp"
Sep 15 21:04:07.626923 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.626818    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j2ds\" (UniqueName: \"kubernetes.io/projected/a77cfe3f-3f07-4074-b11a-45e7bc993ba7-kube-api-access-6j2ds\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp\" (UID: \"a77cfe3f-3f07-4074-b11a-45e7bc993ba7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp"
Sep 15 21:04:07.626923 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.626870    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a77cfe3f-3f07-4074-b11a-45e7bc993ba7-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp\" (UID: \"a77cfe3f-3f07-4074-b11a-45e7bc993ba7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp"
Sep 15 21:04:07.649713 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.649690    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49"
Sep 15 21:04:07.725813 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.725786    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr"]
Sep 15 21:04:07.728043 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.728018    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j2ds\" (UniqueName: \"kubernetes.io/projected/a77cfe3f-3f07-4074-b11a-45e7bc993ba7-kube-api-access-6j2ds\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp\" (UID: \"a77cfe3f-3f07-4074-b11a-45e7bc993ba7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp"
Sep 15 21:04:07.728171 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.728073    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a77cfe3f-3f07-4074-b11a-45e7bc993ba7-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp\" (UID: \"a77cfe3f-3f07-4074-b11a-45e7bc993ba7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp"
Sep 15 21:04:07.728171 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.728157    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a77cfe3f-3f07-4074-b11a-45e7bc993ba7-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp\" (UID: \"a77cfe3f-3f07-4074-b11a-45e7bc993ba7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp"
Sep 15 21:04:07.728510 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.728491    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a77cfe3f-3f07-4074-b11a-45e7bc993ba7-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp\" (UID: \"a77cfe3f-3f07-4074-b11a-45e7bc993ba7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp"
Sep 15 21:04:07.728589 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.728525    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a77cfe3f-3f07-4074-b11a-45e7bc993ba7-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp\" (UID: \"a77cfe3f-3f07-4074-b11a-45e7bc993ba7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp"
Sep 15 21:04:07.728646 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:04:07.728582    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0461793_7189_47d1_85a6_5e768b2e91e9.slice/crio-f2a4ae4a01b26e27da9d5b8244cc6cd5bc8d08d7eadebae78df42f25efbaf2c9 WatchSource:0}: Error finding container f2a4ae4a01b26e27da9d5b8244cc6cd5bc8d08d7eadebae78df42f25efbaf2c9: Status 404 returned error can't find the container with id f2a4ae4a01b26e27da9d5b8244cc6cd5bc8d08d7eadebae78df42f25efbaf2c9
Sep 15 21:04:07.737709 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.737686    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j2ds\" (UniqueName: \"kubernetes.io/projected/a77cfe3f-3f07-4074-b11a-45e7bc993ba7-kube-api-access-6j2ds\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp\" (UID: \"a77cfe3f-3f07-4074-b11a-45e7bc993ba7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp"
Sep 15 21:04:07.754903 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.754882    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp"
Sep 15 21:04:07.828948 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.828853    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv"]
Sep 15 21:04:07.830327 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:04:07.830301    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134507c4_f77f_45ba_abd6_efc840d68fc0.slice/crio-eeac81339b8c263e1f0cfb919414e4111703666d5f0ce4d97404cfe68036d75d WatchSource:0}: Error finding container eeac81339b8c263e1f0cfb919414e4111703666d5f0ce4d97404cfe68036d75d: Status 404 returned error can't find the container with id eeac81339b8c263e1f0cfb919414e4111703666d5f0ce4d97404cfe68036d75d
Sep 15 21:04:07.920039 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:07.919878    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49"]
Sep 15 21:04:07.922608 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:04:07.922582    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b3a59d_d514_48dd_9dfc_724a95a0cb0e.slice/crio-cfa2c29df546cf2c067e6f6709d0592ff43bbd8d961a53fe897fe970160a929c WatchSource:0}: Error finding container cfa2c29df546cf2c067e6f6709d0592ff43bbd8d961a53fe897fe970160a929c: Status 404 returned error can't find the container with id cfa2c29df546cf2c067e6f6709d0592ff43bbd8d961a53fe897fe970160a929c
Sep 15 21:04:08.033661 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:08.033165    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp"]
Sep 15 21:04:08.654572 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:08.654536    2558 generic.go:356] "Generic (PLEG): container finished" podID="a77cfe3f-3f07-4074-b11a-45e7bc993ba7" containerID="7117ec3ac1cb7e17af0c72bbf625d9283e4b6214725b2701849f38ceb34dd011" exitCode=0
Sep 15 21:04:08.654959 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:08.654616    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp" event={"ID":"a77cfe3f-3f07-4074-b11a-45e7bc993ba7","Type":"ContainerDied","Data":"7117ec3ac1cb7e17af0c72bbf625d9283e4b6214725b2701849f38ceb34dd011"}
Sep 15 21:04:08.654959 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:08.654665    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp" event={"ID":"a77cfe3f-3f07-4074-b11a-45e7bc993ba7","Type":"ContainerStarted","Data":"9d86880221323f1ed6330c535eed4e8910b4132be41334e9f5f1ad63b56be4ab"}
Sep 15 21:04:08.656265 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:08.656246    2558 generic.go:356] "Generic (PLEG): container finished" podID="134507c4-f77f-45ba-abd6-efc840d68fc0" containerID="cbf4e4ecfb9458cfd20e0cf3db704251386b9c7370f9ca70aceddb6ddf1b217f" exitCode=0
Sep 15 21:04:08.656353 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:08.656275    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv" event={"ID":"134507c4-f77f-45ba-abd6-efc840d68fc0","Type":"ContainerDied","Data":"cbf4e4ecfb9458cfd20e0cf3db704251386b9c7370f9ca70aceddb6ddf1b217f"}
Sep 15 21:04:08.656353 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:08.656323    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv" event={"ID":"134507c4-f77f-45ba-abd6-efc840d68fc0","Type":"ContainerStarted","Data":"eeac81339b8c263e1f0cfb919414e4111703666d5f0ce4d97404cfe68036d75d"}
Sep 15 21:04:08.657872 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:08.657852    2558 generic.go:356] "Generic (PLEG): container finished" podID="a0461793-7189-47d1-85a6-5e768b2e91e9" containerID="0a83b2bc395da5e45193ae291178562f71084c316ef067d216fe26f265e704ec" exitCode=0
Sep 15 21:04:08.657943 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:08.657920    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr" event={"ID":"a0461793-7189-47d1-85a6-5e768b2e91e9","Type":"ContainerDied","Data":"0a83b2bc395da5e45193ae291178562f71084c316ef067d216fe26f265e704ec"}
Sep 15 21:04:08.658000 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:08.657968    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr" event={"ID":"a0461793-7189-47d1-85a6-5e768b2e91e9","Type":"ContainerStarted","Data":"f2a4ae4a01b26e27da9d5b8244cc6cd5bc8d08d7eadebae78df42f25efbaf2c9"}
Sep 15 21:04:08.659622 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:08.659600    2558 generic.go:356] "Generic (PLEG): container finished" podID="e2b3a59d-d514-48dd-9dfc-724a95a0cb0e" containerID="ffeab93ce16d4df50b9e4c3b3c24dec072fd9fb9c0e80542ed247a4a87622280" exitCode=0
Sep 15 21:04:08.659704 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:08.659634    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49" event={"ID":"e2b3a59d-d514-48dd-9dfc-724a95a0cb0e","Type":"ContainerDied","Data":"ffeab93ce16d4df50b9e4c3b3c24dec072fd9fb9c0e80542ed247a4a87622280"}
Sep 15 21:04:08.659704 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:08.659693    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49" event={"ID":"e2b3a59d-d514-48dd-9dfc-724a95a0cb0e","Type":"ContainerStarted","Data":"cfa2c29df546cf2c067e6f6709d0592ff43bbd8d961a53fe897fe970160a929c"}
Sep 15 21:04:10.670743 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:10.670713    2558 generic.go:356] "Generic (PLEG): container finished" podID="134507c4-f77f-45ba-abd6-efc840d68fc0" containerID="217815b210d4fa5a689b225155fc1456e43a25f4537a0952cb8c71cdbdcbe693" exitCode=0
Sep 15 21:04:10.671153 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:10.670793    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv" event={"ID":"134507c4-f77f-45ba-abd6-efc840d68fc0","Type":"ContainerDied","Data":"217815b210d4fa5a689b225155fc1456e43a25f4537a0952cb8c71cdbdcbe693"}
Sep 15 21:04:10.672699 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:10.672675    2558 generic.go:356] "Generic (PLEG): container finished" podID="a0461793-7189-47d1-85a6-5e768b2e91e9" containerID="235c52a6d7fe8a9ca06c8c0e181e5d2113d16afbe14d4e74aa384ba8d75a488d" exitCode=0
Sep 15 21:04:10.672791 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:10.672744    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr" event={"ID":"a0461793-7189-47d1-85a6-5e768b2e91e9","Type":"ContainerDied","Data":"235c52a6d7fe8a9ca06c8c0e181e5d2113d16afbe14d4e74aa384ba8d75a488d"}
Sep 15 21:04:10.674847 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:10.674819    2558 generic.go:356] "Generic (PLEG): container finished" podID="e2b3a59d-d514-48dd-9dfc-724a95a0cb0e" containerID="a1721c5b976b7e6062422ded4b16334a5a673634e9b5d668c7ff021158a607da" exitCode=0
Sep 15 21:04:10.674921 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:10.674865    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49" event={"ID":"e2b3a59d-d514-48dd-9dfc-724a95a0cb0e","Type":"ContainerDied","Data":"a1721c5b976b7e6062422ded4b16334a5a673634e9b5d668c7ff021158a607da"}
Sep 15 21:04:10.676964 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:10.676944    2558 generic.go:356] "Generic (PLEG): container finished" podID="a77cfe3f-3f07-4074-b11a-45e7bc993ba7" containerID="6fe8b78ca337735b35d3217a1142b14eb1ef03d497563317a5222179cfe12dfa" exitCode=0
Sep 15 21:04:10.677036 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:10.676973    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp" event={"ID":"a77cfe3f-3f07-4074-b11a-45e7bc993ba7","Type":"ContainerDied","Data":"6fe8b78ca337735b35d3217a1142b14eb1ef03d497563317a5222179cfe12dfa"}
Sep 15 21:04:11.684091 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:11.684036    2558 generic.go:356] "Generic (PLEG): container finished" podID="134507c4-f77f-45ba-abd6-efc840d68fc0" containerID="faac5ef56ff0ec519dc50557d3f419b280e6411ae00f0eb71fd7384e3a85c6b1" exitCode=0
Sep 15 21:04:11.684468 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:11.684122    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv" event={"ID":"134507c4-f77f-45ba-abd6-efc840d68fc0","Type":"ContainerDied","Data":"faac5ef56ff0ec519dc50557d3f419b280e6411ae00f0eb71fd7384e3a85c6b1"}
Sep 15 21:04:11.686161 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:11.686136    2558 generic.go:356] "Generic (PLEG): container finished" podID="a0461793-7189-47d1-85a6-5e768b2e91e9" containerID="4d8ce234a371975ecc0ee433ab2406552722e95b96bba6efeb231e01783f65ce" exitCode=0
Sep 15 21:04:11.686291 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:11.686215    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr" event={"ID":"a0461793-7189-47d1-85a6-5e768b2e91e9","Type":"ContainerDied","Data":"4d8ce234a371975ecc0ee433ab2406552722e95b96bba6efeb231e01783f65ce"}
Sep 15 21:04:11.688179 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:11.688159    2558 generic.go:356] "Generic (PLEG): container finished" podID="e2b3a59d-d514-48dd-9dfc-724a95a0cb0e" containerID="41515b194c7365171d5bf846c41c0412d623d66d776b7a0c616c55ed66815f80" exitCode=0
Sep 15 21:04:11.688303 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:11.688232    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49" event={"ID":"e2b3a59d-d514-48dd-9dfc-724a95a0cb0e","Type":"ContainerDied","Data":"41515b194c7365171d5bf846c41c0412d623d66d776b7a0c616c55ed66815f80"}
Sep 15 21:04:11.690280 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:11.690255    2558 generic.go:356] "Generic (PLEG): container finished" podID="a77cfe3f-3f07-4074-b11a-45e7bc993ba7" containerID="677d0e9abd875c978a97d2c6d4f237a642ddfbdba4db986400317d803c68bd28" exitCode=0
Sep 15 21:04:11.690366 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:11.690289    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp" event={"ID":"a77cfe3f-3f07-4074-b11a-45e7bc993ba7","Type":"ContainerDied","Data":"677d0e9abd875c978a97d2c6d4f237a642ddfbdba4db986400317d803c68bd28"}
Sep 15 21:04:12.831021 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.830812    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr"
Sep 15 21:04:12.871962 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.871933    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49"
Sep 15 21:04:12.878471 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.878447    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2b3a59d-d514-48dd-9dfc-724a95a0cb0e-util\") pod \"e2b3a59d-d514-48dd-9dfc-724a95a0cb0e\" (UID: \"e2b3a59d-d514-48dd-9dfc-724a95a0cb0e\") "
Sep 15 21:04:12.878588 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.878488    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwztm\" (UniqueName: \"kubernetes.io/projected/a0461793-7189-47d1-85a6-5e768b2e91e9-kube-api-access-wwztm\") pod \"a0461793-7189-47d1-85a6-5e768b2e91e9\" (UID: \"a0461793-7189-47d1-85a6-5e768b2e91e9\") "
Sep 15 21:04:12.878588 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.878514    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0461793-7189-47d1-85a6-5e768b2e91e9-util\") pod \"a0461793-7189-47d1-85a6-5e768b2e91e9\" (UID: \"a0461793-7189-47d1-85a6-5e768b2e91e9\") "
Sep 15 21:04:12.878588 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.878558    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2b3a59d-d514-48dd-9dfc-724a95a0cb0e-bundle\") pod \"e2b3a59d-d514-48dd-9dfc-724a95a0cb0e\" (UID: \"e2b3a59d-d514-48dd-9dfc-724a95a0cb0e\") "
Sep 15 21:04:12.878751 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.878605    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0461793-7189-47d1-85a6-5e768b2e91e9-bundle\") pod \"a0461793-7189-47d1-85a6-5e768b2e91e9\" (UID: \"a0461793-7189-47d1-85a6-5e768b2e91e9\") "
Sep 15 21:04:12.878751 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.878633    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsrbx\" (UniqueName: \"kubernetes.io/projected/e2b3a59d-d514-48dd-9dfc-724a95a0cb0e-kube-api-access-nsrbx\") pod \"e2b3a59d-d514-48dd-9dfc-724a95a0cb0e\" (UID: \"e2b3a59d-d514-48dd-9dfc-724a95a0cb0e\") "
Sep 15 21:04:12.879209 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.879166    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b3a59d-d514-48dd-9dfc-724a95a0cb0e-bundle" (OuterVolumeSpecName: "bundle") pod "e2b3a59d-d514-48dd-9dfc-724a95a0cb0e" (UID: "e2b3a59d-d514-48dd-9dfc-724a95a0cb0e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:04:12.879429 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.879385    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0461793-7189-47d1-85a6-5e768b2e91e9-bundle" (OuterVolumeSpecName: "bundle") pod "a0461793-7189-47d1-85a6-5e768b2e91e9" (UID: "a0461793-7189-47d1-85a6-5e768b2e91e9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:04:12.882182 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.882100    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b3a59d-d514-48dd-9dfc-724a95a0cb0e-kube-api-access-nsrbx" (OuterVolumeSpecName: "kube-api-access-nsrbx") pod "e2b3a59d-d514-48dd-9dfc-724a95a0cb0e" (UID: "e2b3a59d-d514-48dd-9dfc-724a95a0cb0e"). InnerVolumeSpecName "kube-api-access-nsrbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:04:12.884447 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.884423    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0461793-7189-47d1-85a6-5e768b2e91e9-kube-api-access-wwztm" (OuterVolumeSpecName: "kube-api-access-wwztm") pod "a0461793-7189-47d1-85a6-5e768b2e91e9" (UID: "a0461793-7189-47d1-85a6-5e768b2e91e9"). InnerVolumeSpecName "kube-api-access-wwztm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:04:12.887819 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.887793    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b3a59d-d514-48dd-9dfc-724a95a0cb0e-util" (OuterVolumeSpecName: "util") pod "e2b3a59d-d514-48dd-9dfc-724a95a0cb0e" (UID: "e2b3a59d-d514-48dd-9dfc-724a95a0cb0e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:04:12.887891 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.887837    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0461793-7189-47d1-85a6-5e768b2e91e9-util" (OuterVolumeSpecName: "util") pod "a0461793-7189-47d1-85a6-5e768b2e91e9" (UID: "a0461793-7189-47d1-85a6-5e768b2e91e9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:04:12.900337 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.900312    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv"
Sep 15 21:04:12.904267 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.904250    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp"
Sep 15 21:04:12.979140 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.979069    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a77cfe3f-3f07-4074-b11a-45e7bc993ba7-util\") pod \"a77cfe3f-3f07-4074-b11a-45e7bc993ba7\" (UID: \"a77cfe3f-3f07-4074-b11a-45e7bc993ba7\") "
Sep 15 21:04:12.979140 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.979131    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmqh2\" (UniqueName: \"kubernetes.io/projected/134507c4-f77f-45ba-abd6-efc840d68fc0-kube-api-access-nmqh2\") pod \"134507c4-f77f-45ba-abd6-efc840d68fc0\" (UID: \"134507c4-f77f-45ba-abd6-efc840d68fc0\") "
Sep 15 21:04:12.979302 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.979155    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j2ds\" (UniqueName: \"kubernetes.io/projected/a77cfe3f-3f07-4074-b11a-45e7bc993ba7-kube-api-access-6j2ds\") pod \"a77cfe3f-3f07-4074-b11a-45e7bc993ba7\" (UID: \"a77cfe3f-3f07-4074-b11a-45e7bc993ba7\") "
Sep 15 21:04:12.979302 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.979216    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/134507c4-f77f-45ba-abd6-efc840d68fc0-bundle\") pod \"134507c4-f77f-45ba-abd6-efc840d68fc0\" (UID: \"134507c4-f77f-45ba-abd6-efc840d68fc0\") "
Sep 15 21:04:12.979302 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.979254    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/134507c4-f77f-45ba-abd6-efc840d68fc0-util\") pod \"134507c4-f77f-45ba-abd6-efc840d68fc0\" (UID: \"134507c4-f77f-45ba-abd6-efc840d68fc0\") "
Sep 15 21:04:12.979302 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.979285    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a77cfe3f-3f07-4074-b11a-45e7bc993ba7-bundle\") pod \"a77cfe3f-3f07-4074-b11a-45e7bc993ba7\" (UID: \"a77cfe3f-3f07-4074-b11a-45e7bc993ba7\") "
Sep 15 21:04:12.979531 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.979512    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2b3a59d-d514-48dd-9dfc-724a95a0cb0e-util\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:04:12.979591 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.979538    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwztm\" (UniqueName: \"kubernetes.io/projected/a0461793-7189-47d1-85a6-5e768b2e91e9-kube-api-access-wwztm\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:04:12.979591 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.979552    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0461793-7189-47d1-85a6-5e768b2e91e9-util\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:04:12.979591 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.979564    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2b3a59d-d514-48dd-9dfc-724a95a0cb0e-bundle\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:04:12.979591 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.979577    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0461793-7189-47d1-85a6-5e768b2e91e9-bundle\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:04:12.979591 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.979590    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsrbx\" (UniqueName: \"kubernetes.io/projected/e2b3a59d-d514-48dd-9dfc-724a95a0cb0e-kube-api-access-nsrbx\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:04:12.980008 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.979977    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/134507c4-f77f-45ba-abd6-efc840d68fc0-bundle" (OuterVolumeSpecName: "bundle") pod "134507c4-f77f-45ba-abd6-efc840d68fc0" (UID: "134507c4-f77f-45ba-abd6-efc840d68fc0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:04:12.980128 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.980032    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77cfe3f-3f07-4074-b11a-45e7bc993ba7-bundle" (OuterVolumeSpecName: "bundle") pod "a77cfe3f-3f07-4074-b11a-45e7bc993ba7" (UID: "a77cfe3f-3f07-4074-b11a-45e7bc993ba7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:04:12.981855 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.981831    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77cfe3f-3f07-4074-b11a-45e7bc993ba7-kube-api-access-6j2ds" (OuterVolumeSpecName: "kube-api-access-6j2ds") pod "a77cfe3f-3f07-4074-b11a-45e7bc993ba7" (UID: "a77cfe3f-3f07-4074-b11a-45e7bc993ba7"). InnerVolumeSpecName "kube-api-access-6j2ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:04:12.982015 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.981999    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134507c4-f77f-45ba-abd6-efc840d68fc0-kube-api-access-nmqh2" (OuterVolumeSpecName: "kube-api-access-nmqh2") pod "134507c4-f77f-45ba-abd6-efc840d68fc0" (UID: "134507c4-f77f-45ba-abd6-efc840d68fc0"). InnerVolumeSpecName "kube-api-access-nmqh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:04:12.984820 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.984799    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77cfe3f-3f07-4074-b11a-45e7bc993ba7-util" (OuterVolumeSpecName: "util") pod "a77cfe3f-3f07-4074-b11a-45e7bc993ba7" (UID: "a77cfe3f-3f07-4074-b11a-45e7bc993ba7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:04:12.985943 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:12.985918    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/134507c4-f77f-45ba-abd6-efc840d68fc0-util" (OuterVolumeSpecName: "util") pod "134507c4-f77f-45ba-abd6-efc840d68fc0" (UID: "134507c4-f77f-45ba-abd6-efc840d68fc0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:04:13.079893 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:13.079866    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a77cfe3f-3f07-4074-b11a-45e7bc993ba7-util\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:04:13.079893 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:13.079888    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmqh2\" (UniqueName: \"kubernetes.io/projected/134507c4-f77f-45ba-abd6-efc840d68fc0-kube-api-access-nmqh2\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:04:13.079893 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:13.079898    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j2ds\" (UniqueName: \"kubernetes.io/projected/a77cfe3f-3f07-4074-b11a-45e7bc993ba7-kube-api-access-6j2ds\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:04:13.080083 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:13.079907    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/134507c4-f77f-45ba-abd6-efc840d68fc0-bundle\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:04:13.080083 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:13.079915    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/134507c4-f77f-45ba-abd6-efc840d68fc0-util\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:04:13.080083 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:13.079925    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a77cfe3f-3f07-4074-b11a-45e7bc993ba7-bundle\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:04:13.706868 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:13.706838    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp"
Sep 15 21:04:13.707041 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:13.706863    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zs8sp" event={"ID":"a77cfe3f-3f07-4074-b11a-45e7bc993ba7","Type":"ContainerDied","Data":"9d86880221323f1ed6330c535eed4e8910b4132be41334e9f5f1ad63b56be4ab"}
Sep 15 21:04:13.707041 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:13.706951    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d86880221323f1ed6330c535eed4e8910b4132be41334e9f5f1ad63b56be4ab"
Sep 15 21:04:13.708724 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:13.708705    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv"
Sep 15 21:04:13.708724 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:13.708715    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5j4wv" event={"ID":"134507c4-f77f-45ba-abd6-efc840d68fc0","Type":"ContainerDied","Data":"eeac81339b8c263e1f0cfb919414e4111703666d5f0ce4d97404cfe68036d75d"}
Sep 15 21:04:13.708889 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:13.708753    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eeac81339b8c263e1f0cfb919414e4111703666d5f0ce4d97404cfe68036d75d"
Sep 15 21:04:13.710818 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:13.710804    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr"
Sep 15 21:04:13.710893 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:13.710826    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wznqr" event={"ID":"a0461793-7189-47d1-85a6-5e768b2e91e9","Type":"ContainerDied","Data":"f2a4ae4a01b26e27da9d5b8244cc6cd5bc8d08d7eadebae78df42f25efbaf2c9"}
Sep 15 21:04:13.710893 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:13.710867    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2a4ae4a01b26e27da9d5b8244cc6cd5bc8d08d7eadebae78df42f25efbaf2c9"
Sep 15 21:04:13.713133 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:13.713112    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49" event={"ID":"e2b3a59d-d514-48dd-9dfc-724a95a0cb0e","Type":"ContainerDied","Data":"cfa2c29df546cf2c067e6f6709d0592ff43bbd8d961a53fe897fe970160a929c"}
Sep 15 21:04:13.713210 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:13.713150    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfa2c29df546cf2c067e6f6709d0592ff43bbd8d961a53fe897fe970160a929c"
Sep 15 21:04:13.713210 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:13.713160    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzc49"
Sep 15 21:04:32.443450 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443413    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-tblrx"]
Sep 15 21:04:32.443843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443704    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0461793-7189-47d1-85a6-5e768b2e91e9" containerName="extract"
Sep 15 21:04:32.443843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443715    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0461793-7189-47d1-85a6-5e768b2e91e9" containerName="extract"
Sep 15 21:04:32.443843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443723    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a77cfe3f-3f07-4074-b11a-45e7bc993ba7" containerName="extract"
Sep 15 21:04:32.443843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443729    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a77cfe3f-3f07-4074-b11a-45e7bc993ba7" containerName="extract"
Sep 15 21:04:32.443843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443736    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2b3a59d-d514-48dd-9dfc-724a95a0cb0e" containerName="util"
Sep 15 21:04:32.443843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443741    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2b3a59d-d514-48dd-9dfc-724a95a0cb0e" containerName="util"
Sep 15 21:04:32.443843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443749    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2b3a59d-d514-48dd-9dfc-724a95a0cb0e" containerName="extract"
Sep 15 21:04:32.443843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443754    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2b3a59d-d514-48dd-9dfc-724a95a0cb0e" containerName="extract"
Sep 15 21:04:32.443843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443761    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0461793-7189-47d1-85a6-5e768b2e91e9" containerName="pull"
Sep 15 21:04:32.443843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443765    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0461793-7189-47d1-85a6-5e768b2e91e9" containerName="pull"
Sep 15 21:04:32.443843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443771    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="134507c4-f77f-45ba-abd6-efc840d68fc0" containerName="extract"
Sep 15 21:04:32.443843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443775    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="134507c4-f77f-45ba-abd6-efc840d68fc0" containerName="extract"
Sep 15 21:04:32.443843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443784    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="134507c4-f77f-45ba-abd6-efc840d68fc0" containerName="util"
Sep 15 21:04:32.443843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443788    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="134507c4-f77f-45ba-abd6-efc840d68fc0" containerName="util"
Sep 15 21:04:32.443843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443794    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2b3a59d-d514-48dd-9dfc-724a95a0cb0e" containerName="pull"
Sep 15 21:04:32.443843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443799    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2b3a59d-d514-48dd-9dfc-724a95a0cb0e" containerName="pull"
Sep 15 21:04:32.443843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443808    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="134507c4-f77f-45ba-abd6-efc840d68fc0" containerName="pull"
Sep 15 21:04:32.443843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443813    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="134507c4-f77f-45ba-abd6-efc840d68fc0" containerName="pull"
Sep 15 21:04:32.443843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443822    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a77cfe3f-3f07-4074-b11a-45e7bc993ba7" containerName="util"
Sep 15 21:04:32.443843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443827    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a77cfe3f-3f07-4074-b11a-45e7bc993ba7" containerName="util"
Sep 15 21:04:32.443843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443833    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0461793-7189-47d1-85a6-5e768b2e91e9" containerName="util"
Sep 15 21:04:32.443843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443838    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0461793-7189-47d1-85a6-5e768b2e91e9" containerName="util"
Sep 15 21:04:32.443843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443843    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a77cfe3f-3f07-4074-b11a-45e7bc993ba7" containerName="pull"
Sep 15 21:04:32.443843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443848    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a77cfe3f-3f07-4074-b11a-45e7bc993ba7" containerName="pull"
Sep 15 21:04:32.444600 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443890    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a77cfe3f-3f07-4074-b11a-45e7bc993ba7" containerName="extract"
Sep 15 21:04:32.444600 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443896    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2b3a59d-d514-48dd-9dfc-724a95a0cb0e" containerName="extract"
Sep 15 21:04:32.444600 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443902    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="134507c4-f77f-45ba-abd6-efc840d68fc0" containerName="extract"
Sep 15 21:04:32.444600 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.443910    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0461793-7189-47d1-85a6-5e768b2e91e9" containerName="extract"
Sep 15 21:04:32.446558 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.446543    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tblrx"
Sep 15 21:04:32.449658 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.449629    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 15 21:04:32.449658 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.449652    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 21:04:32.449814 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.449652    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 21:04:32.450742 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.450726    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 15 21:04:32.450824 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.450732    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-85rzs\""
Sep 15 21:04:32.456739 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.456717    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-tblrx"]
Sep 15 21:04:32.535067 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.530915    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ade0084-60cd-454e-8ab8-16aa903c2bda-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-tblrx\" (UID: \"8ade0084-60cd-454e-8ab8-16aa903c2bda\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tblrx"
Sep 15 21:04:32.535067 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.530962    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bk6l\" (UniqueName: \"kubernetes.io/projected/8ade0084-60cd-454e-8ab8-16aa903c2bda-kube-api-access-8bk6l\") pod \"kuadrant-console-plugin-fbdd4f445-tblrx\" (UID: \"8ade0084-60cd-454e-8ab8-16aa903c2bda\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tblrx"
Sep 15 21:04:32.535067 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.531076    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8ade0084-60cd-454e-8ab8-16aa903c2bda-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-tblrx\" (UID: \"8ade0084-60cd-454e-8ab8-16aa903c2bda\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tblrx"
Sep 15 21:04:32.632156 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.632126    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ade0084-60cd-454e-8ab8-16aa903c2bda-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-tblrx\" (UID: \"8ade0084-60cd-454e-8ab8-16aa903c2bda\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tblrx"
Sep 15 21:04:32.632156 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.632160    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bk6l\" (UniqueName: \"kubernetes.io/projected/8ade0084-60cd-454e-8ab8-16aa903c2bda-kube-api-access-8bk6l\") pod \"kuadrant-console-plugin-fbdd4f445-tblrx\" (UID: \"8ade0084-60cd-454e-8ab8-16aa903c2bda\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tblrx"
Sep 15 21:04:32.632369 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.632353    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8ade0084-60cd-454e-8ab8-16aa903c2bda-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-tblrx\" (UID: \"8ade0084-60cd-454e-8ab8-16aa903c2bda\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tblrx"
Sep 15 21:04:32.632920 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.632898    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8ade0084-60cd-454e-8ab8-16aa903c2bda-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-tblrx\" (UID: \"8ade0084-60cd-454e-8ab8-16aa903c2bda\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tblrx"
Sep 15 21:04:32.634432 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.634406    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ade0084-60cd-454e-8ab8-16aa903c2bda-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-tblrx\" (UID: \"8ade0084-60cd-454e-8ab8-16aa903c2bda\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tblrx"
Sep 15 21:04:32.640819 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.640801    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bk6l\" (UniqueName: \"kubernetes.io/projected/8ade0084-60cd-454e-8ab8-16aa903c2bda-kube-api-access-8bk6l\") pod \"kuadrant-console-plugin-fbdd4f445-tblrx\" (UID: \"8ade0084-60cd-454e-8ab8-16aa903c2bda\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tblrx"
Sep 15 21:04:32.755039 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:32.754981    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tblrx"
Sep 15 21:04:33.030678 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:33.030636    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-tblrx"]
Sep 15 21:04:33.031934 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:04:33.031907    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ade0084_60cd_454e_8ab8_16aa903c2bda.slice/crio-b0baeb8c99ed17dbb365cf4048157cb05d734287b37670af29a9bac78149e46e WatchSource:0}: Error finding container b0baeb8c99ed17dbb365cf4048157cb05d734287b37670af29a9bac78149e46e: Status 404 returned error can't find the container with id b0baeb8c99ed17dbb365cf4048157cb05d734287b37670af29a9bac78149e46e
Sep 15 21:04:33.810104 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:33.810072    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tblrx" event={"ID":"8ade0084-60cd-454e-8ab8-16aa903c2bda","Type":"ContainerStarted","Data":"b0baeb8c99ed17dbb365cf4048157cb05d734287b37670af29a9bac78149e46e"}
Sep 15 21:04:38.841476 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:38.841434    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tblrx" event={"ID":"8ade0084-60cd-454e-8ab8-16aa903c2bda","Type":"ContainerStarted","Data":"633c8d85ff657caae860023ab285bcedfc13de8ffec33fc8455f859054e07e94"}
Sep 15 21:04:38.860459 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:04:38.860415    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tblrx" podStartSLOduration=1.6444426779999999 podStartE2EDuration="6.860399799s" totalImagesPullingTime="5.215957121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:04:38.858491788 +0000 UTC m=+397.349137520" watchObservedRunningTime="2026-09-15 21:04:38.860399799 +0000 UTC m=+397.351045530"
Sep 15 21:05:31.982827 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:31.982792    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:05:31.986081 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:31.986045    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-stpzk"
Sep 15 21:05:31.989064 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:31.989033    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 15 21:05:31.995811 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:31.995787    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:05:32.020373 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:32.020344    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5m5x\" (UniqueName: \"kubernetes.io/projected/e9aa7166-dec9-4129-bc7d-76e4b0ce15c5-kube-api-access-d5m5x\") pod \"limitador-limitador-84994f7ffc-stpzk\" (UID: \"e9aa7166-dec9-4129-bc7d-76e4b0ce15c5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-stpzk"
Sep 15 21:05:32.020549 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:32.020400    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e9aa7166-dec9-4129-bc7d-76e4b0ce15c5-config-file\") pod \"limitador-limitador-84994f7ffc-stpzk\" (UID: \"e9aa7166-dec9-4129-bc7d-76e4b0ce15c5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-stpzk"
Sep 15 21:05:32.035462 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:32.035432    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:05:32.120868 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:32.120827    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5m5x\" (UniqueName: \"kubernetes.io/projected/e9aa7166-dec9-4129-bc7d-76e4b0ce15c5-kube-api-access-d5m5x\") pod \"limitador-limitador-84994f7ffc-stpzk\" (UID: \"e9aa7166-dec9-4129-bc7d-76e4b0ce15c5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-stpzk"
Sep 15 21:05:32.121082 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:32.120890    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e9aa7166-dec9-4129-bc7d-76e4b0ce15c5-config-file\") pod \"limitador-limitador-84994f7ffc-stpzk\" (UID: \"e9aa7166-dec9-4129-bc7d-76e4b0ce15c5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-stpzk"
Sep 15 21:05:32.121531 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:32.121508    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e9aa7166-dec9-4129-bc7d-76e4b0ce15c5-config-file\") pod \"limitador-limitador-84994f7ffc-stpzk\" (UID: \"e9aa7166-dec9-4129-bc7d-76e4b0ce15c5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-stpzk"
Sep 15 21:05:32.130923 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:32.130896    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5m5x\" (UniqueName: \"kubernetes.io/projected/e9aa7166-dec9-4129-bc7d-76e4b0ce15c5-kube-api-access-d5m5x\") pod \"limitador-limitador-84994f7ffc-stpzk\" (UID: \"e9aa7166-dec9-4129-bc7d-76e4b0ce15c5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-stpzk"
Sep 15 21:05:32.297308 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:32.297227    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-stpzk"
Sep 15 21:05:32.571250 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:32.571174    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gfv5f"]
Sep 15 21:05:32.574705 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:32.574670    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gfv5f"
Sep 15 21:05:32.579083 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:32.579064    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2z26s\""
Sep 15 21:05:32.587821 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:32.587796    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:05:32.590018 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:05:32.589995    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9aa7166_dec9_4129_bc7d_76e4b0ce15c5.slice/crio-e3d76e3e433e4c29ce23de08c5fdc31d933c754ea77ae6cef94b94336829cef0 WatchSource:0}: Error finding container e3d76e3e433e4c29ce23de08c5fdc31d933c754ea77ae6cef94b94336829cef0: Status 404 returned error can't find the container with id e3d76e3e433e4c29ce23de08c5fdc31d933c754ea77ae6cef94b94336829cef0
Sep 15 21:05:32.590399 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:32.590376    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gfv5f"]
Sep 15 21:05:32.626172 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:32.626145    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6hf7\" (UniqueName: \"kubernetes.io/projected/f02d976f-a539-4af5-bfca-40e0de195790-kube-api-access-x6hf7\") pod \"authorino-7bcfd8f9cd-gfv5f\" (UID: \"f02d976f-a539-4af5-bfca-40e0de195790\") " pod="kuadrant-system/authorino-7bcfd8f9cd-gfv5f"
Sep 15 21:05:32.727493 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:32.727462    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6hf7\" (UniqueName: \"kubernetes.io/projected/f02d976f-a539-4af5-bfca-40e0de195790-kube-api-access-x6hf7\") pod \"authorino-7bcfd8f9cd-gfv5f\" (UID: \"f02d976f-a539-4af5-bfca-40e0de195790\") " pod="kuadrant-system/authorino-7bcfd8f9cd-gfv5f"
Sep 15 21:05:32.736513 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:32.736487    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6hf7\" (UniqueName: \"kubernetes.io/projected/f02d976f-a539-4af5-bfca-40e0de195790-kube-api-access-x6hf7\") pod \"authorino-7bcfd8f9cd-gfv5f\" (UID: \"f02d976f-a539-4af5-bfca-40e0de195790\") " pod="kuadrant-system/authorino-7bcfd8f9cd-gfv5f"
Sep 15 21:05:32.753997 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:32.753971    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-jh2gd"]
Sep 15 21:05:32.757228 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:32.757214    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-jh2gd"
Sep 15 21:05:32.769582 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:32.769556    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jh2gd"]
Sep 15 21:05:32.828184 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:32.828117    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z876t\" (UniqueName: \"kubernetes.io/projected/3913a398-322b-4f76-acf5-3292bad44ed9-kube-api-access-z876t\") pod \"authorino-59c974bd48-jh2gd\" (UID: \"3913a398-322b-4f76-acf5-3292bad44ed9\") " pod="kuadrant-system/authorino-59c974bd48-jh2gd"
Sep 15 21:05:32.883651 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:32.883626    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gfv5f"
Sep 15 21:05:32.928762 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:32.928676    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z876t\" (UniqueName: \"kubernetes.io/projected/3913a398-322b-4f76-acf5-3292bad44ed9-kube-api-access-z876t\") pod \"authorino-59c974bd48-jh2gd\" (UID: \"3913a398-322b-4f76-acf5-3292bad44ed9\") " pod="kuadrant-system/authorino-59c974bd48-jh2gd"
Sep 15 21:05:32.939301 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:32.939271    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z876t\" (UniqueName: \"kubernetes.io/projected/3913a398-322b-4f76-acf5-3292bad44ed9-kube-api-access-z876t\") pod \"authorino-59c974bd48-jh2gd\" (UID: \"3913a398-322b-4f76-acf5-3292bad44ed9\") " pod="kuadrant-system/authorino-59c974bd48-jh2gd"
Sep 15 21:05:33.066263 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:33.066230    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-jh2gd"
Sep 15 21:05:33.099161 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:33.099080    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-stpzk" event={"ID":"e9aa7166-dec9-4129-bc7d-76e4b0ce15c5","Type":"ContainerStarted","Data":"e3d76e3e433e4c29ce23de08c5fdc31d933c754ea77ae6cef94b94336829cef0"}
Sep 15 21:05:33.153559 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:33.153536    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gfv5f"]
Sep 15 21:05:33.155282 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:05:33.155259    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf02d976f_a539_4af5_bfca_40e0de195790.slice/crio-1b9b580c3e04719e5b7e35dd02ca0899d4eae7ffef1dbedc644b4204c214e436 WatchSource:0}: Error finding container 1b9b580c3e04719e5b7e35dd02ca0899d4eae7ffef1dbedc644b4204c214e436: Status 404 returned error can't find the container with id 1b9b580c3e04719e5b7e35dd02ca0899d4eae7ffef1dbedc644b4204c214e436
Sep 15 21:05:33.339702 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:33.339678    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jh2gd"]
Sep 15 21:05:33.341317 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:05:33.341291    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3913a398_322b_4f76_acf5_3292bad44ed9.slice/crio-38e78abc056d8b3583d7d9d250823ce6737e3489abd240e48978d1ec384ac3ee WatchSource:0}: Error finding container 38e78abc056d8b3583d7d9d250823ce6737e3489abd240e48978d1ec384ac3ee: Status 404 returned error can't find the container with id 38e78abc056d8b3583d7d9d250823ce6737e3489abd240e48978d1ec384ac3ee
Sep 15 21:05:34.106117 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:34.106085    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-jh2gd" event={"ID":"3913a398-322b-4f76-acf5-3292bad44ed9","Type":"ContainerStarted","Data":"38e78abc056d8b3583d7d9d250823ce6737e3489abd240e48978d1ec384ac3ee"}
Sep 15 21:05:34.107247 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:34.107224    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gfv5f" event={"ID":"f02d976f-a539-4af5-bfca-40e0de195790","Type":"ContainerStarted","Data":"1b9b580c3e04719e5b7e35dd02ca0899d4eae7ffef1dbedc644b4204c214e436"}
Sep 15 21:05:36.121816 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:36.121783    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-stpzk" event={"ID":"e9aa7166-dec9-4129-bc7d-76e4b0ce15c5","Type":"ContainerStarted","Data":"63d6db334c8f97600e2bde8a252aad09ef1de509479fc2ab157695894efef233"}
Sep 15 21:05:36.122310 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:36.121921    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-stpzk"
Sep 15 21:05:36.124065 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:36.124014    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-jh2gd" event={"ID":"3913a398-322b-4f76-acf5-3292bad44ed9","Type":"ContainerStarted","Data":"5d1579eb9729cd8a66d26f4cdc305d78de689b8e53e338be8208b3f4b0e00e43"}
Sep 15 21:05:36.126070 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:36.126024    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gfv5f" event={"ID":"f02d976f-a539-4af5-bfca-40e0de195790","Type":"ContainerStarted","Data":"ae11487a0dd928c38368e3866f8c1d382e00724d0f3fd68020449a646ba9dc40"}
Sep 15 21:05:36.142067 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:36.141997    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-stpzk" podStartSLOduration=2.57895108 podStartE2EDuration="5.141979615s" totalImagesPullingTime="2.563028535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:05:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:05:36.140698448 +0000 UTC m=+454.631344182" watchObservedRunningTime="2026-09-15 21:05:36.141979615 +0000 UTC m=+454.632625346"
Sep 15 21:05:36.157672 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:36.157613    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-jh2gd" podStartSLOduration=1.5131240240000001 podStartE2EDuration="4.157597619s" totalImagesPullingTime="2.644473595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:05:36.156438521 +0000 UTC m=+454.647084254" watchObservedRunningTime="2026-09-15 21:05:36.157597619 +0000 UTC m=+454.648243351"
Sep 15 21:05:36.177298 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:36.177231    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-gfv5f" podStartSLOduration=1.342937344 podStartE2EDuration="4.177210674s" totalImagesPullingTime="2.83427333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:05:36.175594362 +0000 UTC m=+454.666240093" watchObservedRunningTime="2026-09-15 21:05:36.177210674 +0000 UTC m=+454.667856426"
Sep 15 21:05:36.183685 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:36.183653    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gfv5f"]
Sep 15 21:05:38.136336 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:38.136299    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-gfv5f" podUID="f02d976f-a539-4af5-bfca-40e0de195790" containerName="authorino" containerID="cri-o://ae11487a0dd928c38368e3866f8c1d382e00724d0f3fd68020449a646ba9dc40" gracePeriod=30
Sep 15 21:05:38.379637 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:38.379607    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gfv5f"
Sep 15 21:05:38.483635 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:38.483608    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6hf7\" (UniqueName: \"kubernetes.io/projected/f02d976f-a539-4af5-bfca-40e0de195790-kube-api-access-x6hf7\") pod \"f02d976f-a539-4af5-bfca-40e0de195790\" (UID: \"f02d976f-a539-4af5-bfca-40e0de195790\") "
Sep 15 21:05:38.485667 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:38.485646    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02d976f-a539-4af5-bfca-40e0de195790-kube-api-access-x6hf7" (OuterVolumeSpecName: "kube-api-access-x6hf7") pod "f02d976f-a539-4af5-bfca-40e0de195790" (UID: "f02d976f-a539-4af5-bfca-40e0de195790"). InnerVolumeSpecName "kube-api-access-x6hf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:05:38.584802 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:38.584762    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6hf7\" (UniqueName: \"kubernetes.io/projected/f02d976f-a539-4af5-bfca-40e0de195790-kube-api-access-x6hf7\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:05:39.142451 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:39.142416    2558 generic.go:356] "Generic (PLEG): container finished" podID="f02d976f-a539-4af5-bfca-40e0de195790" containerID="ae11487a0dd928c38368e3866f8c1d382e00724d0f3fd68020449a646ba9dc40" exitCode=0
Sep 15 21:05:39.142878 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:39.142465    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gfv5f"
Sep 15 21:05:39.142878 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:39.142506    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gfv5f" event={"ID":"f02d976f-a539-4af5-bfca-40e0de195790","Type":"ContainerDied","Data":"ae11487a0dd928c38368e3866f8c1d382e00724d0f3fd68020449a646ba9dc40"}
Sep 15 21:05:39.142878 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:39.142552    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gfv5f" event={"ID":"f02d976f-a539-4af5-bfca-40e0de195790","Type":"ContainerDied","Data":"1b9b580c3e04719e5b7e35dd02ca0899d4eae7ffef1dbedc644b4204c214e436"}
Sep 15 21:05:39.142878 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:39.142574    2558 scope.go:117] "RemoveContainer" containerID="ae11487a0dd928c38368e3866f8c1d382e00724d0f3fd68020449a646ba9dc40"
Sep 15 21:05:39.151280 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:39.151261    2558 scope.go:117] "RemoveContainer" containerID="ae11487a0dd928c38368e3866f8c1d382e00724d0f3fd68020449a646ba9dc40"
Sep 15 21:05:39.151559 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:05:39.151540    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae11487a0dd928c38368e3866f8c1d382e00724d0f3fd68020449a646ba9dc40\": container with ID starting with ae11487a0dd928c38368e3866f8c1d382e00724d0f3fd68020449a646ba9dc40 not found: ID does not exist" containerID="ae11487a0dd928c38368e3866f8c1d382e00724d0f3fd68020449a646ba9dc40"
Sep 15 21:05:39.151608 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:39.151567    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae11487a0dd928c38368e3866f8c1d382e00724d0f3fd68020449a646ba9dc40"} err="failed to get container status \"ae11487a0dd928c38368e3866f8c1d382e00724d0f3fd68020449a646ba9dc40\": rpc error: code = NotFound desc = could not find container \"ae11487a0dd928c38368e3866f8c1d382e00724d0f3fd68020449a646ba9dc40\": container with ID starting with ae11487a0dd928c38368e3866f8c1d382e00724d0f3fd68020449a646ba9dc40 not found: ID does not exist"
Sep 15 21:05:39.168034 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:39.168010    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gfv5f"]
Sep 15 21:05:39.173738 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:39.173714    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gfv5f"]
Sep 15 21:05:39.941797 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:39.941767    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f02d976f-a539-4af5-bfca-40e0de195790" path="/var/lib/kubelet/pods/f02d976f-a539-4af5-bfca-40e0de195790/volumes"
Sep 15 21:05:47.132822 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:05:47.132790    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-stpzk"
Sep 15 21:06:04.231204 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:04.231169    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cd5b77dcb-dqd62"]
Sep 15 21:06:04.231743 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:04.231468    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f02d976f-a539-4af5-bfca-40e0de195790" containerName="authorino"
Sep 15 21:06:04.231743 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:04.231479    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f02d976f-a539-4af5-bfca-40e0de195790" containerName="authorino"
Sep 15 21:06:04.231743 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:04.231548    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f02d976f-a539-4af5-bfca-40e0de195790" containerName="authorino"
Sep 15 21:06:04.235724 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:04.235707    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cd5b77dcb-dqd62"
Sep 15 21:06:04.239138 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:04.239111    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 15 21:06:04.241219 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:04.241196    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cd5b77dcb-dqd62"]
Sep 15 21:06:04.303353 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:04.303307    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f-tls-cert\") pod \"authorino-7cd5b77dcb-dqd62\" (UID: \"c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f\") " pod="kuadrant-system/authorino-7cd5b77dcb-dqd62"
Sep 15 21:06:04.303353 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:04.303351    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrt6s\" (UniqueName: \"kubernetes.io/projected/c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f-kube-api-access-wrt6s\") pod \"authorino-7cd5b77dcb-dqd62\" (UID: \"c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f\") " pod="kuadrant-system/authorino-7cd5b77dcb-dqd62"
Sep 15 21:06:04.404099 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:04.404060    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f-tls-cert\") pod \"authorino-7cd5b77dcb-dqd62\" (UID: \"c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f\") " pod="kuadrant-system/authorino-7cd5b77dcb-dqd62"
Sep 15 21:06:04.404251 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:04.404107    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrt6s\" (UniqueName: \"kubernetes.io/projected/c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f-kube-api-access-wrt6s\") pod \"authorino-7cd5b77dcb-dqd62\" (UID: \"c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f\") " pod="kuadrant-system/authorino-7cd5b77dcb-dqd62"
Sep 15 21:06:04.406499 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:04.406465    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f-tls-cert\") pod \"authorino-7cd5b77dcb-dqd62\" (UID: \"c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f\") " pod="kuadrant-system/authorino-7cd5b77dcb-dqd62"
Sep 15 21:06:04.411347 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:04.411323    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrt6s\" (UniqueName: \"kubernetes.io/projected/c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f-kube-api-access-wrt6s\") pod \"authorino-7cd5b77dcb-dqd62\" (UID: \"c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f\") " pod="kuadrant-system/authorino-7cd5b77dcb-dqd62"
Sep 15 21:06:04.546509 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:04.546431    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cd5b77dcb-dqd62"
Sep 15 21:06:04.815182 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:04.815153    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cd5b77dcb-dqd62"]
Sep 15 21:06:04.816797 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:06:04.816771    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4d5bd3c_9f79_418a_8c8c_6e7b2ae4220f.slice/crio-ca6c67802f470b1080232333fc1348619274a9ef898251b10223762e31e944a9 WatchSource:0}: Error finding container ca6c67802f470b1080232333fc1348619274a9ef898251b10223762e31e944a9: Status 404 returned error can't find the container with id ca6c67802f470b1080232333fc1348619274a9ef898251b10223762e31e944a9
Sep 15 21:06:05.274658 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:05.274617    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cd5b77dcb-dqd62" event={"ID":"c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f","Type":"ContainerStarted","Data":"ca6c67802f470b1080232333fc1348619274a9ef898251b10223762e31e944a9"}
Sep 15 21:06:06.281438 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:06.281397    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cd5b77dcb-dqd62" event={"ID":"c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f","Type":"ContainerStarted","Data":"e4403a6047f711ec45dae34fda6b1ab1fdef394b483b251ea38e36c7e5308804"}
Sep 15 21:06:06.297839 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:06.297788    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cd5b77dcb-dqd62" podStartSLOduration=1.597301284 podStartE2EDuration="2.297772707s" totalImagesPullingTime="700.471423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:06:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:06:06.296850613 +0000 UTC m=+484.787496346" watchObservedRunningTime="2026-09-15 21:06:06.297772707 +0000 UTC m=+484.788418457"
Sep 15 21:06:06.324704 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:06.324670    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jh2gd"]
Sep 15 21:06:06.324932 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:06.324904    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-jh2gd" podUID="3913a398-322b-4f76-acf5-3292bad44ed9" containerName="authorino" containerID="cri-o://5d1579eb9729cd8a66d26f4cdc305d78de689b8e53e338be8208b3f4b0e00e43" gracePeriod=30
Sep 15 21:06:06.563534 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:06.563512    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-jh2gd"
Sep 15 21:06:06.621265 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:06.621235    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z876t\" (UniqueName: \"kubernetes.io/projected/3913a398-322b-4f76-acf5-3292bad44ed9-kube-api-access-z876t\") pod \"3913a398-322b-4f76-acf5-3292bad44ed9\" (UID: \"3913a398-322b-4f76-acf5-3292bad44ed9\") "
Sep 15 21:06:06.623265 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:06.623233    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3913a398-322b-4f76-acf5-3292bad44ed9-kube-api-access-z876t" (OuterVolumeSpecName: "kube-api-access-z876t") pod "3913a398-322b-4f76-acf5-3292bad44ed9" (UID: "3913a398-322b-4f76-acf5-3292bad44ed9"). InnerVolumeSpecName "kube-api-access-z876t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:06:06.721970 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:06.721916    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z876t\" (UniqueName: \"kubernetes.io/projected/3913a398-322b-4f76-acf5-3292bad44ed9-kube-api-access-z876t\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:06:07.288148 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:07.288113    2558 generic.go:356] "Generic (PLEG): container finished" podID="3913a398-322b-4f76-acf5-3292bad44ed9" containerID="5d1579eb9729cd8a66d26f4cdc305d78de689b8e53e338be8208b3f4b0e00e43" exitCode=0
Sep 15 21:06:07.288541 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:07.288160    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-jh2gd"
Sep 15 21:06:07.288541 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:07.288205    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-jh2gd" event={"ID":"3913a398-322b-4f76-acf5-3292bad44ed9","Type":"ContainerDied","Data":"5d1579eb9729cd8a66d26f4cdc305d78de689b8e53e338be8208b3f4b0e00e43"}
Sep 15 21:06:07.288541 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:07.288245    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-jh2gd" event={"ID":"3913a398-322b-4f76-acf5-3292bad44ed9","Type":"ContainerDied","Data":"38e78abc056d8b3583d7d9d250823ce6737e3489abd240e48978d1ec384ac3ee"}
Sep 15 21:06:07.288541 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:07.288261    2558 scope.go:117] "RemoveContainer" containerID="5d1579eb9729cd8a66d26f4cdc305d78de689b8e53e338be8208b3f4b0e00e43"
Sep 15 21:06:07.296694 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:07.296674    2558 scope.go:117] "RemoveContainer" containerID="5d1579eb9729cd8a66d26f4cdc305d78de689b8e53e338be8208b3f4b0e00e43"
Sep 15 21:06:07.296948 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:06:07.296928    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d1579eb9729cd8a66d26f4cdc305d78de689b8e53e338be8208b3f4b0e00e43\": container with ID starting with 5d1579eb9729cd8a66d26f4cdc305d78de689b8e53e338be8208b3f4b0e00e43 not found: ID does not exist" containerID="5d1579eb9729cd8a66d26f4cdc305d78de689b8e53e338be8208b3f4b0e00e43"
Sep 15 21:06:07.297008 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:07.296960    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1579eb9729cd8a66d26f4cdc305d78de689b8e53e338be8208b3f4b0e00e43"} err="failed to get container status \"5d1579eb9729cd8a66d26f4cdc305d78de689b8e53e338be8208b3f4b0e00e43\": rpc error: code = NotFound desc = could not find container \"5d1579eb9729cd8a66d26f4cdc305d78de689b8e53e338be8208b3f4b0e00e43\": container with ID starting with 5d1579eb9729cd8a66d26f4cdc305d78de689b8e53e338be8208b3f4b0e00e43 not found: ID does not exist"
Sep 15 21:06:07.309204 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:07.309184    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jh2gd"]
Sep 15 21:06:07.311006 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:07.310986    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jh2gd"]
Sep 15 21:06:07.942209 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:07.942179    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3913a398-322b-4f76-acf5-3292bad44ed9" path="/var/lib/kubelet/pods/3913a398-322b-4f76-acf5-3292bad44ed9/volumes"
Sep 15 21:06:48.134580 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:06:48.134541    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:07:07.985671 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:07:07.985634    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:07:31.384769 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:07:31.384729    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:07:48.986113 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:07:48.986072    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:07:59.793335 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:07:59.793302    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:08:22.101773 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:22.101734    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:08:27.900678 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:27.900644    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:08:35.087027 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.086986    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"]
Sep 15 21:08:35.087400 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.087317    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3913a398-322b-4f76-acf5-3292bad44ed9" containerName="authorino"
Sep 15 21:08:35.087400 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.087329    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3913a398-322b-4f76-acf5-3292bad44ed9" containerName="authorino"
Sep 15 21:08:35.087400 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.087382    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3913a398-322b-4f76-acf5-3292bad44ed9" containerName="authorino"
Sep 15 21:08:35.090321 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.090304    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"
Sep 15 21:08:35.094786 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.094760    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 21:08:35.094905 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.094808    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-b2fk7\""
Sep 15 21:08:35.094905 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.094813    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 15 21:08:35.094905 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.094812    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 21:08:35.102425 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.102405    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"]
Sep 15 21:08:35.216914 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.216879    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/934756fd-2f06-46a2-95f7-1ac46b50d4a2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz\" (UID: \"934756fd-2f06-46a2-95f7-1ac46b50d4a2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"
Sep 15 21:08:35.216914 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.216914    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/934756fd-2f06-46a2-95f7-1ac46b50d4a2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz\" (UID: \"934756fd-2f06-46a2-95f7-1ac46b50d4a2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"
Sep 15 21:08:35.217181 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.216943    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/934756fd-2f06-46a2-95f7-1ac46b50d4a2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz\" (UID: \"934756fd-2f06-46a2-95f7-1ac46b50d4a2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"
Sep 15 21:08:35.217181 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.217017    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/934756fd-2f06-46a2-95f7-1ac46b50d4a2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz\" (UID: \"934756fd-2f06-46a2-95f7-1ac46b50d4a2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"
Sep 15 21:08:35.217181 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.217120    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/934756fd-2f06-46a2-95f7-1ac46b50d4a2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz\" (UID: \"934756fd-2f06-46a2-95f7-1ac46b50d4a2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"
Sep 15 21:08:35.217316 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.217179    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f5xg\" (UniqueName: \"kubernetes.io/projected/934756fd-2f06-46a2-95f7-1ac46b50d4a2-kube-api-access-9f5xg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz\" (UID: \"934756fd-2f06-46a2-95f7-1ac46b50d4a2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"
Sep 15 21:08:35.318430 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.318393    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/934756fd-2f06-46a2-95f7-1ac46b50d4a2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz\" (UID: \"934756fd-2f06-46a2-95f7-1ac46b50d4a2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"
Sep 15 21:08:35.318430 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.318435    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/934756fd-2f06-46a2-95f7-1ac46b50d4a2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz\" (UID: \"934756fd-2f06-46a2-95f7-1ac46b50d4a2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"
Sep 15 21:08:35.318668 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.318471    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/934756fd-2f06-46a2-95f7-1ac46b50d4a2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz\" (UID: \"934756fd-2f06-46a2-95f7-1ac46b50d4a2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"
Sep 15 21:08:35.318668 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.318505    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/934756fd-2f06-46a2-95f7-1ac46b50d4a2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz\" (UID: \"934756fd-2f06-46a2-95f7-1ac46b50d4a2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"
Sep 15 21:08:35.318668 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.318553    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/934756fd-2f06-46a2-95f7-1ac46b50d4a2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz\" (UID: \"934756fd-2f06-46a2-95f7-1ac46b50d4a2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"
Sep 15 21:08:35.318668 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.318595    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f5xg\" (UniqueName: \"kubernetes.io/projected/934756fd-2f06-46a2-95f7-1ac46b50d4a2-kube-api-access-9f5xg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz\" (UID: \"934756fd-2f06-46a2-95f7-1ac46b50d4a2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"
Sep 15 21:08:35.318861 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.318841    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/934756fd-2f06-46a2-95f7-1ac46b50d4a2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz\" (UID: \"934756fd-2f06-46a2-95f7-1ac46b50d4a2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"
Sep 15 21:08:35.318971 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.318886    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/934756fd-2f06-46a2-95f7-1ac46b50d4a2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz\" (UID: \"934756fd-2f06-46a2-95f7-1ac46b50d4a2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"
Sep 15 21:08:35.319017 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.318973    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/934756fd-2f06-46a2-95f7-1ac46b50d4a2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz\" (UID: \"934756fd-2f06-46a2-95f7-1ac46b50d4a2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"
Sep 15 21:08:35.320840 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.320820    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/934756fd-2f06-46a2-95f7-1ac46b50d4a2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz\" (UID: \"934756fd-2f06-46a2-95f7-1ac46b50d4a2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"
Sep 15 21:08:35.320977 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.320961    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/934756fd-2f06-46a2-95f7-1ac46b50d4a2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz\" (UID: \"934756fd-2f06-46a2-95f7-1ac46b50d4a2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"
Sep 15 21:08:35.329082 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.329038    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f5xg\" (UniqueName: \"kubernetes.io/projected/934756fd-2f06-46a2-95f7-1ac46b50d4a2-kube-api-access-9f5xg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz\" (UID: \"934756fd-2f06-46a2-95f7-1ac46b50d4a2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"
Sep 15 21:08:35.399805 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.399729    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"
Sep 15 21:08:35.672693 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.672668    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"]
Sep 15 21:08:35.673532 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:08:35.673508    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod934756fd_2f06_46a2_95f7_1ac46b50d4a2.slice/crio-a19da79c02c2d6503718b7e3142518333c2341def30ede6927385ed0976a0d41 WatchSource:0}: Error finding container a19da79c02c2d6503718b7e3142518333c2341def30ede6927385ed0976a0d41: Status 404 returned error can't find the container with id a19da79c02c2d6503718b7e3142518333c2341def30ede6927385ed0976a0d41
Sep 15 21:08:35.674941 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:35.674925    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:08:36.024465 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:36.024385    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz" event={"ID":"934756fd-2f06-46a2-95f7-1ac46b50d4a2","Type":"ContainerStarted","Data":"a19da79c02c2d6503718b7e3142518333c2341def30ede6927385ed0976a0d41"}
Sep 15 21:08:37.030852 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:37.030766    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz" event={"ID":"934756fd-2f06-46a2-95f7-1ac46b50d4a2","Type":"ContainerStarted","Data":"9e4c7b00a3f11f46548d9f69dd5621fca5386d1a7dc1204f843e98f079acae52"}
Sep 15 21:08:37.051622 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:37.051573    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz" podStartSLOduration=0.962564888 podStartE2EDuration="2.051559079s" totalImagesPullingTime="1.088994191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:37.049941515 +0000 UTC m=+635.540587263" watchObservedRunningTime="2026-09-15 21:08:37.051559079 +0000 UTC m=+635.542204811"
Sep 15 21:08:37.211039 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:37.210994    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:08:45.400806 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:45.400753    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"
Sep 15 21:08:45.401425 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:45.401065    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"
Sep 15 21:08:45.413521 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:45.413501    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"
Sep 15 21:08:46.086149 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:08:46.086119    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz"
Sep 15 21:09:56.600952 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:56.600876    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-cc985957c-4q9c7"]
Sep 15 21:09:56.604091 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:56.604073    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cc985957c-4q9c7"
Sep 15 21:09:56.617473 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:56.617449    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cc985957c-4q9c7"]
Sep 15 21:09:56.767568 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:56.767538    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/73e75579-00ec-424b-8b07-852bae98085b-tls-cert\") pod \"authorino-cc985957c-4q9c7\" (UID: \"73e75579-00ec-424b-8b07-852bae98085b\") " pod="kuadrant-system/authorino-cc985957c-4q9c7"
Sep 15 21:09:56.767741 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:56.767594    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jprlg\" (UniqueName: \"kubernetes.io/projected/73e75579-00ec-424b-8b07-852bae98085b-kube-api-access-jprlg\") pod \"authorino-cc985957c-4q9c7\" (UID: \"73e75579-00ec-424b-8b07-852bae98085b\") " pod="kuadrant-system/authorino-cc985957c-4q9c7"
Sep 15 21:09:56.868138 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:56.868066    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jprlg\" (UniqueName: \"kubernetes.io/projected/73e75579-00ec-424b-8b07-852bae98085b-kube-api-access-jprlg\") pod \"authorino-cc985957c-4q9c7\" (UID: \"73e75579-00ec-424b-8b07-852bae98085b\") " pod="kuadrant-system/authorino-cc985957c-4q9c7"
Sep 15 21:09:56.868138 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:56.868136    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/73e75579-00ec-424b-8b07-852bae98085b-tls-cert\") pod \"authorino-cc985957c-4q9c7\" (UID: \"73e75579-00ec-424b-8b07-852bae98085b\") " pod="kuadrant-system/authorino-cc985957c-4q9c7"
Sep 15 21:09:56.870502 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:56.870485    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/73e75579-00ec-424b-8b07-852bae98085b-tls-cert\") pod \"authorino-cc985957c-4q9c7\" (UID: \"73e75579-00ec-424b-8b07-852bae98085b\") " pod="kuadrant-system/authorino-cc985957c-4q9c7"
Sep 15 21:09:56.877300 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:56.877277    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jprlg\" (UniqueName: \"kubernetes.io/projected/73e75579-00ec-424b-8b07-852bae98085b-kube-api-access-jprlg\") pod \"authorino-cc985957c-4q9c7\" (UID: \"73e75579-00ec-424b-8b07-852bae98085b\") " pod="kuadrant-system/authorino-cc985957c-4q9c7"
Sep 15 21:09:56.912991 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:56.912970    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cc985957c-4q9c7"
Sep 15 21:09:57.188833 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:57.188799    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cc985957c-4q9c7"]
Sep 15 21:09:57.190025 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:09:57.189999    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73e75579_00ec_424b_8b07_852bae98085b.slice/crio-76f2e684e0c0a7d8b9d64919fded613a03db02e689a0a05bba15c1b74b51bf5b WatchSource:0}: Error finding container 76f2e684e0c0a7d8b9d64919fded613a03db02e689a0a05bba15c1b74b51bf5b: Status 404 returned error can't find the container with id 76f2e684e0c0a7d8b9d64919fded613a03db02e689a0a05bba15c1b74b51bf5b
Sep 15 21:09:57.434335 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:57.434302    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cc985957c-4q9c7" event={"ID":"73e75579-00ec-424b-8b07-852bae98085b","Type":"ContainerStarted","Data":"76f2e684e0c0a7d8b9d64919fded613a03db02e689a0a05bba15c1b74b51bf5b"}
Sep 15 21:09:58.440810 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:58.440780    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cc985957c-4q9c7" event={"ID":"73e75579-00ec-424b-8b07-852bae98085b","Type":"ContainerStarted","Data":"11ef951b6efa40bee41fc1adc78332bb969cc91b5a8c2f9e48bf6046f5e447cc"}
Sep 15 21:09:58.470206 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:58.470138    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-cc985957c-4q9c7" podStartSLOduration=1.311158376 podStartE2EDuration="2.470121786s" totalImagesPullingTime="1.15896341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:09:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:09:58.465108979 +0000 UTC m=+716.955754710" watchObservedRunningTime="2026-09-15 21:09:58.470121786 +0000 UTC m=+716.960767519"
Sep 15 21:09:58.515800 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:58.515767    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7cd5b77dcb-dqd62"]
Sep 15 21:09:58.516184 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:58.515975    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7cd5b77dcb-dqd62" podUID="c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f" containerName="authorino" containerID="cri-o://e4403a6047f711ec45dae34fda6b1ab1fdef394b483b251ea38e36c7e5308804" gracePeriod=30
Sep 15 21:09:58.768762 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:58.768733    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cd5b77dcb-dqd62"
Sep 15 21:09:58.885267 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:58.885185    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrt6s\" (UniqueName: \"kubernetes.io/projected/c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f-kube-api-access-wrt6s\") pod \"c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f\" (UID: \"c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f\") "
Sep 15 21:09:58.885267 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:58.885246    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f-tls-cert\") pod \"c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f\" (UID: \"c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f\") "
Sep 15 21:09:58.887360 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:58.887326    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f-kube-api-access-wrt6s" (OuterVolumeSpecName: "kube-api-access-wrt6s") pod "c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f" (UID: "c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f"). InnerVolumeSpecName "kube-api-access-wrt6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:09:58.895552 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:58.895522    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f" (UID: "c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:09:58.986841 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:58.986800    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrt6s\" (UniqueName: \"kubernetes.io/projected/c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f-kube-api-access-wrt6s\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:09:58.986841 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:58.986843    2558 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f-tls-cert\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:09:59.447748 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:59.447711    2558 generic.go:356] "Generic (PLEG): container finished" podID="c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f" containerID="e4403a6047f711ec45dae34fda6b1ab1fdef394b483b251ea38e36c7e5308804" exitCode=0
Sep 15 21:09:59.447748 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:59.447751    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cd5b77dcb-dqd62"
Sep 15 21:09:59.448277 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:59.447801    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cd5b77dcb-dqd62" event={"ID":"c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f","Type":"ContainerDied","Data":"e4403a6047f711ec45dae34fda6b1ab1fdef394b483b251ea38e36c7e5308804"}
Sep 15 21:09:59.448277 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:59.447844    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cd5b77dcb-dqd62" event={"ID":"c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f","Type":"ContainerDied","Data":"ca6c67802f470b1080232333fc1348619274a9ef898251b10223762e31e944a9"}
Sep 15 21:09:59.448277 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:59.447865    2558 scope.go:117] "RemoveContainer" containerID="e4403a6047f711ec45dae34fda6b1ab1fdef394b483b251ea38e36c7e5308804"
Sep 15 21:09:59.457214 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:59.457192    2558 scope.go:117] "RemoveContainer" containerID="e4403a6047f711ec45dae34fda6b1ab1fdef394b483b251ea38e36c7e5308804"
Sep 15 21:09:59.457496 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:09:59.457477    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4403a6047f711ec45dae34fda6b1ab1fdef394b483b251ea38e36c7e5308804\": container with ID starting with e4403a6047f711ec45dae34fda6b1ab1fdef394b483b251ea38e36c7e5308804 not found: ID does not exist" containerID="e4403a6047f711ec45dae34fda6b1ab1fdef394b483b251ea38e36c7e5308804"
Sep 15 21:09:59.457541 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:59.457508    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4403a6047f711ec45dae34fda6b1ab1fdef394b483b251ea38e36c7e5308804"} err="failed to get container status \"e4403a6047f711ec45dae34fda6b1ab1fdef394b483b251ea38e36c7e5308804\": rpc error: code = NotFound desc = could not find container \"e4403a6047f711ec45dae34fda6b1ab1fdef394b483b251ea38e36c7e5308804\": container with ID starting with e4403a6047f711ec45dae34fda6b1ab1fdef394b483b251ea38e36c7e5308804 not found: ID does not exist"
Sep 15 21:09:59.491294 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:59.491260    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7cd5b77dcb-dqd62"]
Sep 15 21:09:59.500484 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:59.500459    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7cd5b77dcb-dqd62"]
Sep 15 21:09:59.944440 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:09:59.944402    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f" path="/var/lib/kubelet/pods/c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f/volumes"
Sep 15 21:10:17.438393 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.438362    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"]
Sep 15 21:10:17.438981 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.438841    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f" containerName="authorino"
Sep 15 21:10:17.438981 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.438861    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f" containerName="authorino"
Sep 15 21:10:17.438981 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.438965    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4d5bd3c-9f79-418a-8c8c-6e7b2ae4220f" containerName="authorino"
Sep 15 21:10:17.443274 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.443252    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.446410 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.446385    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 21:10:17.446540 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.446420    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-544890-openshift-default-dockercfg-pkfmb\""
Sep 15 21:10:17.455035 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.455014    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"]
Sep 15 21:10:17.535811 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.535779    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istio-token\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.535958 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.535826    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-credential-socket\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.535958 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.535889    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istio-podinfo\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.536079 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.535974    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-workload-certs\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.536079 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.536001    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istio-data\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.536079 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.536018    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istiod-ca-cert\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.536200 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.536082    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dcqk\" (UniqueName: \"kubernetes.io/projected/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-kube-api-access-8dcqk\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.536200 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.536100    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istio-envoy\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.536200 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.536119    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-workload-socket\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.637271 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.637237    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-credential-socket\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.637435 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.637278    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istio-podinfo\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.637435 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.637325    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-workload-certs\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.637435 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.637350    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istio-data\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.637435 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.637374    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istiod-ca-cert\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.637663 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.637519    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dcqk\" (UniqueName: \"kubernetes.io/projected/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-kube-api-access-8dcqk\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.637663 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.637568    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istio-envoy\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.637663 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.637599    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-workload-socket\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.637663 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.637637    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istio-token\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.637868 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.637666    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-credential-socket\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.637868 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.637711    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-workload-certs\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.637868 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.637735    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istio-data\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.638076 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.638035    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-workload-socket\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.638211 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.638141    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istiod-ca-cert\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.639768 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.639746    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istio-envoy\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.639906 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.639891    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istio-podinfo\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.652390 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.652365    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istio-token\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.652697 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.652679    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dcqk\" (UniqueName: \"kubernetes.io/projected/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-kube-api-access-8dcqk\") pod \"e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") " pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:17.754866 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:17.754800    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:18.031954 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:18.031930    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"]
Sep 15 21:10:18.033618 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:10:18.033593    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed5bb3c_b04d_4d34_a04f_f02c037fe755.slice/crio-c8c6dd8b56fbf3bcb0bf1b547f03d131eca5416a423395d8cfc511915703f2c1 WatchSource:0}: Error finding container c8c6dd8b56fbf3bcb0bf1b547f03d131eca5416a423395d8cfc511915703f2c1: Status 404 returned error can't find the container with id c8c6dd8b56fbf3bcb0bf1b547f03d131eca5416a423395d8cfc511915703f2c1
Sep 15 21:10:18.548502 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:18.548468    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw" event={"ID":"2ed5bb3c-b04d-4d34-a04f-f02c037fe755","Type":"ContainerStarted","Data":"c8c6dd8b56fbf3bcb0bf1b547f03d131eca5416a423395d8cfc511915703f2c1"}
Sep 15 21:10:18.929225 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:18.929187    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7937-3c92ca"]
Sep 15 21:10:18.932699 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:18.932673    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7937-3c92ca"
Sep 15 21:10:18.936333 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:18.936308    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-r6zqd\""
Sep 15 21:10:18.937816 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:18.937790    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7937-3c92ca"]
Sep 15 21:10:19.050889 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:19.050849    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcxvl\" (UniqueName: \"kubernetes.io/projected/528b75dd-8015-4e02-810f-5a708f7173d5-kube-api-access-hcxvl\") pod \"test-curl-7937-3c92ca\" (UID: \"528b75dd-8015-4e02-810f-5a708f7173d5\") " pod="opendatahub/test-curl-7937-3c92ca"
Sep 15 21:10:19.151849 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:19.151816    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcxvl\" (UniqueName: \"kubernetes.io/projected/528b75dd-8015-4e02-810f-5a708f7173d5-kube-api-access-hcxvl\") pod \"test-curl-7937-3c92ca\" (UID: \"528b75dd-8015-4e02-810f-5a708f7173d5\") " pod="opendatahub/test-curl-7937-3c92ca"
Sep 15 21:10:19.164718 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:19.164682    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcxvl\" (UniqueName: \"kubernetes.io/projected/528b75dd-8015-4e02-810f-5a708f7173d5-kube-api-access-hcxvl\") pod \"test-curl-7937-3c92ca\" (UID: \"528b75dd-8015-4e02-810f-5a708f7173d5\") " pod="opendatahub/test-curl-7937-3c92ca"
Sep 15 21:10:19.245599 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:19.245527    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7937-3c92ca"
Sep 15 21:10:19.526380 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:19.526352    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7937-3c92ca"]
Sep 15 21:10:19.527260 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:10:19.527220    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod528b75dd_8015_4e02_810f_5a708f7173d5.slice/crio-c53621b7cfc8d0d546d3094a120473cebb2f79d9d362c67ac4462bfcc9e71005 WatchSource:0}: Error finding container c53621b7cfc8d0d546d3094a120473cebb2f79d9d362c67ac4462bfcc9e71005: Status 404 returned error can't find the container with id c53621b7cfc8d0d546d3094a120473cebb2f79d9d362c67ac4462bfcc9e71005
Sep 15 21:10:19.555093 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:19.555035    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7937-3c92ca" event={"ID":"528b75dd-8015-4e02-810f-5a708f7173d5","Type":"ContainerStarted","Data":"c53621b7cfc8d0d546d3094a120473cebb2f79d9d362c67ac4462bfcc9e71005"}
Sep 15 21:10:20.093633 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:20.093596    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:10:20.657412 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:20.657379    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:10:20.657687 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:20.657446    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:10:20.657687 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:20.657473    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:10:21.567583 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:21.567549    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw" event={"ID":"2ed5bb3c-b04d-4d34-a04f-f02c037fe755","Type":"ContainerStarted","Data":"92a2c5c3762fad6e01220efe922a07deb0a29179c0c209be51be669c63c16ec5"}
Sep 15 21:10:21.591029 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:21.590980    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw" podStartSLOduration=1.969500891 podStartE2EDuration="4.59096435s" totalImagesPullingTime="2.621463459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:10:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:21.58843296 +0000 UTC m=+740.079078702" watchObservedRunningTime="2026-09-15 21:10:21.59096435 +0000 UTC m=+740.081610151"
Sep 15 21:10:21.755592 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:21.755560    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:21.760457 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:21.760432    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:22.573567 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:22.573531    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:22.574493 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:22.574463    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:33.337702 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:33.337614    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"]
Sep 15 21:10:33.341218 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:33.341195    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"
Sep 15 21:10:33.344167 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:33.344143    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-bfe4d22a-serving-cert\""
Sep 15 21:10:33.345217 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:33.345192    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:10:33.345217 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:33.345207    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 21:10:33.345496 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:33.345477    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8hfhw\""
Sep 15 21:10:33.345560 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:33.345532    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 21:10:33.353930 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:33.353866    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"]
Sep 15 21:10:33.478927 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:33.478898    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b73f391a-b69f-4597-bed0-19a4d03bba3e-maas-api-tls\") pod \"maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6\" (UID: \"b73f391a-b69f-4597-bed0-19a4d03bba3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"
Sep 15 21:10:33.479087 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:33.478950    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b73f391a-b69f-4597-bed0-19a4d03bba3e-metrics-certs\") pod \"maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6\" (UID: \"b73f391a-b69f-4597-bed0-19a4d03bba3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"
Sep 15 21:10:33.479087 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:33.479009    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv26j\" (UniqueName: \"kubernetes.io/projected/b73f391a-b69f-4597-bed0-19a4d03bba3e-kube-api-access-nv26j\") pod \"maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6\" (UID: \"b73f391a-b69f-4597-bed0-19a4d03bba3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"
Sep 15 21:10:33.479087 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:33.479032    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73f391a-b69f-4597-bed0-19a4d03bba3e-openshift-service-ca\") pod \"maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6\" (UID: \"b73f391a-b69f-4597-bed0-19a4d03bba3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"
Sep 15 21:10:33.580275 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:33.580243    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b73f391a-b69f-4597-bed0-19a4d03bba3e-metrics-certs\") pod \"maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6\" (UID: \"b73f391a-b69f-4597-bed0-19a4d03bba3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"
Sep 15 21:10:33.580433 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:33.580304    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv26j\" (UniqueName: \"kubernetes.io/projected/b73f391a-b69f-4597-bed0-19a4d03bba3e-kube-api-access-nv26j\") pod \"maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6\" (UID: \"b73f391a-b69f-4597-bed0-19a4d03bba3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"
Sep 15 21:10:33.580433 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:33.580336    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73f391a-b69f-4597-bed0-19a4d03bba3e-openshift-service-ca\") pod \"maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6\" (UID: \"b73f391a-b69f-4597-bed0-19a4d03bba3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"
Sep 15 21:10:33.580433 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:33.580368    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b73f391a-b69f-4597-bed0-19a4d03bba3e-maas-api-tls\") pod \"maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6\" (UID: \"b73f391a-b69f-4597-bed0-19a4d03bba3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"
Sep 15 21:10:33.580546 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:10:33.580490    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-serving-cert: secret "maas-api-e2e-ait-bfe4d22a-serving-cert" not found
Sep 15 21:10:33.580580 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:10:33.580551    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b73f391a-b69f-4597-bed0-19a4d03bba3e-maas-api-tls podName:b73f391a-b69f-4597-bed0-19a4d03bba3e nodeName:}" failed. No retries permitted until 2026-09-15 21:10:34.080532951 +0000 UTC m=+752.571178661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b73f391a-b69f-4597-bed0-19a4d03bba3e-maas-api-tls") pod "maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6" (UID: "b73f391a-b69f-4597-bed0-19a4d03bba3e") : secret "maas-api-e2e-ait-bfe4d22a-serving-cert" not found
Sep 15 21:10:33.581116 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:33.581097    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73f391a-b69f-4597-bed0-19a4d03bba3e-openshift-service-ca\") pod \"maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6\" (UID: \"b73f391a-b69f-4597-bed0-19a4d03bba3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"
Sep 15 21:10:33.582644 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:33.582623    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b73f391a-b69f-4597-bed0-19a4d03bba3e-metrics-certs\") pod \"maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6\" (UID: \"b73f391a-b69f-4597-bed0-19a4d03bba3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"
Sep 15 21:10:33.591456 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:33.591405    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv26j\" (UniqueName: \"kubernetes.io/projected/b73f391a-b69f-4597-bed0-19a4d03bba3e-kube-api-access-nv26j\") pod \"maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6\" (UID: \"b73f391a-b69f-4597-bed0-19a4d03bba3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"
Sep 15 21:10:33.632330 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:33.632299    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7937-3c92ca" event={"ID":"528b75dd-8015-4e02-810f-5a708f7173d5","Type":"ContainerStarted","Data":"8f9e070904cbae3f15ba33afee83031892dceb800be8fc8117a5a2dc5970f99e"}
Sep 15 21:10:33.653964 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:33.653919    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7937-3c92ca" podStartSLOduration=2.261591349 podStartE2EDuration="15.653904264s" totalImagesPullingTime="13.392312915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:10:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:33.651032127 +0000 UTC m=+752.141677859" watchObservedRunningTime="2026-09-15 21:10:33.653904264 +0000 UTC m=+752.144549995"
Sep 15 21:10:34.085382 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:34.085351    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b73f391a-b69f-4597-bed0-19a4d03bba3e-maas-api-tls\") pod \"maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6\" (UID: \"b73f391a-b69f-4597-bed0-19a4d03bba3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"
Sep 15 21:10:34.087751 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:34.087728    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b73f391a-b69f-4597-bed0-19a4d03bba3e-maas-api-tls\") pod \"maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6\" (UID: \"b73f391a-b69f-4597-bed0-19a4d03bba3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"
Sep 15 21:10:34.255210 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:34.255172    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"
Sep 15 21:10:34.283973 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:34.283945    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7937-3c92ca"]
Sep 15 21:10:34.288608 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:34.288579    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7937-3c92ca"]
Sep 15 21:10:34.402492 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:34.402413    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7937-191953"]
Sep 15 21:10:34.402838 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:34.402795    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="528b75dd-8015-4e02-810f-5a708f7173d5" containerName="test-curl-7937-3c92ca"
Sep 15 21:10:34.402838 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:34.402810    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="528b75dd-8015-4e02-810f-5a708f7173d5" containerName="test-curl-7937-3c92ca"
Sep 15 21:10:34.402910 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:34.402857    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="528b75dd-8015-4e02-810f-5a708f7173d5" containerName="test-curl-7937-3c92ca"
Sep 15 21:10:34.406881 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:34.406866    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7937-191953"
Sep 15 21:10:34.410585 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:34.410559    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7937-191953"]
Sep 15 21:10:34.488125 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:34.488093    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh6k4\" (UniqueName: \"kubernetes.io/projected/dc612399-0690-4869-bbff-1afcf9d4dd64-kube-api-access-mh6k4\") pod \"test-curl-7937-191953\" (UID: \"dc612399-0690-4869-bbff-1afcf9d4dd64\") " pod="opendatahub/test-curl-7937-191953"
Sep 15 21:10:34.534353 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:34.534326    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"]
Sep 15 21:10:34.535880 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:10:34.535848    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73f391a_b69f_4597_bed0_19a4d03bba3e.slice/crio-31747ad03453a049ccac087786c21f84c64e4db2afd4592b57066f2b8fcbd46c WatchSource:0}: Error finding container 31747ad03453a049ccac087786c21f84c64e4db2afd4592b57066f2b8fcbd46c: Status 404 returned error can't find the container with id 31747ad03453a049ccac087786c21f84c64e4db2afd4592b57066f2b8fcbd46c
Sep 15 21:10:34.589084 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:34.589038    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh6k4\" (UniqueName: \"kubernetes.io/projected/dc612399-0690-4869-bbff-1afcf9d4dd64-kube-api-access-mh6k4\") pod \"test-curl-7937-191953\" (UID: \"dc612399-0690-4869-bbff-1afcf9d4dd64\") " pod="opendatahub/test-curl-7937-191953"
Sep 15 21:10:34.597854 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:34.597834    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh6k4\" (UniqueName: \"kubernetes.io/projected/dc612399-0690-4869-bbff-1afcf9d4dd64-kube-api-access-mh6k4\") pod \"test-curl-7937-191953\" (UID: \"dc612399-0690-4869-bbff-1afcf9d4dd64\") " pod="opendatahub/test-curl-7937-191953"
Sep 15 21:10:34.638737 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:34.638701    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6" event={"ID":"b73f391a-b69f-4597-bed0-19a4d03bba3e","Type":"ContainerStarted","Data":"31747ad03453a049ccac087786c21f84c64e4db2afd4592b57066f2b8fcbd46c"}
Sep 15 21:10:34.638950 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:10:34.638936    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7937-3c92ca" container="test-curl-7937-3c92ca"
Sep 15 21:10:34.718143 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:34.718113    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7937-191953"
Sep 15 21:10:34.985556 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:34.985534    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7937-191953"]
Sep 15 21:10:34.989828 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:10:34.989799    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc612399_0690_4869_bbff_1afcf9d4dd64.slice/crio-3f4996c147fed17bafc0f87b339faa6a40088fe85cdc0075aa76d7ab1c3068a5 WatchSource:0}: Error finding container 3f4996c147fed17bafc0f87b339faa6a40088fe85cdc0075aa76d7ab1c3068a5: Status 404 returned error can't find the container with id 3f4996c147fed17bafc0f87b339faa6a40088fe85cdc0075aa76d7ab1c3068a5
Sep 15 21:10:35.631444 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.631408    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"]
Sep 15 21:10:35.635245 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.635222    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.638464 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.638265    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-253ca1-openshift-default-dockercfg-5hwrq\""
Sep 15 21:10:35.649437 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.649149    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"]
Sep 15 21:10:35.649991 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.649937    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7937-3c92ca" podUID="528b75dd-8015-4e02-810f-5a708f7173d5" containerName="test-curl-7937-3c92ca" containerID="cri-o://8f9e070904cbae3f15ba33afee83031892dceb800be8fc8117a5a2dc5970f99e" gracePeriod=2
Sep 15 21:10:35.650165 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.650135    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7937-191953" event={"ID":"dc612399-0690-4869-bbff-1afcf9d4dd64","Type":"ContainerStarted","Data":"2854b9cb803f934f5c6ba5228fbf0657380144db4179111f2ae4b554d06667db"}
Sep 15 21:10:35.650235 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.650179    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7937-191953" event={"ID":"dc612399-0690-4869-bbff-1afcf9d4dd64","Type":"ContainerStarted","Data":"3f4996c147fed17bafc0f87b339faa6a40088fe85cdc0075aa76d7ab1c3068a5"}
Sep 15 21:10:35.684816 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.684766    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7937-191953" podStartSLOduration=1.684746354 podStartE2EDuration="1.684746354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:35.684243798 +0000 UTC m=+754.174889531" watchObservedRunningTime="2026-09-15 21:10:35.684746354 +0000 UTC m=+754.175392090"
Sep 15 21:10:35.686595 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:10:35.686565    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7937-3c92ca\" is forbidden: User \"system:node:ip-10-0-142-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-207.ec2.internal' and this object" podUID="528b75dd-8015-4e02-810f-5a708f7173d5" pod="opendatahub/test-curl-7937-3c92ca"
Sep 15 21:10:35.800031 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.799993    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de574df-a42f-4b09-ab0f-ba38a26d6968-istio-token\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.800211 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.800064    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfbqq\" (UniqueName: \"kubernetes.io/projected/5de574df-a42f-4b09-ab0f-ba38a26d6968-kube-api-access-qfbqq\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.800211 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.800154    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-workload-socket\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.800321 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.800229    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-istio-envoy\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.800321 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.800257    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de574df-a42f-4b09-ab0f-ba38a26d6968-istiod-ca-cert\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.800321 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.800282    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-credential-socket\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.800448 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.800388    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-workload-certs\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.800514 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.800489    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de574df-a42f-4b09-ab0f-ba38a26d6968-istio-podinfo\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.800558 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.800536    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-istio-data\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.901515 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.901439    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-workload-socket\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.901515 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.901488    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-istio-envoy\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.901715 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.901630    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de574df-a42f-4b09-ab0f-ba38a26d6968-istiod-ca-cert\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.901715 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.901675    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-credential-socket\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.901837 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.901724    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-workload-certs\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.901837 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.901806    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de574df-a42f-4b09-ab0f-ba38a26d6968-istio-podinfo\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.901937 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.901845    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-istio-data\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.901937 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.901890    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de574df-a42f-4b09-ab0f-ba38a26d6968-istio-token\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.901937 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.901924    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfbqq\" (UniqueName: \"kubernetes.io/projected/5de574df-a42f-4b09-ab0f-ba38a26d6968-kube-api-access-qfbqq\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.902122 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.902100    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-credential-socket\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.902429 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.902392    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-istio-data\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.902553 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.901839    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-workload-socket\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.902619 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.902591    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de574df-a42f-4b09-ab0f-ba38a26d6968-istiod-ca-cert\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.903115 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.902951    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-workload-certs\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.904448 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.904429    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-istio-envoy\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.905911 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.905889    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de574df-a42f-4b09-ab0f-ba38a26d6968-istio-podinfo\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.912576 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.912548    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfbqq\" (UniqueName: \"kubernetes.io/projected/5de574df-a42f-4b09-ab0f-ba38a26d6968-kube-api-access-qfbqq\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.912819 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.912794    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de574df-a42f-4b09-ab0f-ba38a26d6968-istio-token\") pod \"e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") " pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:35.954841 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:35.954813    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:36.370785 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:36.370749    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7937-191953"]
Sep 15 21:10:36.384798 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:36.384762    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7937-191953"]
Sep 15 21:10:36.655703 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:10:36.655629    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7937-191953" container="test-curl-7937-191953"
Sep 15 21:10:37.160513 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.160368    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"]
Sep 15 21:10:37.163170 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:10:37.163145    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de574df_a42f_4b09_ab0f_ba38a26d6968.slice/crio-12c9c3e2c03092da5e194989037a8c2aba77ec86415fb5b8bd26d032905baa1d WatchSource:0}: Error finding container 12c9c3e2c03092da5e194989037a8c2aba77ec86415fb5b8bd26d032905baa1d: Status 404 returned error can't find the container with id 12c9c3e2c03092da5e194989037a8c2aba77ec86415fb5b8bd26d032905baa1d
Sep 15 21:10:37.165689 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.165657    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:10:37.165760 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.165723    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:10:37.165760 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.165748    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:10:37.241377 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.241343    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"]
Sep 15 21:10:37.241685 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.241672    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc612399-0690-4869-bbff-1afcf9d4dd64" containerName="test-curl-7937-191953"
Sep 15 21:10:37.241685 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.241686    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc612399-0690-4869-bbff-1afcf9d4dd64" containerName="test-curl-7937-191953"
Sep 15 21:10:37.241782 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.241751    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc612399-0690-4869-bbff-1afcf9d4dd64" containerName="test-curl-7937-191953"
Sep 15 21:10:37.244779 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.244758    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.248480 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.248456    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-05731f-openshift-default-dockercfg-v5h8j\""
Sep 15 21:10:37.265981 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.265953    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"]
Sep 15 21:10:37.416617 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.416573    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istio-envoy\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.416617 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.416616    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istiod-ca-cert\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.416856 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.416686    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-credential-socket\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.416856 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.416733    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istio-podinfo\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.416856 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.416775    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqwtc\" (UniqueName: \"kubernetes.io/projected/4f826e85-24be-4260-be0f-ff7c30ae5f2e-kube-api-access-sqwtc\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.416856 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.416848    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-workload-socket\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.417124 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.416876    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istio-data\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.417124 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.416896    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-workload-certs\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.417124 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.416952    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istio-token\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.517783 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.517739    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istio-podinfo\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.517783 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.517773    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqwtc\" (UniqueName: \"kubernetes.io/projected/4f826e85-24be-4260-be0f-ff7c30ae5f2e-kube-api-access-sqwtc\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.518014 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.517817    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-workload-socket\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.518014 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.517847    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istio-data\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.518014 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.517874    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-workload-certs\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.518014 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.517948    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istio-token\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.518014 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.518001    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istio-envoy\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.518299 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.518025    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istiod-ca-cert\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.518299 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.518095    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-credential-socket\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.518421 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.518320    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-workload-socket\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.518421 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.518345    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istio-data\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.518421 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.518393    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-workload-certs\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.518575 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.518499    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-credential-socket\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.518840 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.518813    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istiod-ca-cert\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.520328 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.520302    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istio-envoy\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.520505 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.520484    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istio-podinfo\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.528622 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.528598    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istio-token\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.529356 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.529335    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqwtc\" (UniqueName: \"kubernetes.io/projected/4f826e85-24be-4260-be0f-ff7c30ae5f2e-kube-api-access-sqwtc\") pod \"e2e-autores-05731f-openshift-default-675d57d64b-gpd89\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") " pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.555853 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.555831    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:37.663383 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.663349    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6" event={"ID":"b73f391a-b69f-4597-bed0-19a4d03bba3e","Type":"ContainerStarted","Data":"f53bb27938ef2fddc743e952ef47320b09f0f0b34f5d58c28c05b12d80a3c39c"}
Sep 15 21:10:37.663792 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.663556    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"
Sep 15 21:10:37.665437 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.665411    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst" event={"ID":"5de574df-a42f-4b09-ab0f-ba38a26d6968","Type":"ContainerStarted","Data":"868e6be7774c9b2972671d0c08024db5f6ec63ef2962b6c4392e48d1abac2cbb"}
Sep 15 21:10:37.665554 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.665460    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst" event={"ID":"5de574df-a42f-4b09-ab0f-ba38a26d6968","Type":"ContainerStarted","Data":"12c9c3e2c03092da5e194989037a8c2aba77ec86415fb5b8bd26d032905baa1d"}
Sep 15 21:10:37.665554 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.665499    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7937-191953" podUID="dc612399-0690-4869-bbff-1afcf9d4dd64" containerName="test-curl-7937-191953" containerID="cri-o://2854b9cb803f934f5c6ba5228fbf0657380144db4179111f2ae4b554d06667db" gracePeriod=2
Sep 15 21:10:37.685352 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.685262    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6" podStartSLOduration=2.3011966409999998 podStartE2EDuration="4.685246683s" totalImagesPullingTime="2.384050042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:10:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:37.683272429 +0000 UTC m=+756.173918162" watchObservedRunningTime="2026-09-15 21:10:37.685246683 +0000 UTC m=+756.175892414"
Sep 15 21:10:37.685594 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:10:37.685567    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7937-191953\" is forbidden: User \"system:node:ip-10-0-142-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-207.ec2.internal' and this object" podUID="dc612399-0690-4869-bbff-1afcf9d4dd64" pod="opendatahub/test-curl-7937-191953"
Sep 15 21:10:37.713694 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.713644    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst" podStartSLOduration=2.713628906 podStartE2EDuration="2.713628906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:37.711326369 +0000 UTC m=+756.201972135" watchObservedRunningTime="2026-09-15 21:10:37.713628906 +0000 UTC m=+756.204274698"
Sep 15 21:10:37.814555 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.814530    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7937-3c92ca"
Sep 15 21:10:37.817679 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:10:37.817649    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7937-191953\" is forbidden: User \"system:node:ip-10-0-142-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-207.ec2.internal' and this object" podUID="dc612399-0690-4869-bbff-1afcf9d4dd64" pod="opendatahub/test-curl-7937-191953"
Sep 15 21:10:37.833422 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.833398    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"]
Sep 15 21:10:37.835755 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:10:37.835729    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f826e85_24be_4260_be0f_ff7c30ae5f2e.slice/crio-3d87c8e7d653ecac0f797a00ff980fe13b9ad5b71e042371b81ab280c79c9063 WatchSource:0}: Error finding container 3d87c8e7d653ecac0f797a00ff980fe13b9ad5b71e042371b81ab280c79c9063: Status 404 returned error can't find the container with id 3d87c8e7d653ecac0f797a00ff980fe13b9ad5b71e042371b81ab280c79c9063
Sep 15 21:10:37.838185 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.838154    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:10:37.838270 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.838220    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:10:37.838270 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.838249    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:10:37.921829 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.921807    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcxvl\" (UniqueName: \"kubernetes.io/projected/528b75dd-8015-4e02-810f-5a708f7173d5-kube-api-access-hcxvl\") pod \"528b75dd-8015-4e02-810f-5a708f7173d5\" (UID: \"528b75dd-8015-4e02-810f-5a708f7173d5\") "
Sep 15 21:10:37.923774 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.923747    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528b75dd-8015-4e02-810f-5a708f7173d5-kube-api-access-hcxvl" (OuterVolumeSpecName: "kube-api-access-hcxvl") pod "528b75dd-8015-4e02-810f-5a708f7173d5" (UID: "528b75dd-8015-4e02-810f-5a708f7173d5"). InnerVolumeSpecName "kube-api-access-hcxvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:10:37.944723 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.944650    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="528b75dd-8015-4e02-810f-5a708f7173d5" path="/var/lib/kubelet/pods/528b75dd-8015-4e02-810f-5a708f7173d5/volumes"
Sep 15 21:10:37.955679 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:37.955660    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:38.023422 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:38.023383    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcxvl\" (UniqueName: \"kubernetes.io/projected/528b75dd-8015-4e02-810f-5a708f7173d5-kube-api-access-hcxvl\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:10:38.672626 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:38.672592    2558 generic.go:356] "Generic (PLEG): container finished" podID="528b75dd-8015-4e02-810f-5a708f7173d5" containerID="8f9e070904cbae3f15ba33afee83031892dceb800be8fc8117a5a2dc5970f99e" exitCode=137
Sep 15 21:10:38.673091 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:38.672633    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7937-3c92ca"
Sep 15 21:10:38.673091 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:38.672690    2558 scope.go:117] "RemoveContainer" containerID="8f9e070904cbae3f15ba33afee83031892dceb800be8fc8117a5a2dc5970f99e"
Sep 15 21:10:38.674556 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:38.674471    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89" event={"ID":"4f826e85-24be-4260-be0f-ff7c30ae5f2e","Type":"ContainerStarted","Data":"3078e7b4937cefa97ce6caf61725654ebae1972097365c0411e6a8dffa05e3c1"}
Sep 15 21:10:38.674556 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:38.674540    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89" event={"ID":"4f826e85-24be-4260-be0f-ff7c30ae5f2e","Type":"ContainerStarted","Data":"3d87c8e7d653ecac0f797a00ff980fe13b9ad5b71e042371b81ab280c79c9063"}
Sep 15 21:10:38.681552 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:38.681536    2558 scope.go:117] "RemoveContainer" containerID="8f9e070904cbae3f15ba33afee83031892dceb800be8fc8117a5a2dc5970f99e"
Sep 15 21:10:38.681804 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:10:38.681785    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f9e070904cbae3f15ba33afee83031892dceb800be8fc8117a5a2dc5970f99e\": container with ID starting with 8f9e070904cbae3f15ba33afee83031892dceb800be8fc8117a5a2dc5970f99e not found: ID does not exist" containerID="8f9e070904cbae3f15ba33afee83031892dceb800be8fc8117a5a2dc5970f99e"
Sep 15 21:10:38.681856 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:38.681812    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9e070904cbae3f15ba33afee83031892dceb800be8fc8117a5a2dc5970f99e"} err="failed to get container status \"8f9e070904cbae3f15ba33afee83031892dceb800be8fc8117a5a2dc5970f99e\": rpc error: code = NotFound desc = could not find container \"8f9e070904cbae3f15ba33afee83031892dceb800be8fc8117a5a2dc5970f99e\": container with ID starting with 8f9e070904cbae3f15ba33afee83031892dceb800be8fc8117a5a2dc5970f99e not found: ID does not exist"
Sep 15 21:10:38.701902 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:38.701850    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89" podStartSLOduration=1.701831076 podStartE2EDuration="1.701831076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:38.698995749 +0000 UTC m=+757.189641481" watchObservedRunningTime="2026-09-15 21:10:38.701831076 +0000 UTC m=+757.192476810"
Sep 15 21:10:38.979753 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:38.979672    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:39.556357 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:39.556323    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:39.561280 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:39.561258    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:39.686252 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:39.686223    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:39.686623 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:39.686308    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:39.687001 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:39.686978    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:10:39.687952 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:39.687934    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:10:39.861739 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:39.861719    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7937-191953"
Sep 15 21:10:39.916987 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:39.916953    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:10:39.942852 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:39.942812    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mh6k4\" (UniqueName: \"kubernetes.io/projected/dc612399-0690-4869-bbff-1afcf9d4dd64-kube-api-access-mh6k4\") pod \"dc612399-0690-4869-bbff-1afcf9d4dd64\" (UID: \"dc612399-0690-4869-bbff-1afcf9d4dd64\") "
Sep 15 21:10:39.944962 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:39.944930    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc612399-0690-4869-bbff-1afcf9d4dd64-kube-api-access-mh6k4" (OuterVolumeSpecName: "kube-api-access-mh6k4") pod "dc612399-0690-4869-bbff-1afcf9d4dd64" (UID: "dc612399-0690-4869-bbff-1afcf9d4dd64"). InnerVolumeSpecName "kube-api-access-mh6k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:10:40.044473 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:40.044434    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh6k4\" (UniqueName: \"kubernetes.io/projected/dc612399-0690-4869-bbff-1afcf9d4dd64-kube-api-access-mh6k4\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:10:40.694274 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:40.694243    2558 generic.go:356] "Generic (PLEG): container finished" podID="dc612399-0690-4869-bbff-1afcf9d4dd64" containerID="2854b9cb803f934f5c6ba5228fbf0657380144db4179111f2ae4b554d06667db" exitCode=137
Sep 15 21:10:40.694730 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:40.694292    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7937-191953"
Sep 15 21:10:40.694730 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:40.694343    2558 scope.go:117] "RemoveContainer" containerID="2854b9cb803f934f5c6ba5228fbf0657380144db4179111f2ae4b554d06667db"
Sep 15 21:10:40.705447 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:40.705430    2558 scope.go:117] "RemoveContainer" containerID="2854b9cb803f934f5c6ba5228fbf0657380144db4179111f2ae4b554d06667db"
Sep 15 21:10:40.705743 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:10:40.705724    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2854b9cb803f934f5c6ba5228fbf0657380144db4179111f2ae4b554d06667db\": container with ID starting with 2854b9cb803f934f5c6ba5228fbf0657380144db4179111f2ae4b554d06667db not found: ID does not exist" containerID="2854b9cb803f934f5c6ba5228fbf0657380144db4179111f2ae4b554d06667db"
Sep 15 21:10:40.705809 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:40.705753    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2854b9cb803f934f5c6ba5228fbf0657380144db4179111f2ae4b554d06667db"} err="failed to get container status \"2854b9cb803f934f5c6ba5228fbf0657380144db4179111f2ae4b554d06667db\": rpc error: code = NotFound desc = could not find container \"2854b9cb803f934f5c6ba5228fbf0657380144db4179111f2ae4b554d06667db\": container with ID starting with 2854b9cb803f934f5c6ba5228fbf0657380144db4179111f2ae4b554d06667db not found: ID does not exist"
Sep 15 21:10:41.942162 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:41.942124    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc612399-0690-4869-bbff-1afcf9d4dd64" path="/var/lib/kubelet/pods/dc612399-0690-4869-bbff-1afcf9d4dd64/volumes"
Sep 15 21:10:43.680135 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:43.680102    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"
Sep 15 21:10:45.448518 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:45.448481    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"]
Sep 15 21:10:45.459589 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:45.459567    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"
Sep 15 21:10:45.462952 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:45.462932    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-05731f-serving-cert\""
Sep 15 21:10:45.482340 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:45.482287    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"]
Sep 15 21:10:45.600243 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:45.600210    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c21937f-e198-4bd9-8f77-03f6d9ce455f-maas-api-tls\") pod \"maas-api-e2e-autores-05731f-7d5c66c988-xrgp9\" (UID: \"4c21937f-e198-4bd9-8f77-03f6d9ce455f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"
Sep 15 21:10:45.600243 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:45.600245    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c21937f-e198-4bd9-8f77-03f6d9ce455f-openshift-service-ca\") pod \"maas-api-e2e-autores-05731f-7d5c66c988-xrgp9\" (UID: \"4c21937f-e198-4bd9-8f77-03f6d9ce455f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"
Sep 15 21:10:45.600446 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:45.600317    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c21937f-e198-4bd9-8f77-03f6d9ce455f-metrics-certs\") pod \"maas-api-e2e-autores-05731f-7d5c66c988-xrgp9\" (UID: \"4c21937f-e198-4bd9-8f77-03f6d9ce455f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"
Sep 15 21:10:45.600446 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:45.600418    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p74dl\" (UniqueName: \"kubernetes.io/projected/4c21937f-e198-4bd9-8f77-03f6d9ce455f-kube-api-access-p74dl\") pod \"maas-api-e2e-autores-05731f-7d5c66c988-xrgp9\" (UID: \"4c21937f-e198-4bd9-8f77-03f6d9ce455f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"
Sep 15 21:10:45.700883 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:45.700803    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p74dl\" (UniqueName: \"kubernetes.io/projected/4c21937f-e198-4bd9-8f77-03f6d9ce455f-kube-api-access-p74dl\") pod \"maas-api-e2e-autores-05731f-7d5c66c988-xrgp9\" (UID: \"4c21937f-e198-4bd9-8f77-03f6d9ce455f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"
Sep 15 21:10:45.701190 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:45.701155    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c21937f-e198-4bd9-8f77-03f6d9ce455f-maas-api-tls\") pod \"maas-api-e2e-autores-05731f-7d5c66c988-xrgp9\" (UID: \"4c21937f-e198-4bd9-8f77-03f6d9ce455f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"
Sep 15 21:10:45.701286 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:45.701198    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c21937f-e198-4bd9-8f77-03f6d9ce455f-openshift-service-ca\") pod \"maas-api-e2e-autores-05731f-7d5c66c988-xrgp9\" (UID: \"4c21937f-e198-4bd9-8f77-03f6d9ce455f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"
Sep 15 21:10:45.701286 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:10:45.701162    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-05731f-serving-cert: secret "maas-api-e2e-autores-05731f-serving-cert" not found
Sep 15 21:10:45.701286 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:45.701265    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c21937f-e198-4bd9-8f77-03f6d9ce455f-metrics-certs\") pod \"maas-api-e2e-autores-05731f-7d5c66c988-xrgp9\" (UID: \"4c21937f-e198-4bd9-8f77-03f6d9ce455f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"
Sep 15 21:10:45.701458 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:10:45.701304    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c21937f-e198-4bd9-8f77-03f6d9ce455f-maas-api-tls podName:4c21937f-e198-4bd9-8f77-03f6d9ce455f nodeName:}" failed. No retries permitted until 2026-09-15 21:10:46.201285801 +0000 UTC m=+764.691931511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4c21937f-e198-4bd9-8f77-03f6d9ce455f-maas-api-tls") pod "maas-api-e2e-autores-05731f-7d5c66c988-xrgp9" (UID: "4c21937f-e198-4bd9-8f77-03f6d9ce455f") : secret "maas-api-e2e-autores-05731f-serving-cert" not found
Sep 15 21:10:45.702033 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:45.702002    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c21937f-e198-4bd9-8f77-03f6d9ce455f-openshift-service-ca\") pod \"maas-api-e2e-autores-05731f-7d5c66c988-xrgp9\" (UID: \"4c21937f-e198-4bd9-8f77-03f6d9ce455f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"
Sep 15 21:10:45.703581 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:45.703564    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c21937f-e198-4bd9-8f77-03f6d9ce455f-metrics-certs\") pod \"maas-api-e2e-autores-05731f-7d5c66c988-xrgp9\" (UID: \"4c21937f-e198-4bd9-8f77-03f6d9ce455f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"
Sep 15 21:10:45.709824 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:45.709803    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p74dl\" (UniqueName: \"kubernetes.io/projected/4c21937f-e198-4bd9-8f77-03f6d9ce455f-kube-api-access-p74dl\") pod \"maas-api-e2e-autores-05731f-7d5c66c988-xrgp9\" (UID: \"4c21937f-e198-4bd9-8f77-03f6d9ce455f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"
Sep 15 21:10:46.205866 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:46.205831    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c21937f-e198-4bd9-8f77-03f6d9ce455f-maas-api-tls\") pod \"maas-api-e2e-autores-05731f-7d5c66c988-xrgp9\" (UID: \"4c21937f-e198-4bd9-8f77-03f6d9ce455f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"
Sep 15 21:10:46.206012 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:10:46.205992    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-05731f-serving-cert: secret "maas-api-e2e-autores-05731f-serving-cert" not found
Sep 15 21:10:46.206086 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:10:46.206074    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c21937f-e198-4bd9-8f77-03f6d9ce455f-maas-api-tls podName:4c21937f-e198-4bd9-8f77-03f6d9ce455f nodeName:}" failed. No retries permitted until 2026-09-15 21:10:47.206035805 +0000 UTC m=+765.696681516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4c21937f-e198-4bd9-8f77-03f6d9ce455f-maas-api-tls") pod "maas-api-e2e-autores-05731f-7d5c66c988-xrgp9" (UID: "4c21937f-e198-4bd9-8f77-03f6d9ce455f") : secret "maas-api-e2e-autores-05731f-serving-cert" not found
Sep 15 21:10:47.217066 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:47.217013    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c21937f-e198-4bd9-8f77-03f6d9ce455f-maas-api-tls\") pod \"maas-api-e2e-autores-05731f-7d5c66c988-xrgp9\" (UID: \"4c21937f-e198-4bd9-8f77-03f6d9ce455f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"
Sep 15 21:10:47.219468 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:47.219446    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c21937f-e198-4bd9-8f77-03f6d9ce455f-maas-api-tls\") pod \"maas-api-e2e-autores-05731f-7d5c66c988-xrgp9\" (UID: \"4c21937f-e198-4bd9-8f77-03f6d9ce455f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"
Sep 15 21:10:47.281027 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:47.280998    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"
Sep 15 21:10:47.583789 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:47.583764    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"]
Sep 15 21:10:47.585521 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:10:47.585490    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c21937f_e198_4bd9_8f77_03f6d9ce455f.slice/crio-45782108a9fb3e9397334764519298f5ff7f00f4a50600cd3fd9e9b8e1822515 WatchSource:0}: Error finding container 45782108a9fb3e9397334764519298f5ff7f00f4a50600cd3fd9e9b8e1822515: Status 404 returned error can't find the container with id 45782108a9fb3e9397334764519298f5ff7f00f4a50600cd3fd9e9b8e1822515
Sep 15 21:10:47.731829 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:47.731788    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9" event={"ID":"4c21937f-e198-4bd9-8f77-03f6d9ce455f","Type":"ContainerStarted","Data":"45782108a9fb3e9397334764519298f5ff7f00f4a50600cd3fd9e9b8e1822515"}
Sep 15 21:10:48.739886 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:48.739845    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9" event={"ID":"4c21937f-e198-4bd9-8f77-03f6d9ce455f","Type":"ContainerStarted","Data":"75d4f67b1674a2b6775531431f15c48824807a1ba3fb73446f08fe7adf6a68b1"}
Sep 15 21:10:48.740511 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:48.740067    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"
Sep 15 21:10:48.763771 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:48.763718    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9" podStartSLOduration=3.418958367 podStartE2EDuration="3.763700954s" totalImagesPullingTime="344.742587ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:48.762997913 +0000 UTC m=+767.253643646" watchObservedRunningTime="2026-09-15 21:10:48.763700954 +0000 UTC m=+767.254346686"
Sep 15 21:10:51.758602 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:51.758567    2558 generic.go:356] "Generic (PLEG): container finished" podID="2ed5bb3c-b04d-4d34-a04f-f02c037fe755" containerID="92a2c5c3762fad6e01220efe922a07deb0a29179c0c209be51be669c63c16ec5" exitCode=0
Sep 15 21:10:51.759037 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:51.758644    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw" event={"ID":"2ed5bb3c-b04d-4d34-a04f-f02c037fe755","Type":"ContainerDied","Data":"92a2c5c3762fad6e01220efe922a07deb0a29179c0c209be51be669c63c16ec5"}
Sep 15 21:10:51.759037 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:51.759033    2558 scope.go:117] "RemoveContainer" containerID="92a2c5c3762fad6e01220efe922a07deb0a29179c0c209be51be669c63c16ec5"
Sep 15 21:10:51.760140 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:51.760113    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:10:51.760196 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:51.760180    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:10:51.760231 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:51.760211    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:10:52.574477 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:52.574434    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:52.770527 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:52.770482    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw" event={"ID":"2ed5bb3c-b04d-4d34-a04f-f02c037fe755","Type":"ContainerStarted","Data":"096fe319cde25ac53ab6203b827fb00f01127a47eec857288d3d9ffab9d637c0"}
Sep 15 21:10:53.755255 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:53.755224    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:53.756810 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:53.756778    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 15 21:10:53.756949 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:53.756843    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw" podUID="2ed5bb3c-b04d-4d34-a04f-f02c037fe755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 15 21:10:54.083966 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.083892    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"]
Sep 15 21:10:54.091883 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.091863    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:10:54.096492 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.096467    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-dca193\"/\"default-dockercfg-9254x\""
Sep 15 21:10:54.096619 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.096534    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-dca193\"/\"simulator-w4-dca193-kserve-self-signed-certs\""
Sep 15 21:10:54.096619 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.096539    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-dca193\"/\"kube-root-ca.crt\""
Sep 15 21:10:54.096619 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.096573    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-dca193\"/\"openshift-service-ca.crt\""
Sep 15 21:10:54.097864 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.097842    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"]
Sep 15 21:10:54.187732 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.187695    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5c14493-efa5-435f-b1bb-c5fd7491f62e-model-cache\") pod \"simulator-w4-dca193-kserve-7db95ff868-rb6s6\" (UID: \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\") " pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:10:54.187904 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.187760    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5c14493-efa5-435f-b1bb-c5fd7491f62e-home\") pod \"simulator-w4-dca193-kserve-7db95ff868-rb6s6\" (UID: \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\") " pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:10:54.187904 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.187800    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-297br\" (UniqueName: \"kubernetes.io/projected/f5c14493-efa5-435f-b1bb-c5fd7491f62e-kube-api-access-297br\") pod \"simulator-w4-dca193-kserve-7db95ff868-rb6s6\" (UID: \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\") " pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:10:54.187988 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.187923    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5c14493-efa5-435f-b1bb-c5fd7491f62e-dshm\") pod \"simulator-w4-dca193-kserve-7db95ff868-rb6s6\" (UID: \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\") " pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:10:54.187988 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.187951    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5c14493-efa5-435f-b1bb-c5fd7491f62e-tmp-dir\") pod \"simulator-w4-dca193-kserve-7db95ff868-rb6s6\" (UID: \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\") " pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:10:54.187988 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.187972    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5c14493-efa5-435f-b1bb-c5fd7491f62e-tls-certs\") pod \"simulator-w4-dca193-kserve-7db95ff868-rb6s6\" (UID: \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\") " pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:10:54.288572 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.288533    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-297br\" (UniqueName: \"kubernetes.io/projected/f5c14493-efa5-435f-b1bb-c5fd7491f62e-kube-api-access-297br\") pod \"simulator-w4-dca193-kserve-7db95ff868-rb6s6\" (UID: \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\") " pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:10:54.288758 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.288674    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5c14493-efa5-435f-b1bb-c5fd7491f62e-dshm\") pod \"simulator-w4-dca193-kserve-7db95ff868-rb6s6\" (UID: \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\") " pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:10:54.288758 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.288691    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5c14493-efa5-435f-b1bb-c5fd7491f62e-tmp-dir\") pod \"simulator-w4-dca193-kserve-7db95ff868-rb6s6\" (UID: \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\") " pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:10:54.288758 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.288707    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5c14493-efa5-435f-b1bb-c5fd7491f62e-tls-certs\") pod \"simulator-w4-dca193-kserve-7db95ff868-rb6s6\" (UID: \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\") " pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:10:54.288758 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.288731    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5c14493-efa5-435f-b1bb-c5fd7491f62e-model-cache\") pod \"simulator-w4-dca193-kserve-7db95ff868-rb6s6\" (UID: \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\") " pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:10:54.289004 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.288773    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5c14493-efa5-435f-b1bb-c5fd7491f62e-home\") pod \"simulator-w4-dca193-kserve-7db95ff868-rb6s6\" (UID: \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\") " pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:10:54.289222 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.289193    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5c14493-efa5-435f-b1bb-c5fd7491f62e-tmp-dir\") pod \"simulator-w4-dca193-kserve-7db95ff868-rb6s6\" (UID: \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\") " pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:10:54.289330 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.289230    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5c14493-efa5-435f-b1bb-c5fd7491f62e-model-cache\") pod \"simulator-w4-dca193-kserve-7db95ff868-rb6s6\" (UID: \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\") " pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:10:54.289330 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.289257    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5c14493-efa5-435f-b1bb-c5fd7491f62e-home\") pod \"simulator-w4-dca193-kserve-7db95ff868-rb6s6\" (UID: \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\") " pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:10:54.291164 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.291137    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5c14493-efa5-435f-b1bb-c5fd7491f62e-dshm\") pod \"simulator-w4-dca193-kserve-7db95ff868-rb6s6\" (UID: \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\") " pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:10:54.291420 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.291404    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5c14493-efa5-435f-b1bb-c5fd7491f62e-tls-certs\") pod \"simulator-w4-dca193-kserve-7db95ff868-rb6s6\" (UID: \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\") " pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:10:54.296550 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.296528    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-297br\" (UniqueName: \"kubernetes.io/projected/f5c14493-efa5-435f-b1bb-c5fd7491f62e-kube-api-access-297br\") pod \"simulator-w4-dca193-kserve-7db95ff868-rb6s6\" (UID: \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\") " pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:10:54.401934 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.401851    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:10:54.715683 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.715654    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"]
Sep 15 21:10:54.717255 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:10:54.717226    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5c14493_efa5_435f_b1bb_c5fd7491f62e.slice/crio-9c67fe84b0a169eaeac62e1343aa333bdda11544783c802df656e23740185b08 WatchSource:0}: Error finding container 9c67fe84b0a169eaeac62e1343aa333bdda11544783c802df656e23740185b08: Status 404 returned error can't find the container with id 9c67fe84b0a169eaeac62e1343aa333bdda11544783c802df656e23740185b08
Sep 15 21:10:54.755779 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.755741    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 15 21:10:54.755948 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.755814    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw" podUID="2ed5bb3c-b04d-4d34-a04f-f02c037fe755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 15 21:10:54.761476 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.760430    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"
Sep 15 21:10:54.783532 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:54.783496    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6" event={"ID":"f5c14493-efa5-435f-b1bb-c5fd7491f62e","Type":"ContainerStarted","Data":"9c67fe84b0a169eaeac62e1343aa333bdda11544783c802df656e23740185b08"}
Sep 15 21:10:55.759082 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:55.756969    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 15 21:10:55.759082 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:55.757035    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw" podUID="2ed5bb3c-b04d-4d34-a04f-f02c037fe755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 15 21:10:55.798851 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:55.798819    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6" event={"ID":"f5c14493-efa5-435f-b1bb-c5fd7491f62e","Type":"ContainerStarted","Data":"aff9b49f4d7862b275da127a6628fb1f6eae5b532436cfd01ed6490b5ea3774e"}
Sep 15 21:10:55.819589 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:55.819537    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6" podStartSLOduration=1.819520002 podStartE2EDuration="1.819520002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:55.817122307 +0000 UTC m=+774.307768036" watchObservedRunningTime="2026-09-15 21:10:55.819520002 +0000 UTC m=+774.310165734"
Sep 15 21:10:56.756222 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:56.755770    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 15 21:10:56.756222 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:56.755832    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw" podUID="2ed5bb3c-b04d-4d34-a04f-f02c037fe755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 15 21:10:57.755446 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:57.755398    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 15 21:10:57.755944 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:57.755472    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw" podUID="2ed5bb3c-b04d-4d34-a04f-f02c037fe755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 15 21:10:58.768562 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:58.768523    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:58.769676 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:58.769646    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:10:58.770598 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:10:58.770570    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:11:03.719134 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:03.719100    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"]
Sep 15 21:11:03.719599 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:03.719371    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6" podUID="b73f391a-b69f-4597-bed0-19a4d03bba3e" containerName="maas-api" containerID="cri-o://f53bb27938ef2fddc743e952ef47320b09f0f0b34f5d58c28c05b12d80a3c39c" gracePeriod=30
Sep 15 21:11:03.856892 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:03.856643    2558 generic.go:356] "Generic (PLEG): container finished" podID="b73f391a-b69f-4597-bed0-19a4d03bba3e" containerID="f53bb27938ef2fddc743e952ef47320b09f0f0b34f5d58c28c05b12d80a3c39c" exitCode=0
Sep 15 21:11:03.856892 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:03.856833    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6" event={"ID":"b73f391a-b69f-4597-bed0-19a4d03bba3e","Type":"ContainerDied","Data":"f53bb27938ef2fddc743e952ef47320b09f0f0b34f5d58c28c05b12d80a3c39c"}
Sep 15 21:11:03.982197 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:03.982126    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"
Sep 15 21:11:04.097761 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.093671    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73f391a-b69f-4597-bed0-19a4d03bba3e-openshift-service-ca\") pod \"b73f391a-b69f-4597-bed0-19a4d03bba3e\" (UID: \"b73f391a-b69f-4597-bed0-19a4d03bba3e\") "
Sep 15 21:11:04.097761 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.093725    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nv26j\" (UniqueName: \"kubernetes.io/projected/b73f391a-b69f-4597-bed0-19a4d03bba3e-kube-api-access-nv26j\") pod \"b73f391a-b69f-4597-bed0-19a4d03bba3e\" (UID: \"b73f391a-b69f-4597-bed0-19a4d03bba3e\") "
Sep 15 21:11:04.097761 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.093825    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b73f391a-b69f-4597-bed0-19a4d03bba3e-maas-api-tls\") pod \"b73f391a-b69f-4597-bed0-19a4d03bba3e\" (UID: \"b73f391a-b69f-4597-bed0-19a4d03bba3e\") "
Sep 15 21:11:04.097761 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.093897    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b73f391a-b69f-4597-bed0-19a4d03bba3e-metrics-certs\") pod \"b73f391a-b69f-4597-bed0-19a4d03bba3e\" (UID: \"b73f391a-b69f-4597-bed0-19a4d03bba3e\") "
Sep 15 21:11:04.097761 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.094150    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b73f391a-b69f-4597-bed0-19a4d03bba3e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b73f391a-b69f-4597-bed0-19a4d03bba3e" (UID: "b73f391a-b69f-4597-bed0-19a4d03bba3e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:11:04.097761 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.094422    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73f391a-b69f-4597-bed0-19a4d03bba3e-openshift-service-ca\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:11:04.097761 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.096743    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b73f391a-b69f-4597-bed0-19a4d03bba3e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b73f391a-b69f-4597-bed0-19a4d03bba3e" (UID: "b73f391a-b69f-4597-bed0-19a4d03bba3e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:11:04.108997 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.108960    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b73f391a-b69f-4597-bed0-19a4d03bba3e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b73f391a-b69f-4597-bed0-19a4d03bba3e" (UID: "b73f391a-b69f-4597-bed0-19a4d03bba3e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:11:04.109341 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.109321    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73f391a-b69f-4597-bed0-19a4d03bba3e-kube-api-access-nv26j" (OuterVolumeSpecName: "kube-api-access-nv26j") pod "b73f391a-b69f-4597-bed0-19a4d03bba3e" (UID: "b73f391a-b69f-4597-bed0-19a4d03bba3e"). InnerVolumeSpecName "kube-api-access-nv26j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:11:04.195591 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.195552    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv26j\" (UniqueName: \"kubernetes.io/projected/b73f391a-b69f-4597-bed0-19a4d03bba3e-kube-api-access-nv26j\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:11:04.195591 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.195582    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b73f391a-b69f-4597-bed0-19a4d03bba3e-maas-api-tls\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:11:04.195591 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.195592    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b73f391a-b69f-4597-bed0-19a4d03bba3e-metrics-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:11:04.401996 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.401915    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:11:04.401996 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.401970    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:11:04.415820 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.415785    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:11:04.863319 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.863240    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"
Sep 15 21:11:04.863319 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.863273    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6" event={"ID":"b73f391a-b69f-4597-bed0-19a4d03bba3e","Type":"ContainerDied","Data":"31747ad03453a049ccac087786c21f84c64e4db2afd4592b57066f2b8fcbd46c"}
Sep 15 21:11:04.863849 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.863335    2558 scope.go:117] "RemoveContainer" containerID="f53bb27938ef2fddc743e952ef47320b09f0f0b34f5d58c28c05b12d80a3c39c"
Sep 15 21:11:04.865165 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.865139    2558 generic.go:356] "Generic (PLEG): container finished" podID="5de574df-a42f-4b09-ab0f-ba38a26d6968" containerID="868e6be7774c9b2972671d0c08024db5f6ec63ef2962b6c4392e48d1abac2cbb" exitCode=0
Sep 15 21:11:04.865277 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.865167    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst" event={"ID":"5de574df-a42f-4b09-ab0f-ba38a26d6968","Type":"ContainerDied","Data":"868e6be7774c9b2972671d0c08024db5f6ec63ef2962b6c4392e48d1abac2cbb"}
Sep 15 21:11:04.865718 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.865697    2558 scope.go:117] "RemoveContainer" containerID="868e6be7774c9b2972671d0c08024db5f6ec63ef2962b6c4392e48d1abac2cbb"
Sep 15 21:11:04.867029 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.867001    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:11:04.867153 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.867099    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:11:04.867153 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.867136    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:11:04.879840 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.879818    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:11:04.940146 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.940118    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"]
Sep 15 21:11:04.946382 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:04.946358    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bfe4d22a-cd5c95c89-mp7x6"]
Sep 15 21:11:05.133456 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.133376    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"]
Sep 15 21:11:05.133985 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.133965    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b73f391a-b69f-4597-bed0-19a4d03bba3e" containerName="maas-api"
Sep 15 21:11:05.134027 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.133989    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b73f391a-b69f-4597-bed0-19a4d03bba3e" containerName="maas-api"
Sep 15 21:11:05.134190 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.134172    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b73f391a-b69f-4597-bed0-19a4d03bba3e" containerName="maas-api"
Sep 15 21:11:05.137767 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.137745    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.140930 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.140909    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-6529f90b-gw-openshift-default-dockercfg-vqcv8\""
Sep 15 21:11:05.156413 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.156385    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"]
Sep 15 21:11:05.204939 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.204906    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-istio-data\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.205099 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.204959    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b42117-39a7-44cc-badb-33d7ca26ecfb-istio-token\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.205099 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.205084    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-istio-envoy\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.205193 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.205122    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b42117-39a7-44cc-badb-33d7ca26ecfb-istio-podinfo\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.205193 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.205186    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-workload-certs\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.205273 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.205232    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-credential-socket\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.205311 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.205269    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-workload-socket\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.205311 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.205295    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57czq\" (UniqueName: \"kubernetes.io/projected/33b42117-39a7-44cc-badb-33d7ca26ecfb-kube-api-access-57czq\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.205386 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.205327    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b42117-39a7-44cc-badb-33d7ca26ecfb-istiod-ca-cert\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.306385 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.306347    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-workload-certs\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.306578 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.306402    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-credential-socket\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.306578 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.306436    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-workload-socket\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.306578 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.306459    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57czq\" (UniqueName: \"kubernetes.io/projected/33b42117-39a7-44cc-badb-33d7ca26ecfb-kube-api-access-57czq\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.306578 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.306495    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b42117-39a7-44cc-badb-33d7ca26ecfb-istiod-ca-cert\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.306578 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.306542    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-istio-data\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.306857 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.306590    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b42117-39a7-44cc-badb-33d7ca26ecfb-istio-token\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.306857 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.306640    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-istio-envoy\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.306857 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.306665    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b42117-39a7-44cc-badb-33d7ca26ecfb-istio-podinfo\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.306857 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.306822    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-workload-certs\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.306857 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.306839    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-credential-socket\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.307135 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.306884    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-workload-socket\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.307135 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.307112    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-istio-data\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.307291 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.307270    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b42117-39a7-44cc-badb-33d7ca26ecfb-istiod-ca-cert\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.308919 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.308898    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-istio-envoy\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.309462 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.309443    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b42117-39a7-44cc-badb-33d7ca26ecfb-istio-podinfo\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.338998 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.338973    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57czq\" (UniqueName: \"kubernetes.io/projected/33b42117-39a7-44cc-badb-33d7ca26ecfb-kube-api-access-57czq\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.345775 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.345742    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b42117-39a7-44cc-badb-33d7ca26ecfb-istio-token\") pod \"e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") " pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.449321 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.449277    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:05.743083 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.743013    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"]
Sep 15 21:11:05.776318 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.775422    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:11:05.776318 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.775593    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:11:05.776318 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.775696    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:11:05.875453 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.875420    2558 generic.go:356] "Generic (PLEG): container finished" podID="4f826e85-24be-4260-be0f-ff7c30ae5f2e" containerID="3078e7b4937cefa97ce6caf61725654ebae1972097365c0411e6a8dffa05e3c1" exitCode=0
Sep 15 21:11:05.875842 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.875494    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89" event={"ID":"4f826e85-24be-4260-be0f-ff7c30ae5f2e","Type":"ContainerDied","Data":"3078e7b4937cefa97ce6caf61725654ebae1972097365c0411e6a8dffa05e3c1"}
Sep 15 21:11:05.875925 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.875873    2558 scope.go:117] "RemoveContainer" containerID="3078e7b4937cefa97ce6caf61725654ebae1972097365c0411e6a8dffa05e3c1"
Sep 15 21:11:05.877207 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.877172    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:11:05.877327 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.877266    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:11:05.877327 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.877313    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:11:05.877756 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.877732    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst" event={"ID":"5de574df-a42f-4b09-ab0f-ba38a26d6968","Type":"ContainerStarted","Data":"66ea917bd61bbfe9cfca81e86c0ab4a97eeb24abfafd0c5d9c1f95877a7ce3a4"}
Sep 15 21:11:05.879566 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.879546    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf" event={"ID":"33b42117-39a7-44cc-badb-33d7ca26ecfb","Type":"ContainerStarted","Data":"ff4fd4ff950208ceaec562ee7bd93ead796193969b00a323d4d7e9dbc76a74a5"}
Sep 15 21:11:05.960084 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.949692    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b73f391a-b69f-4597-bed0-19a4d03bba3e" path="/var/lib/kubelet/pods/b73f391a-b69f-4597-bed0-19a4d03bba3e/volumes"
Sep 15 21:11:05.961607 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.961571    2558 patch_prober.go:28] interesting pod/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 15 21:11:05.961748 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.961644    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst" podUID="5de574df-a42f-4b09-ab0f-ba38a26d6968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 15 21:11:05.961935 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.961857    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:11:05.998252 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:05.997846    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf" podStartSLOduration=0.997823655 podStartE2EDuration="997.823655ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:11:05.960313427 +0000 UTC m=+784.450959161" watchObservedRunningTime="2026-09-15 21:11:05.997823655 +0000 UTC m=+784.488469390"
Sep 15 21:11:06.450298 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:06.450257    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:06.452215 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:06.451874    2558 patch_prober.go:28] interesting pod/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 15 21:11:06.452215 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:06.451931    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf" podUID="33b42117-39a7-44cc-badb-33d7ca26ecfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 15 21:11:06.887688 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:06.887598    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89" event={"ID":"4f826e85-24be-4260-be0f-ff7c30ae5f2e","Type":"ContainerStarted","Data":"6b1bd45e4a9b4828a484a0b5dd973192fcf62b8b5592e4e7cddd33e451f6379a"}
Sep 15 21:11:06.889306 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:06.889278    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf" event={"ID":"33b42117-39a7-44cc-badb-33d7ca26ecfb","Type":"ContainerStarted","Data":"33e11825fb9dc2ab3aa36c8e388de7b186099a0610716b69b466c2c6f2767f52"}
Sep 15 21:11:06.955680 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:06.955642    2558 patch_prober.go:28] interesting pod/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 15 21:11:06.955848 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:06.955708    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst" podUID="5de574df-a42f-4b09-ab0f-ba38a26d6968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 15 21:11:07.457730 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:07.457690    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:07.557727 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:07.557685    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:11:07.557934 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:07.557826    2558 patch_prober.go:28] interesting pod/e2e-autores-05731f-openshift-default-675d57d64b-gpd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 15 21:11:07.557934 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:07.557871    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89" podUID="4f826e85-24be-4260-be0f-ff7c30ae5f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 15 21:11:07.896379 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:07.896297    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:07.897381 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:07.897362    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:07.956476 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:07.956445    2558 patch_prober.go:28] interesting pod/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 15 21:11:07.956727 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:07.956698    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst" podUID="5de574df-a42f-4b09-ab0f-ba38a26d6968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 15 21:11:08.195928 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:08.195862    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:11:08.556771 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:08.556673    2558 patch_prober.go:28] interesting pod/e2e-autores-05731f-openshift-default-675d57d64b-gpd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 15 21:11:08.556771 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:08.556741    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89" podUID="4f826e85-24be-4260-be0f-ff7c30ae5f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 15 21:11:08.955869 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:08.955826    2558 patch_prober.go:28] interesting pod/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 15 21:11:08.956383 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:08.955897    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst" podUID="5de574df-a42f-4b09-ab0f-ba38a26d6968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 15 21:11:09.556967 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:09.556929    2558 patch_prober.go:28] interesting pod/e2e-autores-05731f-openshift-default-675d57d64b-gpd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 15 21:11:09.557278 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:09.557246    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89" podUID="4f826e85-24be-4260-be0f-ff7c30ae5f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 15 21:11:09.686980 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:09.686146    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:11:09.687280 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:09.687260    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:11:09.955506 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:09.955467    2558 patch_prober.go:28] interesting pod/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 15 21:11:09.955754 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:09.955728    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst" podUID="5de574df-a42f-4b09-ab0f-ba38a26d6968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 15 21:11:10.556951 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:10.556902    2558 patch_prober.go:28] interesting pod/e2e-autores-05731f-openshift-default-675d57d64b-gpd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 15 21:11:10.557503 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:10.556983    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89" podUID="4f826e85-24be-4260-be0f-ff7c30ae5f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 15 21:11:10.854541 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:10.854447    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"]
Sep 15 21:11:10.877625 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:10.877598    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"
Sep 15 21:11:10.880302 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:10.880277    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"]
Sep 15 21:11:10.881720 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:10.881700    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-6529f90b-serving-cert\""
Sep 15 21:11:10.974879 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:10.974847    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41fd4860-e61f-45bc-bac3-485575c6a71f-maas-api-tls\") pod \"maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t\" (UID: \"41fd4860-e61f-45bc-bac3-485575c6a71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"
Sep 15 21:11:10.975039 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:10.974885    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41fd4860-e61f-45bc-bac3-485575c6a71f-metrics-certs\") pod \"maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t\" (UID: \"41fd4860-e61f-45bc-bac3-485575c6a71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"
Sep 15 21:11:10.975039 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:10.974915    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t26ll\" (UniqueName: \"kubernetes.io/projected/41fd4860-e61f-45bc-bac3-485575c6a71f-kube-api-access-t26ll\") pod \"maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t\" (UID: \"41fd4860-e61f-45bc-bac3-485575c6a71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"
Sep 15 21:11:10.975039 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:10.975024    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41fd4860-e61f-45bc-bac3-485575c6a71f-openshift-service-ca\") pod \"maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t\" (UID: \"41fd4860-e61f-45bc-bac3-485575c6a71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"
Sep 15 21:11:11.038129 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:11.038087    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:11:11.039726 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:11.039697    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:11:11.076212 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:11.076179    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41fd4860-e61f-45bc-bac3-485575c6a71f-maas-api-tls\") pod \"maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t\" (UID: \"41fd4860-e61f-45bc-bac3-485575c6a71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"
Sep 15 21:11:11.076212 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:11.076213    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41fd4860-e61f-45bc-bac3-485575c6a71f-metrics-certs\") pod \"maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t\" (UID: \"41fd4860-e61f-45bc-bac3-485575c6a71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"
Sep 15 21:11:11.076473 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:11.076232    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t26ll\" (UniqueName: \"kubernetes.io/projected/41fd4860-e61f-45bc-bac3-485575c6a71f-kube-api-access-t26ll\") pod \"maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t\" (UID: \"41fd4860-e61f-45bc-bac3-485575c6a71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"
Sep 15 21:11:11.076473 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:11.076248    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41fd4860-e61f-45bc-bac3-485575c6a71f-openshift-service-ca\") pod \"maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t\" (UID: \"41fd4860-e61f-45bc-bac3-485575c6a71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"
Sep 15 21:11:11.076473 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:11:11.076362    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-serving-cert: secret "maas-api-e2e-migrate-6529f90b-serving-cert" not found
Sep 15 21:11:11.076473 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:11:11.076444    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41fd4860-e61f-45bc-bac3-485575c6a71f-maas-api-tls podName:41fd4860-e61f-45bc-bac3-485575c6a71f nodeName:}" failed. No retries permitted until 2026-09-15 21:11:11.576424226 +0000 UTC m=+790.067069940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/41fd4860-e61f-45bc-bac3-485575c6a71f-maas-api-tls") pod "maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t" (UID: "41fd4860-e61f-45bc-bac3-485575c6a71f") : secret "maas-api-e2e-migrate-6529f90b-serving-cert" not found
Sep 15 21:11:11.076944 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:11.076924    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41fd4860-e61f-45bc-bac3-485575c6a71f-openshift-service-ca\") pod \"maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t\" (UID: \"41fd4860-e61f-45bc-bac3-485575c6a71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"
Sep 15 21:11:11.078707 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:11.078688    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41fd4860-e61f-45bc-bac3-485575c6a71f-metrics-certs\") pod \"maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t\" (UID: \"41fd4860-e61f-45bc-bac3-485575c6a71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"
Sep 15 21:11:11.084648 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:11.084631    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t26ll\" (UniqueName: \"kubernetes.io/projected/41fd4860-e61f-45bc-bac3-485575c6a71f-kube-api-access-t26ll\") pod \"maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t\" (UID: \"41fd4860-e61f-45bc-bac3-485575c6a71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"
Sep 15 21:11:11.556537 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:11.556492    2558 patch_prober.go:28] interesting pod/e2e-autores-05731f-openshift-default-675d57d64b-gpd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 15 21:11:11.556749 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:11.556627    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89" podUID="4f826e85-24be-4260-be0f-ff7c30ae5f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 15 21:11:11.580916 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:11.580884    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41fd4860-e61f-45bc-bac3-485575c6a71f-maas-api-tls\") pod \"maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t\" (UID: \"41fd4860-e61f-45bc-bac3-485575c6a71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"
Sep 15 21:11:11.583359 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:11.583335    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41fd4860-e61f-45bc-bac3-485575c6a71f-maas-api-tls\") pod \"maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t\" (UID: \"41fd4860-e61f-45bc-bac3-485575c6a71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"
Sep 15 21:11:11.791696 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:11.791657    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"
Sep 15 21:11:12.085885 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:12.085855    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"]
Sep 15 21:11:12.087509 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:11:12.087482    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd4860_e61f_45bc_bac3_485575c6a71f.slice/crio-7afb79433de27568e5f99623b7f7ca3dc1b7a7cf8f8480a3c6df786f5d586e8c WatchSource:0}: Error finding container 7afb79433de27568e5f99623b7f7ca3dc1b7a7cf8f8480a3c6df786f5d586e8c: Status 404 returned error can't find the container with id 7afb79433de27568e5f99623b7f7ca3dc1b7a7cf8f8480a3c6df786f5d586e8c
Sep 15 21:11:12.556478 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:12.556438    2558 patch_prober.go:28] interesting pod/e2e-autores-05731f-openshift-default-675d57d64b-gpd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 15 21:11:12.556782 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:12.556753    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89" podUID="4f826e85-24be-4260-be0f-ff7c30ae5f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 15 21:11:12.925614 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:12.925585    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t" event={"ID":"41fd4860-e61f-45bc-bac3-485575c6a71f","Type":"ContainerStarted","Data":"7afb79433de27568e5f99623b7f7ca3dc1b7a7cf8f8480a3c6df786f5d586e8c"}
Sep 15 21:11:13.560123 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:13.560097    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:11:13.560988 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:13.560972    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:11:13.932702 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:13.932668    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t" event={"ID":"41fd4860-e61f-45bc-bac3-485575c6a71f","Type":"ContainerStarted","Data":"1fbb0b63f1b82cb3b2ac6b5758dcb80a2416c60c824838ce4f23b7700652961d"}
Sep 15 21:11:13.933182 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:13.932733    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"
Sep 15 21:11:13.952154 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:13.952108    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t" podStartSLOduration=3.149239367 podStartE2EDuration="3.952091873s" totalImagesPullingTime="802.852506ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:11:13.95045093 +0000 UTC m=+792.441096662" watchObservedRunningTime="2026-09-15 21:11:13.952091873 +0000 UTC m=+792.442737596"
Sep 15 21:11:19.947329 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:19.947291    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"
Sep 15 21:11:27.893098 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:27.892991    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:11:28.018928 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:28.018897    2558 generic.go:356] "Generic (PLEG): container finished" podID="33b42117-39a7-44cc-badb-33d7ca26ecfb" containerID="33e11825fb9dc2ab3aa36c8e388de7b186099a0610716b69b466c2c6f2767f52" exitCode=0
Sep 15 21:11:28.019140 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:28.018974    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf" event={"ID":"33b42117-39a7-44cc-badb-33d7ca26ecfb","Type":"ContainerDied","Data":"33e11825fb9dc2ab3aa36c8e388de7b186099a0610716b69b466c2c6f2767f52"}
Sep 15 21:11:28.019416 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:28.019399    2558 scope.go:117] "RemoveContainer" containerID="33e11825fb9dc2ab3aa36c8e388de7b186099a0610716b69b466c2c6f2767f52"
Sep 15 21:11:28.020500 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:28.020476    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:11:28.020557 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:28.020538    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:11:28.020593 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:28.020566    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:11:29.027441 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:29.027397    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf" event={"ID":"33b42117-39a7-44cc-badb-33d7ca26ecfb","Type":"ContainerStarted","Data":"3db41fc4501150c653eee717d692c12650fa953e250ddc227dbcedcf42df1a43"}
Sep 15 21:11:29.450496 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:29.450435    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:29.450756 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:29.450726    2558 patch_prober.go:28] interesting pod/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 15 21:11:29.450842 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:29.450800    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf" podUID="33b42117-39a7-44cc-badb-33d7ca26ecfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 15 21:11:30.450252 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:30.450208    2558 patch_prober.go:28] interesting pod/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 15 21:11:30.450756 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:30.450278    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf" podUID="33b42117-39a7-44cc-badb-33d7ca26ecfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 15 21:11:31.450881 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:31.450827    2558 patch_prober.go:28] interesting pod/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 15 21:11:31.451401 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:31.450911    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf" podUID="33b42117-39a7-44cc-badb-33d7ca26ecfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 15 21:11:32.450320 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:32.450278    2558 patch_prober.go:28] interesting pod/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 15 21:11:32.450538 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:32.450361    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf" podUID="33b42117-39a7-44cc-badb-33d7ca26ecfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 15 21:11:33.450148 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:33.450099    2558 patch_prober.go:28] interesting pod/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 15 21:11:33.450636 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:33.450181    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf" podUID="33b42117-39a7-44cc-badb-33d7ca26ecfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 15 21:11:34.454358 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:34.454326    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:34.454756 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:34.454609    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:34.455308 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:34.455287    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:44.294405 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:44.294371    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:11:48.744136 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:48.744104    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"]
Sep 15 21:11:48.744540 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:48.744373    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t" podUID="41fd4860-e61f-45bc-bac3-485575c6a71f" containerName="maas-api" containerID="cri-o://1fbb0b63f1b82cb3b2ac6b5758dcb80a2416c60c824838ce4f23b7700652961d" gracePeriod=30
Sep 15 21:11:48.985198 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:48.985173    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"
Sep 15 21:11:49.031138 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.031046    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41fd4860-e61f-45bc-bac3-485575c6a71f-metrics-certs\") pod \"41fd4860-e61f-45bc-bac3-485575c6a71f\" (UID: \"41fd4860-e61f-45bc-bac3-485575c6a71f\") "
Sep 15 21:11:49.031271 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.031228    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41fd4860-e61f-45bc-bac3-485575c6a71f-openshift-service-ca\") pod \"41fd4860-e61f-45bc-bac3-485575c6a71f\" (UID: \"41fd4860-e61f-45bc-bac3-485575c6a71f\") "
Sep 15 21:11:49.031335 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.031277    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t26ll\" (UniqueName: \"kubernetes.io/projected/41fd4860-e61f-45bc-bac3-485575c6a71f-kube-api-access-t26ll\") pod \"41fd4860-e61f-45bc-bac3-485575c6a71f\" (UID: \"41fd4860-e61f-45bc-bac3-485575c6a71f\") "
Sep 15 21:11:49.031335 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.031294    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41fd4860-e61f-45bc-bac3-485575c6a71f-maas-api-tls\") pod \"41fd4860-e61f-45bc-bac3-485575c6a71f\" (UID: \"41fd4860-e61f-45bc-bac3-485575c6a71f\") "
Sep 15 21:11:49.031708 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.031680    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41fd4860-e61f-45bc-bac3-485575c6a71f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "41fd4860-e61f-45bc-bac3-485575c6a71f" (UID: "41fd4860-e61f-45bc-bac3-485575c6a71f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:11:49.033364 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.033337    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41fd4860-e61f-45bc-bac3-485575c6a71f-kube-api-access-t26ll" (OuterVolumeSpecName: "kube-api-access-t26ll") pod "41fd4860-e61f-45bc-bac3-485575c6a71f" (UID: "41fd4860-e61f-45bc-bac3-485575c6a71f"). InnerVolumeSpecName "kube-api-access-t26ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:11:49.033639 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.033607    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41fd4860-e61f-45bc-bac3-485575c6a71f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "41fd4860-e61f-45bc-bac3-485575c6a71f" (UID: "41fd4860-e61f-45bc-bac3-485575c6a71f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:11:49.033639 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.033614    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41fd4860-e61f-45bc-bac3-485575c6a71f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "41fd4860-e61f-45bc-bac3-485575c6a71f" (UID: "41fd4860-e61f-45bc-bac3-485575c6a71f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:11:49.094399 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.094368    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"]
Sep 15 21:11:49.094666 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.094619    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf" podUID="33b42117-39a7-44cc-badb-33d7ca26ecfb" containerName="istio-proxy" containerID="cri-o://3db41fc4501150c653eee717d692c12650fa953e250ddc227dbcedcf42df1a43" gracePeriod=30
Sep 15 21:11:49.132844 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.132818    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41fd4860-e61f-45bc-bac3-485575c6a71f-openshift-service-ca\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:11:49.132844 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.132842    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t26ll\" (UniqueName: \"kubernetes.io/projected/41fd4860-e61f-45bc-bac3-485575c6a71f-kube-api-access-t26ll\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:11:49.133011 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.132852    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41fd4860-e61f-45bc-bac3-485575c6a71f-maas-api-tls\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:11:49.133011 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.132862    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41fd4860-e61f-45bc-bac3-485575c6a71f-metrics-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:11:49.156931 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.156898    2558 generic.go:356] "Generic (PLEG): container finished" podID="41fd4860-e61f-45bc-bac3-485575c6a71f" containerID="1fbb0b63f1b82cb3b2ac6b5758dcb80a2416c60c824838ce4f23b7700652961d" exitCode=0
Sep 15 21:11:49.157123 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.156953    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"
Sep 15 21:11:49.157123 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.156979    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t" event={"ID":"41fd4860-e61f-45bc-bac3-485575c6a71f","Type":"ContainerDied","Data":"1fbb0b63f1b82cb3b2ac6b5758dcb80a2416c60c824838ce4f23b7700652961d"}
Sep 15 21:11:49.157123 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.157031    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t" event={"ID":"41fd4860-e61f-45bc-bac3-485575c6a71f","Type":"ContainerDied","Data":"7afb79433de27568e5f99623b7f7ca3dc1b7a7cf8f8480a3c6df786f5d586e8c"}
Sep 15 21:11:49.157123 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.157074    2558 scope.go:117] "RemoveContainer" containerID="1fbb0b63f1b82cb3b2ac6b5758dcb80a2416c60c824838ce4f23b7700652961d"
Sep 15 21:11:49.167044 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.167027    2558 scope.go:117] "RemoveContainer" containerID="1fbb0b63f1b82cb3b2ac6b5758dcb80a2416c60c824838ce4f23b7700652961d"
Sep 15 21:11:49.167332 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:11:49.167315    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fbb0b63f1b82cb3b2ac6b5758dcb80a2416c60c824838ce4f23b7700652961d\": container with ID starting with 1fbb0b63f1b82cb3b2ac6b5758dcb80a2416c60c824838ce4f23b7700652961d not found: ID does not exist" containerID="1fbb0b63f1b82cb3b2ac6b5758dcb80a2416c60c824838ce4f23b7700652961d"
Sep 15 21:11:49.167373 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.167341    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fbb0b63f1b82cb3b2ac6b5758dcb80a2416c60c824838ce4f23b7700652961d"} err="failed to get container status \"1fbb0b63f1b82cb3b2ac6b5758dcb80a2416c60c824838ce4f23b7700652961d\": rpc error: code = NotFound desc = could not find container \"1fbb0b63f1b82cb3b2ac6b5758dcb80a2416c60c824838ce4f23b7700652961d\": container with ID starting with 1fbb0b63f1b82cb3b2ac6b5758dcb80a2416c60c824838ce4f23b7700652961d not found: ID does not exist"
Sep 15 21:11:49.181400 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.181375    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"]
Sep 15 21:11:49.185065 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.185034    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6529f90b-5b64b5cb45-lqh7t"]
Sep 15 21:11:49.457347 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.457310    2558 patch_prober.go:28] interesting pod/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 21:11:49.457509 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.457394    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf" podUID="33b42117-39a7-44cc-badb-33d7ca26ecfb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 21:11:49.854499 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.854421    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"]
Sep 15 21:11:49.854921 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.854852    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41fd4860-e61f-45bc-bac3-485575c6a71f" containerName="maas-api"
Sep 15 21:11:49.854921 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.854870    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="41fd4860-e61f-45bc-bac3-485575c6a71f" containerName="maas-api"
Sep 15 21:11:49.855016 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.854968    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="41fd4860-e61f-45bc-bac3-485575c6a71f" containerName="maas-api"
Sep 15 21:11:49.859552 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.859532    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:49.862628 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.862607    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-817c786c-openshift-default-dockercfg-bwjwl\""
Sep 15 21:11:49.870963 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.870941    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"]
Sep 15 21:11:49.939962 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.939934    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-workload-socket\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:49.940109 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.939969    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cfd3f7-ee6b-478a-b913-85104242a65d-istiod-ca-cert\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:49.940109 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.939987    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-istio-envoy\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:49.940109 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.940016    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cfd3f7-ee6b-478a-b913-85104242a65d-istio-token\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:49.940246 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.940123    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cfd3f7-ee6b-478a-b913-85104242a65d-istio-podinfo\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:49.940246 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.940203    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-credential-socket\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:49.940246 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.940226    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-workload-certs\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:49.940398 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.940291    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp8rl\" (UniqueName: \"kubernetes.io/projected/b6cfd3f7-ee6b-478a-b913-85104242a65d-kube-api-access-qp8rl\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:49.940398 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.940367    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-istio-data\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:49.943114 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:49.943091    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41fd4860-e61f-45bc-bac3-485575c6a71f" path="/var/lib/kubelet/pods/41fd4860-e61f-45bc-bac3-485575c6a71f/volumes"
Sep 15 21:11:50.041574 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:50.041545    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-credential-socket\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:50.041574 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:50.041576    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-workload-certs\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:50.041829 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:50.041608    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp8rl\" (UniqueName: \"kubernetes.io/projected/b6cfd3f7-ee6b-478a-b913-85104242a65d-kube-api-access-qp8rl\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:50.041829 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:50.041763    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-istio-data\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:50.041829 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:50.041805    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-workload-socket\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:50.041997 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:50.041843    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cfd3f7-ee6b-478a-b913-85104242a65d-istiod-ca-cert\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:50.041997 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:50.041873    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-istio-envoy\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:50.041997 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:50.041922    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cfd3f7-ee6b-478a-b913-85104242a65d-istio-token\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:50.041997 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:50.041971    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cfd3f7-ee6b-478a-b913-85104242a65d-istio-podinfo\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:50.042232 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:50.042012    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-workload-certs\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:50.042284 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:50.042270    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-istio-data\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:50.042338 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:50.042295    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-credential-socket\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:50.042475 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:50.042443    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-workload-socket\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:50.042789 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:50.042765    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cfd3f7-ee6b-478a-b913-85104242a65d-istiod-ca-cert\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:50.044263 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:50.044237    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-istio-envoy\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:50.044474 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:50.044453    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cfd3f7-ee6b-478a-b913-85104242a65d-istio-podinfo\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:50.053175 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:50.053152    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cfd3f7-ee6b-478a-b913-85104242a65d-istio-token\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:50.053344 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:50.053324    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp8rl\" (UniqueName: \"kubernetes.io/projected/b6cfd3f7-ee6b-478a-b913-85104242a65d-kube-api-access-qp8rl\") pod \"e2e-ait-817c786c-openshift-default-c7596967d-v96bq\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") " pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:50.170245 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:50.170219    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:50.446451 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:50.446425    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"]
Sep 15 21:11:50.447448 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:11:50.447423    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6cfd3f7_ee6b_478a_b913_85104242a65d.slice/crio-c0035b124b26ed7c53fe9cb956b66af7615d66d13c94a94e871b902bdbe347c5 WatchSource:0}: Error finding container c0035b124b26ed7c53fe9cb956b66af7615d66d13c94a94e871b902bdbe347c5: Status 404 returned error can't find the container with id c0035b124b26ed7c53fe9cb956b66af7615d66d13c94a94e871b902bdbe347c5
Sep 15 21:11:50.450193 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:50.450164    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:11:50.450258 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:50.450232    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:11:50.450296 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:50.450262    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:11:51.170804 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:51.170756    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq" event={"ID":"b6cfd3f7-ee6b-478a-b913-85104242a65d","Type":"ContainerStarted","Data":"291fbe9fa35a7680e9b8f3cfe7f306998dd1315c4f5f3649917cbc4bcc15d134"}
Sep 15 21:11:51.171303 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:51.170842    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq" event={"ID":"b6cfd3f7-ee6b-478a-b913-85104242a65d","Type":"ContainerStarted","Data":"c0035b124b26ed7c53fe9cb956b66af7615d66d13c94a94e871b902bdbe347c5"}
Sep 15 21:11:51.193536 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:51.193489    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq" podStartSLOduration=2.193475464 podStartE2EDuration="2.193475464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:11:51.192192893 +0000 UTC m=+829.682838642" watchObservedRunningTime="2026-09-15 21:11:51.193475464 +0000 UTC m=+829.684121195"
Sep 15 21:11:52.171135 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:52.171098    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:52.177086 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:52.177062    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:53.181285 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:53.181248    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:53.182353 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:53.182333    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:11:54.343442 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.343419    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:54.486699 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.486670    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-istio-data\") pod \"33b42117-39a7-44cc-badb-33d7ca26ecfb\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") "
Sep 15 21:11:54.486861 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.486733    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57czq\" (UniqueName: \"kubernetes.io/projected/33b42117-39a7-44cc-badb-33d7ca26ecfb-kube-api-access-57czq\") pod \"33b42117-39a7-44cc-badb-33d7ca26ecfb\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") "
Sep 15 21:11:54.486861 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.486764    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b42117-39a7-44cc-badb-33d7ca26ecfb-istiod-ca-cert\") pod \"33b42117-39a7-44cc-badb-33d7ca26ecfb\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") "
Sep 15 21:11:54.486861 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.486807    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b42117-39a7-44cc-badb-33d7ca26ecfb-istio-token\") pod \"33b42117-39a7-44cc-badb-33d7ca26ecfb\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") "
Sep 15 21:11:54.486861 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.486843    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b42117-39a7-44cc-badb-33d7ca26ecfb-istio-podinfo\") pod \"33b42117-39a7-44cc-badb-33d7ca26ecfb\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") "
Sep 15 21:11:54.487081 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.486922    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-credential-socket\") pod \"33b42117-39a7-44cc-badb-33d7ca26ecfb\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") "
Sep 15 21:11:54.487081 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.486955    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-workload-socket\") pod \"33b42117-39a7-44cc-badb-33d7ca26ecfb\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") "
Sep 15 21:11:54.487081 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.486977    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-istio-envoy\") pod \"33b42117-39a7-44cc-badb-33d7ca26ecfb\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") "
Sep 15 21:11:54.487081 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.487027    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-workload-certs\") pod \"33b42117-39a7-44cc-badb-33d7ca26ecfb\" (UID: \"33b42117-39a7-44cc-badb-33d7ca26ecfb\") "
Sep 15 21:11:54.487360 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.486972    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-istio-data" (OuterVolumeSpecName: "istio-data") pod "33b42117-39a7-44cc-badb-33d7ca26ecfb" (UID: "33b42117-39a7-44cc-badb-33d7ca26ecfb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:11:54.487539 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.487217    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "33b42117-39a7-44cc-badb-33d7ca26ecfb" (UID: "33b42117-39a7-44cc-badb-33d7ca26ecfb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:11:54.487539 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.487270    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "33b42117-39a7-44cc-badb-33d7ca26ecfb" (UID: "33b42117-39a7-44cc-badb-33d7ca26ecfb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:11:54.487695 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.487544    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "33b42117-39a7-44cc-badb-33d7ca26ecfb" (UID: "33b42117-39a7-44cc-badb-33d7ca26ecfb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:11:54.487695 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.487608    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b42117-39a7-44cc-badb-33d7ca26ecfb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "33b42117-39a7-44cc-badb-33d7ca26ecfb" (UID: "33b42117-39a7-44cc-badb-33d7ca26ecfb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:11:54.489014 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.488983    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b42117-39a7-44cc-badb-33d7ca26ecfb-istio-token" (OuterVolumeSpecName: "istio-token") pod "33b42117-39a7-44cc-badb-33d7ca26ecfb" (UID: "33b42117-39a7-44cc-badb-33d7ca26ecfb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:11:54.489116 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.489009    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33b42117-39a7-44cc-badb-33d7ca26ecfb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "33b42117-39a7-44cc-badb-33d7ca26ecfb" (UID: "33b42117-39a7-44cc-badb-33d7ca26ecfb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:11:54.489261 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.489247    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b42117-39a7-44cc-badb-33d7ca26ecfb-kube-api-access-57czq" (OuterVolumeSpecName: "kube-api-access-57czq") pod "33b42117-39a7-44cc-badb-33d7ca26ecfb" (UID: "33b42117-39a7-44cc-badb-33d7ca26ecfb"). InnerVolumeSpecName "kube-api-access-57czq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:11:54.489305 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.489283    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "33b42117-39a7-44cc-badb-33d7ca26ecfb" (UID: "33b42117-39a7-44cc-badb-33d7ca26ecfb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:11:54.587839 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.587809    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57czq\" (UniqueName: \"kubernetes.io/projected/33b42117-39a7-44cc-badb-33d7ca26ecfb-kube-api-access-57czq\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:11:54.587839 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.587838    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b42117-39a7-44cc-badb-33d7ca26ecfb-istiod-ca-cert\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:11:54.588005 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.587850    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b42117-39a7-44cc-badb-33d7ca26ecfb-istio-token\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:11:54.588005 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.587864    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b42117-39a7-44cc-badb-33d7ca26ecfb-istio-podinfo\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:11:54.588005 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.587876    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-credential-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:11:54.588005 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.587892    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-workload-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:11:54.588005 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.587903    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-istio-envoy\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:11:54.588005 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.587914    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-workload-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:11:54.588005 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.587923    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b42117-39a7-44cc-badb-33d7ca26ecfb-istio-data\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:11:54.657862 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.657831    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"]
Sep 15 21:11:54.658267 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.658247    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33b42117-39a7-44cc-badb-33d7ca26ecfb" containerName="istio-proxy"
Sep 15 21:11:54.658267 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.658266    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="33b42117-39a7-44cc-badb-33d7ca26ecfb" containerName="istio-proxy"
Sep 15 21:11:54.658427 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.658275    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33b42117-39a7-44cc-badb-33d7ca26ecfb" containerName="istio-proxy"
Sep 15 21:11:54.658427 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.658281    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="33b42117-39a7-44cc-badb-33d7ca26ecfb" containerName="istio-proxy"
Sep 15 21:11:54.658427 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.658349    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="33b42117-39a7-44cc-badb-33d7ca26ecfb" containerName="istio-proxy"
Sep 15 21:11:54.658427 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.658358    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="33b42117-39a7-44cc-badb-33d7ca26ecfb" containerName="istio-proxy"
Sep 15 21:11:54.661838 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.661817    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"
Sep 15 21:11:54.664714 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.664689    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-817c786c-serving-cert\""
Sep 15 21:11:54.672750 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.672726    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"]
Sep 15 21:11:54.789599 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.789517    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkvqt\" (UniqueName: \"kubernetes.io/projected/95d90a70-8bce-4f72-b1d9-9971e8249870-kube-api-access-wkvqt\") pod \"maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw\" (UID: \"95d90a70-8bce-4f72-b1d9-9971e8249870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"
Sep 15 21:11:54.789599 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.789571    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d90a70-8bce-4f72-b1d9-9971e8249870-maas-api-tls\") pod \"maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw\" (UID: \"95d90a70-8bce-4f72-b1d9-9971e8249870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"
Sep 15 21:11:54.789887 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.789655    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d90a70-8bce-4f72-b1d9-9971e8249870-openshift-service-ca\") pod \"maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw\" (UID: \"95d90a70-8bce-4f72-b1d9-9971e8249870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"
Sep 15 21:11:54.789887 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.789695    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d90a70-8bce-4f72-b1d9-9971e8249870-metrics-certs\") pod \"maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw\" (UID: \"95d90a70-8bce-4f72-b1d9-9971e8249870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"
Sep 15 21:11:54.890429 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.890394    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkvqt\" (UniqueName: \"kubernetes.io/projected/95d90a70-8bce-4f72-b1d9-9971e8249870-kube-api-access-wkvqt\") pod \"maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw\" (UID: \"95d90a70-8bce-4f72-b1d9-9971e8249870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"
Sep 15 21:11:54.890602 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.890446    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d90a70-8bce-4f72-b1d9-9971e8249870-maas-api-tls\") pod \"maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw\" (UID: \"95d90a70-8bce-4f72-b1d9-9971e8249870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"
Sep 15 21:11:54.890602 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.890497    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d90a70-8bce-4f72-b1d9-9971e8249870-openshift-service-ca\") pod \"maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw\" (UID: \"95d90a70-8bce-4f72-b1d9-9971e8249870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"
Sep 15 21:11:54.890602 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.890525    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d90a70-8bce-4f72-b1d9-9971e8249870-metrics-certs\") pod \"maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw\" (UID: \"95d90a70-8bce-4f72-b1d9-9971e8249870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"
Sep 15 21:11:54.890602 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:11:54.890559    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-serving-cert: secret "maas-api-e2e-ait-817c786c-serving-cert" not found
Sep 15 21:11:54.890779 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:11:54.890629    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95d90a70-8bce-4f72-b1d9-9971e8249870-maas-api-tls podName:95d90a70-8bce-4f72-b1d9-9971e8249870 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:55.390608634 +0000 UTC m=+833.881254360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/95d90a70-8bce-4f72-b1d9-9971e8249870-maas-api-tls") pod "maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw" (UID: "95d90a70-8bce-4f72-b1d9-9971e8249870") : secret "maas-api-e2e-ait-817c786c-serving-cert" not found
Sep 15 21:11:54.891216 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.891196    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d90a70-8bce-4f72-b1d9-9971e8249870-openshift-service-ca\") pod \"maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw\" (UID: \"95d90a70-8bce-4f72-b1d9-9971e8249870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"
Sep 15 21:11:54.892760 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.892743    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d90a70-8bce-4f72-b1d9-9971e8249870-metrics-certs\") pod \"maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw\" (UID: \"95d90a70-8bce-4f72-b1d9-9971e8249870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"
Sep 15 21:11:54.902563 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:54.902544    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkvqt\" (UniqueName: \"kubernetes.io/projected/95d90a70-8bce-4f72-b1d9-9971e8249870-kube-api-access-wkvqt\") pod \"maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw\" (UID: \"95d90a70-8bce-4f72-b1d9-9971e8249870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"
Sep 15 21:11:55.194506 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:55.194463    2558 generic.go:356] "Generic (PLEG): container finished" podID="33b42117-39a7-44cc-badb-33d7ca26ecfb" containerID="3db41fc4501150c653eee717d692c12650fa953e250ddc227dbcedcf42df1a43" exitCode=0
Sep 15 21:11:55.194642 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:55.194535    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"
Sep 15 21:11:55.194642 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:55.194542    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf" event={"ID":"33b42117-39a7-44cc-badb-33d7ca26ecfb","Type":"ContainerDied","Data":"3db41fc4501150c653eee717d692c12650fa953e250ddc227dbcedcf42df1a43"}
Sep 15 21:11:55.194642 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:55.194598    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf" event={"ID":"33b42117-39a7-44cc-badb-33d7ca26ecfb","Type":"ContainerDied","Data":"ff4fd4ff950208ceaec562ee7bd93ead796193969b00a323d4d7e9dbc76a74a5"}
Sep 15 21:11:55.194642 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:55.194624    2558 scope.go:117] "RemoveContainer" containerID="3db41fc4501150c653eee717d692c12650fa953e250ddc227dbcedcf42df1a43"
Sep 15 21:11:55.203936 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:55.203917    2558 scope.go:117] "RemoveContainer" containerID="33e11825fb9dc2ab3aa36c8e388de7b186099a0610716b69b466c2c6f2767f52"
Sep 15 21:11:55.213771 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:55.213623    2558 scope.go:117] "RemoveContainer" containerID="3db41fc4501150c653eee717d692c12650fa953e250ddc227dbcedcf42df1a43"
Sep 15 21:11:55.213914 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:11:55.213891    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3db41fc4501150c653eee717d692c12650fa953e250ddc227dbcedcf42df1a43\": container with ID starting with 3db41fc4501150c653eee717d692c12650fa953e250ddc227dbcedcf42df1a43 not found: ID does not exist" containerID="3db41fc4501150c653eee717d692c12650fa953e250ddc227dbcedcf42df1a43"
Sep 15 21:11:55.213954 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:55.213922    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db41fc4501150c653eee717d692c12650fa953e250ddc227dbcedcf42df1a43"} err="failed to get container status \"3db41fc4501150c653eee717d692c12650fa953e250ddc227dbcedcf42df1a43\": rpc error: code = NotFound desc = could not find container \"3db41fc4501150c653eee717d692c12650fa953e250ddc227dbcedcf42df1a43\": container with ID starting with 3db41fc4501150c653eee717d692c12650fa953e250ddc227dbcedcf42df1a43 not found: ID does not exist"
Sep 15 21:11:55.213954 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:55.213939    2558 scope.go:117] "RemoveContainer" containerID="33e11825fb9dc2ab3aa36c8e388de7b186099a0610716b69b466c2c6f2767f52"
Sep 15 21:11:55.214272 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:11:55.214252    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33e11825fb9dc2ab3aa36c8e388de7b186099a0610716b69b466c2c6f2767f52\": container with ID starting with 33e11825fb9dc2ab3aa36c8e388de7b186099a0610716b69b466c2c6f2767f52 not found: ID does not exist" containerID="33e11825fb9dc2ab3aa36c8e388de7b186099a0610716b69b466c2c6f2767f52"
Sep 15 21:11:55.214327 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:55.214279    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33e11825fb9dc2ab3aa36c8e388de7b186099a0610716b69b466c2c6f2767f52"} err="failed to get container status \"33e11825fb9dc2ab3aa36c8e388de7b186099a0610716b69b466c2c6f2767f52\": rpc error: code = NotFound desc = could not find container \"33e11825fb9dc2ab3aa36c8e388de7b186099a0610716b69b466c2c6f2767f52\": container with ID starting with 33e11825fb9dc2ab3aa36c8e388de7b186099a0610716b69b466c2c6f2767f52 not found: ID does not exist"
Sep 15 21:11:55.225622 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:55.225598    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"]
Sep 15 21:11:55.227806 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:55.227787    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-6529f90b-gw-openshift-default-67fd644f7b-78dmf"]
Sep 15 21:11:55.395633 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:55.395603    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d90a70-8bce-4f72-b1d9-9971e8249870-maas-api-tls\") pod \"maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw\" (UID: \"95d90a70-8bce-4f72-b1d9-9971e8249870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"
Sep 15 21:11:55.397970 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:55.397947    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d90a70-8bce-4f72-b1d9-9971e8249870-maas-api-tls\") pod \"maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw\" (UID: \"95d90a70-8bce-4f72-b1d9-9971e8249870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"
Sep 15 21:11:55.574581 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:55.574465    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"
Sep 15 21:11:55.859417 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:55.859390    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"]
Sep 15 21:11:55.860675 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:11:55.860645    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d90a70_8bce_4f72_b1d9_9971e8249870.slice/crio-1c3366f950e04c128be02201a1e922fbe7aae7518c413f9ccc3bf87373140ac6 WatchSource:0}: Error finding container 1c3366f950e04c128be02201a1e922fbe7aae7518c413f9ccc3bf87373140ac6: Status 404 returned error can't find the container with id 1c3366f950e04c128be02201a1e922fbe7aae7518c413f9ccc3bf87373140ac6
Sep 15 21:11:55.942167 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:55.942138    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33b42117-39a7-44cc-badb-33d7ca26ecfb" path="/var/lib/kubelet/pods/33b42117-39a7-44cc-badb-33d7ca26ecfb/volumes"
Sep 15 21:11:56.202031 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:56.201992    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw" event={"ID":"95d90a70-8bce-4f72-b1d9-9971e8249870","Type":"ContainerStarted","Data":"1c3366f950e04c128be02201a1e922fbe7aae7518c413f9ccc3bf87373140ac6"}
Sep 15 21:11:57.209422 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:57.209385    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw" event={"ID":"95d90a70-8bce-4f72-b1d9-9971e8249870","Type":"ContainerStarted","Data":"49c73c1ec06a89bd74458996cb0fd7cb34729fa6ce44013a1809d392f92a5f48"}
Sep 15 21:11:57.209794 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:57.209447    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"
Sep 15 21:11:57.231611 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:11:57.231570    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw" podStartSLOduration=2.72344498 podStartE2EDuration="3.231554475s" totalImagesPullingTime="508.109495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:11:57.230204186 +0000 UTC m=+835.720849920" watchObservedRunningTime="2026-09-15 21:11:57.231554475 +0000 UTC m=+835.722200206"
Sep 15 21:12:03.219342 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:03.219313    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"
Sep 15 21:12:04.404729 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:04.404696    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:12:14.283962 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.283929    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"]
Sep 15 21:12:14.288800 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.288781    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:12:14.293384 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.293358    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0df719\"/\"kube-root-ca.crt\""
Sep 15 21:12:14.293521 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.293407    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0df719\"/\"openshift-service-ca.crt\""
Sep 15 21:12:14.293700 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.293677    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0df719\"/\"default-dockercfg-pw7ql\""
Sep 15 21:12:14.293811 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.293687    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0df719\"/\"test-model-0df719-kserve-self-signed-certs\""
Sep 15 21:12:14.296991 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.296768    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"]
Sep 15 21:12:14.374896 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.374866    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc9e1da3-e02a-4262-8bba-d9a4863737d4-tmp-dir\") pod \"test-model-0df719-kserve-7795895b8c-hnz82\" (UID: \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\") " pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:12:14.375079 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.374925    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc9e1da3-e02a-4262-8bba-d9a4863737d4-model-cache\") pod \"test-model-0df719-kserve-7795895b8c-hnz82\" (UID: \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\") " pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:12:14.375079 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.375031    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc9e1da3-e02a-4262-8bba-d9a4863737d4-dshm\") pod \"test-model-0df719-kserve-7795895b8c-hnz82\" (UID: \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\") " pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:12:14.375178 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.375121    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc9e1da3-e02a-4262-8bba-d9a4863737d4-home\") pod \"test-model-0df719-kserve-7795895b8c-hnz82\" (UID: \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\") " pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:12:14.375178 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.375145    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc9e1da3-e02a-4262-8bba-d9a4863737d4-tls-certs\") pod \"test-model-0df719-kserve-7795895b8c-hnz82\" (UID: \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\") " pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:12:14.375178 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.375165    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2t9v\" (UniqueName: \"kubernetes.io/projected/fc9e1da3-e02a-4262-8bba-d9a4863737d4-kube-api-access-p2t9v\") pod \"test-model-0df719-kserve-7795895b8c-hnz82\" (UID: \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\") " pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:12:14.476163 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.476128    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc9e1da3-e02a-4262-8bba-d9a4863737d4-model-cache\") pod \"test-model-0df719-kserve-7795895b8c-hnz82\" (UID: \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\") " pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:12:14.476336 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.476193    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc9e1da3-e02a-4262-8bba-d9a4863737d4-dshm\") pod \"test-model-0df719-kserve-7795895b8c-hnz82\" (UID: \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\") " pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:12:14.476336 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.476229    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc9e1da3-e02a-4262-8bba-d9a4863737d4-home\") pod \"test-model-0df719-kserve-7795895b8c-hnz82\" (UID: \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\") " pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:12:14.476336 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.476245    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc9e1da3-e02a-4262-8bba-d9a4863737d4-tls-certs\") pod \"test-model-0df719-kserve-7795895b8c-hnz82\" (UID: \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\") " pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:12:14.476336 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.476261    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2t9v\" (UniqueName: \"kubernetes.io/projected/fc9e1da3-e02a-4262-8bba-d9a4863737d4-kube-api-access-p2t9v\") pod \"test-model-0df719-kserve-7795895b8c-hnz82\" (UID: \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\") " pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:12:14.476336 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.476298    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc9e1da3-e02a-4262-8bba-d9a4863737d4-tmp-dir\") pod \"test-model-0df719-kserve-7795895b8c-hnz82\" (UID: \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\") " pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:12:14.476618 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.476574    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc9e1da3-e02a-4262-8bba-d9a4863737d4-model-cache\") pod \"test-model-0df719-kserve-7795895b8c-hnz82\" (UID: \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\") " pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:12:14.476726 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.476700    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc9e1da3-e02a-4262-8bba-d9a4863737d4-home\") pod \"test-model-0df719-kserve-7795895b8c-hnz82\" (UID: \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\") " pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:12:14.476788 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.476726    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc9e1da3-e02a-4262-8bba-d9a4863737d4-tmp-dir\") pod \"test-model-0df719-kserve-7795895b8c-hnz82\" (UID: \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\") " pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:12:14.478529 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.478505    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc9e1da3-e02a-4262-8bba-d9a4863737d4-dshm\") pod \"test-model-0df719-kserve-7795895b8c-hnz82\" (UID: \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\") " pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:12:14.478913 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.478889    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc9e1da3-e02a-4262-8bba-d9a4863737d4-tls-certs\") pod \"test-model-0df719-kserve-7795895b8c-hnz82\" (UID: \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\") " pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:12:14.484459 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.484433    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2t9v\" (UniqueName: \"kubernetes.io/projected/fc9e1da3-e02a-4262-8bba-d9a4863737d4-kube-api-access-p2t9v\") pod \"test-model-0df719-kserve-7795895b8c-hnz82\" (UID: \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\") " pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:12:14.600205 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.600130    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:12:14.874067 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:14.874029    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"]
Sep 15 21:12:14.879919 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:12:14.879895    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc9e1da3_e02a_4262_8bba_d9a4863737d4.slice/crio-ab3996684ccf54300389afe41754a88ec8e32abeb076267f459060f2e406ea0c WatchSource:0}: Error finding container ab3996684ccf54300389afe41754a88ec8e32abeb076267f459060f2e406ea0c: Status 404 returned error can't find the container with id ab3996684ccf54300389afe41754a88ec8e32abeb076267f459060f2e406ea0c
Sep 15 21:12:15.312222 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:15.312189    2558 generic.go:356] "Generic (PLEG): container finished" podID="b6cfd3f7-ee6b-478a-b913-85104242a65d" containerID="291fbe9fa35a7680e9b8f3cfe7f306998dd1315c4f5f3649917cbc4bcc15d134" exitCode=0
Sep 15 21:12:15.312699 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:15.312269    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq" event={"ID":"b6cfd3f7-ee6b-478a-b913-85104242a65d","Type":"ContainerDied","Data":"291fbe9fa35a7680e9b8f3cfe7f306998dd1315c4f5f3649917cbc4bcc15d134"}
Sep 15 21:12:15.312790 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:15.312772    2558 scope.go:117] "RemoveContainer" containerID="291fbe9fa35a7680e9b8f3cfe7f306998dd1315c4f5f3649917cbc4bcc15d134"
Sep 15 21:12:15.313935 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:15.313897    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:12:15.313935 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:15.313920    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82" event={"ID":"fc9e1da3-e02a-4262-8bba-d9a4863737d4","Type":"ContainerStarted","Data":"4a43473f568925bdad43e09ca111ef75bc945fdc85a861d55f1f8e1de930137b"}
Sep 15 21:12:15.314115 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:15.313946    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82" event={"ID":"fc9e1da3-e02a-4262-8bba-d9a4863737d4","Type":"ContainerStarted","Data":"ab3996684ccf54300389afe41754a88ec8e32abeb076267f459060f2e406ea0c"}
Sep 15 21:12:15.314115 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:15.313976    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:12:15.314115 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:15.314027    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:12:15.356175 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:15.356132    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82" podStartSLOduration=1.356118126 podStartE2EDuration="1.356118126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:15.354303302 +0000 UTC m=+853.844949034" watchObservedRunningTime="2026-09-15 21:12:15.356118126 +0000 UTC m=+853.846763857"
Sep 15 21:12:16.321211 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:16.321178    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq" event={"ID":"b6cfd3f7-ee6b-478a-b913-85104242a65d","Type":"ContainerStarted","Data":"1844d95da515ebd31c17edcd6b4bf7ac4756432da1f046dba2eae1f36ad5186f"}
Sep 15 21:12:17.171131 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:17.171096    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:12:17.171307 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:17.171286    2558 patch_prober.go:28] interesting pod/e2e-ait-817c786c-openshift-default-c7596967d-v96bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 15 21:12:17.171351 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:17.171323    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq" podUID="b6cfd3f7-ee6b-478a-b913-85104242a65d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 15 21:12:18.170949 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:18.170910    2558 patch_prober.go:28] interesting pod/e2e-ait-817c786c-openshift-default-c7596967d-v96bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 15 21:12:18.171337 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:18.170971    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq" podUID="b6cfd3f7-ee6b-478a-b913-85104242a65d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 15 21:12:19.171499 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:19.171448    2558 patch_prober.go:28] interesting pod/e2e-ait-817c786c-openshift-default-c7596967d-v96bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 15 21:12:19.171893 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:19.171526    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq" podUID="b6cfd3f7-ee6b-478a-b913-85104242a65d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 15 21:12:19.284596 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:19.284561    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:12:20.171018 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:20.170983    2558 patch_prober.go:28] interesting pod/e2e-ait-817c786c-openshift-default-c7596967d-v96bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 15 21:12:20.171254 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:20.171038    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq" podUID="b6cfd3f7-ee6b-478a-b913-85104242a65d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 15 21:12:21.171181 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:21.171140    2558 patch_prober.go:28] interesting pod/e2e-ait-817c786c-openshift-default-c7596967d-v96bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 15 21:12:21.171596 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:21.171219    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq" podUID="b6cfd3f7-ee6b-478a-b913-85104242a65d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 15 21:12:22.174945 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:22.174921    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:12:22.175340 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:22.175283    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:12:22.175885 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:22.175865    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:12:24.334400 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:24.334368    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"]
Sep 15 21:12:24.334899 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:24.334623    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw" podUID="95d90a70-8bce-4f72-b1d9-9971e8249870" containerName="maas-api" containerID="cri-o://49c73c1ec06a89bd74458996cb0fd7cb34729fa6ce44013a1809d392f92a5f48" gracePeriod=30
Sep 15 21:12:24.588707 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:24.588642    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"
Sep 15 21:12:24.600946 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:24.600923    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:12:24.601106 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:24.600972    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:12:24.616271 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:24.616251    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:12:24.676429 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:24.676401    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkvqt\" (UniqueName: \"kubernetes.io/projected/95d90a70-8bce-4f72-b1d9-9971e8249870-kube-api-access-wkvqt\") pod \"95d90a70-8bce-4f72-b1d9-9971e8249870\" (UID: \"95d90a70-8bce-4f72-b1d9-9971e8249870\") "
Sep 15 21:12:24.676561 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:24.676434    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d90a70-8bce-4f72-b1d9-9971e8249870-openshift-service-ca\") pod \"95d90a70-8bce-4f72-b1d9-9971e8249870\" (UID: \"95d90a70-8bce-4f72-b1d9-9971e8249870\") "
Sep 15 21:12:24.676561 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:24.676519    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d90a70-8bce-4f72-b1d9-9971e8249870-metrics-certs\") pod \"95d90a70-8bce-4f72-b1d9-9971e8249870\" (UID: \"95d90a70-8bce-4f72-b1d9-9971e8249870\") "
Sep 15 21:12:24.676646 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:24.676601    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d90a70-8bce-4f72-b1d9-9971e8249870-maas-api-tls\") pod \"95d90a70-8bce-4f72-b1d9-9971e8249870\" (UID: \"95d90a70-8bce-4f72-b1d9-9971e8249870\") "
Sep 15 21:12:24.676842 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:24.676816    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95d90a70-8bce-4f72-b1d9-9971e8249870-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "95d90a70-8bce-4f72-b1d9-9971e8249870" (UID: "95d90a70-8bce-4f72-b1d9-9971e8249870"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:12:24.676989 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:24.676974    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d90a70-8bce-4f72-b1d9-9971e8249870-openshift-service-ca\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:12:24.678638 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:24.678607    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95d90a70-8bce-4f72-b1d9-9971e8249870-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "95d90a70-8bce-4f72-b1d9-9971e8249870" (UID: "95d90a70-8bce-4f72-b1d9-9971e8249870"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:12:24.678725 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:24.678634    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d90a70-8bce-4f72-b1d9-9971e8249870-kube-api-access-wkvqt" (OuterVolumeSpecName: "kube-api-access-wkvqt") pod "95d90a70-8bce-4f72-b1d9-9971e8249870" (UID: "95d90a70-8bce-4f72-b1d9-9971e8249870"). InnerVolumeSpecName "kube-api-access-wkvqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:12:24.678725 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:24.678667    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95d90a70-8bce-4f72-b1d9-9971e8249870-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "95d90a70-8bce-4f72-b1d9-9971e8249870" (UID: "95d90a70-8bce-4f72-b1d9-9971e8249870"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:12:24.778475 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:24.778443    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d90a70-8bce-4f72-b1d9-9971e8249870-maas-api-tls\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:12:24.778475 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:24.778469    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkvqt\" (UniqueName: \"kubernetes.io/projected/95d90a70-8bce-4f72-b1d9-9971e8249870-kube-api-access-wkvqt\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:12:24.778475 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:24.778485    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d90a70-8bce-4f72-b1d9-9971e8249870-metrics-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:12:25.376308 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:25.376271    2558 generic.go:356] "Generic (PLEG): container finished" podID="95d90a70-8bce-4f72-b1d9-9971e8249870" containerID="49c73c1ec06a89bd74458996cb0fd7cb34729fa6ce44013a1809d392f92a5f48" exitCode=0
Sep 15 21:12:25.376768 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:25.376316    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"
Sep 15 21:12:25.376768 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:25.376359    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw" event={"ID":"95d90a70-8bce-4f72-b1d9-9971e8249870","Type":"ContainerDied","Data":"49c73c1ec06a89bd74458996cb0fd7cb34729fa6ce44013a1809d392f92a5f48"}
Sep 15 21:12:25.376768 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:25.376404    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw" event={"ID":"95d90a70-8bce-4f72-b1d9-9971e8249870","Type":"ContainerDied","Data":"1c3366f950e04c128be02201a1e922fbe7aae7518c413f9ccc3bf87373140ac6"}
Sep 15 21:12:25.376768 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:25.376426    2558 scope.go:117] "RemoveContainer" containerID="49c73c1ec06a89bd74458996cb0fd7cb34729fa6ce44013a1809d392f92a5f48"
Sep 15 21:12:25.387297 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:25.387282    2558 scope.go:117] "RemoveContainer" containerID="49c73c1ec06a89bd74458996cb0fd7cb34729fa6ce44013a1809d392f92a5f48"
Sep 15 21:12:25.387574 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:12:25.387555    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49c73c1ec06a89bd74458996cb0fd7cb34729fa6ce44013a1809d392f92a5f48\": container with ID starting with 49c73c1ec06a89bd74458996cb0fd7cb34729fa6ce44013a1809d392f92a5f48 not found: ID does not exist" containerID="49c73c1ec06a89bd74458996cb0fd7cb34729fa6ce44013a1809d392f92a5f48"
Sep 15 21:12:25.387617 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:25.387582    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49c73c1ec06a89bd74458996cb0fd7cb34729fa6ce44013a1809d392f92a5f48"} err="failed to get container status \"49c73c1ec06a89bd74458996cb0fd7cb34729fa6ce44013a1809d392f92a5f48\": rpc error: code = NotFound desc = could not find container \"49c73c1ec06a89bd74458996cb0fd7cb34729fa6ce44013a1809d392f92a5f48\": container with ID starting with 49c73c1ec06a89bd74458996cb0fd7cb34729fa6ce44013a1809d392f92a5f48 not found: ID does not exist"
Sep 15 21:12:25.390045 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:25.390026    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:12:25.398659 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:25.398638    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"]
Sep 15 21:12:25.402761 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:25.402730    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-817c786c-65bbfddbd4-4tzvw"]
Sep 15 21:12:25.713613 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:25.713583    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"]
Sep 15 21:12:25.713790 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:25.713765    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq" podUID="b6cfd3f7-ee6b-478a-b913-85104242a65d" containerName="istio-proxy" containerID="cri-o://1844d95da515ebd31c17edcd6b4bf7ac4756432da1f046dba2eae1f36ad5186f" gracePeriod=30
Sep 15 21:12:25.946722 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:25.946683    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95d90a70-8bce-4f72-b1d9-9971e8249870" path="/var/lib/kubelet/pods/95d90a70-8bce-4f72-b1d9-9971e8249870/volumes"
Sep 15 21:12:26.454324 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.454290    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"]
Sep 15 21:12:26.454776 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.454676    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95d90a70-8bce-4f72-b1d9-9971e8249870" containerName="maas-api"
Sep 15 21:12:26.454776 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.454689    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="95d90a70-8bce-4f72-b1d9-9971e8249870" containerName="maas-api"
Sep 15 21:12:26.454776 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.454750    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="95d90a70-8bce-4f72-b1d9-9971e8249870" containerName="maas-api"
Sep 15 21:12:26.458949 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.458931    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.461675 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.461655    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-4194cf76-openshift-default-dockercfg-cfdz5\""
Sep 15 21:12:26.471204 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.471184    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"]
Sep 15 21:12:26.493417 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.493389    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szgrb\" (UniqueName: \"kubernetes.io/projected/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-kube-api-access-szgrb\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.493511 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.493431    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istio-token\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.493511 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.493461    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istiod-ca-cert\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.493602 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.493542    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-workload-certs\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.493602 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.493578    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istio-podinfo\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.493672 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.493602    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istio-envoy\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.493672 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.493659    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istio-data\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.493769 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.493718    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-credential-socket\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.493769 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.493743    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-workload-socket\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.594462 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.594432    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istio-data\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.594647 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.594504    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-credential-socket\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.594647 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.594539    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-workload-socket\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.594647 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.594573    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szgrb\" (UniqueName: \"kubernetes.io/projected/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-kube-api-access-szgrb\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.594647 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.594611    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istio-token\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.594892 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.594653    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istiod-ca-cert\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.594892 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.594709    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-workload-certs\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.594892 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.594750    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istio-podinfo\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.594892 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.594786    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istio-envoy\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.594892 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.594866    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istio-data\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.595181 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.594903    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-workload-socket\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.595181 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.594922    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-credential-socket\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.595181 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.595148    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-workload-certs\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.595510 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.595486    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istiod-ca-cert\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.597013 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.596985    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istio-envoy\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.597278 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.597258    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istio-podinfo\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.609064 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.609023    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szgrb\" (UniqueName: \"kubernetes.io/projected/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-kube-api-access-szgrb\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.609146 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.609026    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istio-token\") pod \"e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") " pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:26.768964 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:26.768884    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:27.046826 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:27.046795    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"]
Sep 15 21:12:27.047369 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:12:27.047341    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a669e06_19ec_4c9c_ab0d_a9e887cbf97f.slice/crio-57f00cb942199b68112a9eaea1f06d8df7d1aa451600505b6c675eee935959ea WatchSource:0}: Error finding container 57f00cb942199b68112a9eaea1f06d8df7d1aa451600505b6c675eee935959ea: Status 404 returned error can't find the container with id 57f00cb942199b68112a9eaea1f06d8df7d1aa451600505b6c675eee935959ea
Sep 15 21:12:27.049758 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:27.049720    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:12:27.049878 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:27.049788    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:12:27.049878 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:27.049816    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:12:27.389179 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:27.389096    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg" event={"ID":"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f","Type":"ContainerStarted","Data":"6b0311e735fb3e9b61916a8c3bf56190eff27d70feeec34efe36aa020148f26a"}
Sep 15 21:12:27.389179 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:27.389152    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg" event={"ID":"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f","Type":"ContainerStarted","Data":"57f00cb942199b68112a9eaea1f06d8df7d1aa451600505b6c675eee935959ea"}
Sep 15 21:12:27.415995 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:27.415946    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg" podStartSLOduration=1.415916982 podStartE2EDuration="1.415916982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:27.414412181 +0000 UTC m=+865.905057923" watchObservedRunningTime="2026-09-15 21:12:27.415916982 +0000 UTC m=+865.906562713"
Sep 15 21:12:27.769651 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:27.769609    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:28.774555 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:28.774528    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:29.400561 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:29.400526    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:29.401430 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:29.401412    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:30.304955 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:30.304920    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"]
Sep 15 21:12:30.308693 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:30.308672    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"
Sep 15 21:12:30.312254 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:30.312232    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-4194cf76-serving-cert\""
Sep 15 21:12:30.321964 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:30.321943    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"]
Sep 15 21:12:30.428146 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:30.428109    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5172f59a-2de8-48da-aa0d-91ebb1a2f949-maas-api-tls\") pod \"maas-api-e2e-derive-4194cf76-59dd55476-ggzrg\" (UID: \"5172f59a-2de8-48da-aa0d-91ebb1a2f949\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"
Sep 15 21:12:30.428323 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:30.428214    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkl6d\" (UniqueName: \"kubernetes.io/projected/5172f59a-2de8-48da-aa0d-91ebb1a2f949-kube-api-access-zkl6d\") pod \"maas-api-e2e-derive-4194cf76-59dd55476-ggzrg\" (UID: \"5172f59a-2de8-48da-aa0d-91ebb1a2f949\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"
Sep 15 21:12:30.428323 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:30.428277    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5172f59a-2de8-48da-aa0d-91ebb1a2f949-openshift-service-ca\") pod \"maas-api-e2e-derive-4194cf76-59dd55476-ggzrg\" (UID: \"5172f59a-2de8-48da-aa0d-91ebb1a2f949\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"
Sep 15 21:12:30.428429 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:30.428363    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5172f59a-2de8-48da-aa0d-91ebb1a2f949-metrics-certs\") pod \"maas-api-e2e-derive-4194cf76-59dd55476-ggzrg\" (UID: \"5172f59a-2de8-48da-aa0d-91ebb1a2f949\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"
Sep 15 21:12:30.529168 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:30.529133    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5172f59a-2de8-48da-aa0d-91ebb1a2f949-maas-api-tls\") pod \"maas-api-e2e-derive-4194cf76-59dd55476-ggzrg\" (UID: \"5172f59a-2de8-48da-aa0d-91ebb1a2f949\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"
Sep 15 21:12:30.529346 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:30.529210    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkl6d\" (UniqueName: \"kubernetes.io/projected/5172f59a-2de8-48da-aa0d-91ebb1a2f949-kube-api-access-zkl6d\") pod \"maas-api-e2e-derive-4194cf76-59dd55476-ggzrg\" (UID: \"5172f59a-2de8-48da-aa0d-91ebb1a2f949\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"
Sep 15 21:12:30.529346 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:12:30.529274    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-serving-cert: secret "maas-api-e2e-derive-4194cf76-serving-cert" not found
Sep 15 21:12:30.529346 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:30.529288    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5172f59a-2de8-48da-aa0d-91ebb1a2f949-openshift-service-ca\") pod \"maas-api-e2e-derive-4194cf76-59dd55476-ggzrg\" (UID: \"5172f59a-2de8-48da-aa0d-91ebb1a2f949\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"
Sep 15 21:12:30.529510 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:12:30.529372    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5172f59a-2de8-48da-aa0d-91ebb1a2f949-maas-api-tls podName:5172f59a-2de8-48da-aa0d-91ebb1a2f949 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:31.029350342 +0000 UTC m=+869.519996068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5172f59a-2de8-48da-aa0d-91ebb1a2f949-maas-api-tls") pod "maas-api-e2e-derive-4194cf76-59dd55476-ggzrg" (UID: "5172f59a-2de8-48da-aa0d-91ebb1a2f949") : secret "maas-api-e2e-derive-4194cf76-serving-cert" not found
Sep 15 21:12:30.529510 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:30.529409    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5172f59a-2de8-48da-aa0d-91ebb1a2f949-metrics-certs\") pod \"maas-api-e2e-derive-4194cf76-59dd55476-ggzrg\" (UID: \"5172f59a-2de8-48da-aa0d-91ebb1a2f949\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"
Sep 15 21:12:30.529989 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:30.529966    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5172f59a-2de8-48da-aa0d-91ebb1a2f949-openshift-service-ca\") pod \"maas-api-e2e-derive-4194cf76-59dd55476-ggzrg\" (UID: \"5172f59a-2de8-48da-aa0d-91ebb1a2f949\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"
Sep 15 21:12:30.531804 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:30.531783    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5172f59a-2de8-48da-aa0d-91ebb1a2f949-metrics-certs\") pod \"maas-api-e2e-derive-4194cf76-59dd55476-ggzrg\" (UID: \"5172f59a-2de8-48da-aa0d-91ebb1a2f949\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"
Sep 15 21:12:30.543604 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:30.543586    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkl6d\" (UniqueName: \"kubernetes.io/projected/5172f59a-2de8-48da-aa0d-91ebb1a2f949-kube-api-access-zkl6d\") pod \"maas-api-e2e-derive-4194cf76-59dd55476-ggzrg\" (UID: \"5172f59a-2de8-48da-aa0d-91ebb1a2f949\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"
Sep 15 21:12:30.958904 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:30.958881    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:12:31.032904 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.032826    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-istio-data\") pod \"b6cfd3f7-ee6b-478a-b913-85104242a65d\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") "
Sep 15 21:12:31.032904 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.032876    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-credential-socket\") pod \"b6cfd3f7-ee6b-478a-b913-85104242a65d\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") "
Sep 15 21:12:31.032904 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.032893    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-workload-socket\") pod \"b6cfd3f7-ee6b-478a-b913-85104242a65d\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") "
Sep 15 21:12:31.033184 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.032935    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cfd3f7-ee6b-478a-b913-85104242a65d-istio-podinfo\") pod \"b6cfd3f7-ee6b-478a-b913-85104242a65d\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") "
Sep 15 21:12:31.033184 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.032958    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-workload-certs\") pod \"b6cfd3f7-ee6b-478a-b913-85104242a65d\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") "
Sep 15 21:12:31.033184 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.032985    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qp8rl\" (UniqueName: \"kubernetes.io/projected/b6cfd3f7-ee6b-478a-b913-85104242a65d-kube-api-access-qp8rl\") pod \"b6cfd3f7-ee6b-478a-b913-85104242a65d\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") "
Sep 15 21:12:31.033184 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.033034    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-istio-envoy\") pod \"b6cfd3f7-ee6b-478a-b913-85104242a65d\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") "
Sep 15 21:12:31.033184 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.033121    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cfd3f7-ee6b-478a-b913-85104242a65d-istiod-ca-cert\") pod \"b6cfd3f7-ee6b-478a-b913-85104242a65d\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") "
Sep 15 21:12:31.033184 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.033119    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-istio-data" (OuterVolumeSpecName: "istio-data") pod "b6cfd3f7-ee6b-478a-b913-85104242a65d" (UID: "b6cfd3f7-ee6b-478a-b913-85104242a65d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:12:31.033184 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.033151    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cfd3f7-ee6b-478a-b913-85104242a65d-istio-token\") pod \"b6cfd3f7-ee6b-478a-b913-85104242a65d\" (UID: \"b6cfd3f7-ee6b-478a-b913-85104242a65d\") "
Sep 15 21:12:31.033184 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.033162    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b6cfd3f7-ee6b-478a-b913-85104242a65d" (UID: "b6cfd3f7-ee6b-478a-b913-85104242a65d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:12:31.033580 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.033277    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b6cfd3f7-ee6b-478a-b913-85104242a65d" (UID: "b6cfd3f7-ee6b-478a-b913-85104242a65d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:12:31.033580 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.033288    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b6cfd3f7-ee6b-478a-b913-85104242a65d" (UID: "b6cfd3f7-ee6b-478a-b913-85104242a65d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:12:31.033580 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.033521    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6cfd3f7-ee6b-478a-b913-85104242a65d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b6cfd3f7-ee6b-478a-b913-85104242a65d" (UID: "b6cfd3f7-ee6b-478a-b913-85104242a65d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:12:31.033580 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.033543    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5172f59a-2de8-48da-aa0d-91ebb1a2f949-maas-api-tls\") pod \"maas-api-e2e-derive-4194cf76-59dd55476-ggzrg\" (UID: \"5172f59a-2de8-48da-aa0d-91ebb1a2f949\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"
Sep 15 21:12:31.033774 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.033731    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-workload-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:12:31.033774 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.033750    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cfd3f7-ee6b-478a-b913-85104242a65d-istiod-ca-cert\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:12:31.033774 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.033764    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-istio-data\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:12:31.033932 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.033777    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-credential-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:12:31.033932 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.033791    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-workload-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:12:31.035414 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.035391    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6cfd3f7-ee6b-478a-b913-85104242a65d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b6cfd3f7-ee6b-478a-b913-85104242a65d" (UID: "b6cfd3f7-ee6b-478a-b913-85104242a65d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:12:31.035414 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.035402    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6cfd3f7-ee6b-478a-b913-85104242a65d-kube-api-access-qp8rl" (OuterVolumeSpecName: "kube-api-access-qp8rl") pod "b6cfd3f7-ee6b-478a-b913-85104242a65d" (UID: "b6cfd3f7-ee6b-478a-b913-85104242a65d"). InnerVolumeSpecName "kube-api-access-qp8rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:12:31.035589 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.035560    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6cfd3f7-ee6b-478a-b913-85104242a65d-istio-token" (OuterVolumeSpecName: "istio-token") pod "b6cfd3f7-ee6b-478a-b913-85104242a65d" (UID: "b6cfd3f7-ee6b-478a-b913-85104242a65d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:12:31.035589 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.035575    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b6cfd3f7-ee6b-478a-b913-85104242a65d" (UID: "b6cfd3f7-ee6b-478a-b913-85104242a65d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:12:31.035924 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.035906    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5172f59a-2de8-48da-aa0d-91ebb1a2f949-maas-api-tls\") pod \"maas-api-e2e-derive-4194cf76-59dd55476-ggzrg\" (UID: \"5172f59a-2de8-48da-aa0d-91ebb1a2f949\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"
Sep 15 21:12:31.134269 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.134240    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cfd3f7-ee6b-478a-b913-85104242a65d-istio-token\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:12:31.134269 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.134266    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cfd3f7-ee6b-478a-b913-85104242a65d-istio-podinfo\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:12:31.134269 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.134275    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp8rl\" (UniqueName: \"kubernetes.io/projected/b6cfd3f7-ee6b-478a-b913-85104242a65d-kube-api-access-qp8rl\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:12:31.134493 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.134285    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cfd3f7-ee6b-478a-b913-85104242a65d-istio-envoy\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:12:31.218936 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.218897    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"
Sep 15 21:12:31.413617 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.413529    2558 generic.go:356] "Generic (PLEG): container finished" podID="b6cfd3f7-ee6b-478a-b913-85104242a65d" containerID="1844d95da515ebd31c17edcd6b4bf7ac4756432da1f046dba2eae1f36ad5186f" exitCode=0
Sep 15 21:12:31.413617 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.413585    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"
Sep 15 21:12:31.413617 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.413605    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq" event={"ID":"b6cfd3f7-ee6b-478a-b913-85104242a65d","Type":"ContainerDied","Data":"1844d95da515ebd31c17edcd6b4bf7ac4756432da1f046dba2eae1f36ad5186f"}
Sep 15 21:12:31.414205 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.413659    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq" event={"ID":"b6cfd3f7-ee6b-478a-b913-85104242a65d","Type":"ContainerDied","Data":"c0035b124b26ed7c53fe9cb956b66af7615d66d13c94a94e871b902bdbe347c5"}
Sep 15 21:12:31.414205 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.413682    2558 scope.go:117] "RemoveContainer" containerID="1844d95da515ebd31c17edcd6b4bf7ac4756432da1f046dba2eae1f36ad5186f"
Sep 15 21:12:31.423269 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.423254    2558 scope.go:117] "RemoveContainer" containerID="291fbe9fa35a7680e9b8f3cfe7f306998dd1315c4f5f3649917cbc4bcc15d134"
Sep 15 21:12:31.431676 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.431658    2558 scope.go:117] "RemoveContainer" containerID="1844d95da515ebd31c17edcd6b4bf7ac4756432da1f046dba2eae1f36ad5186f"
Sep 15 21:12:31.431941 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:12:31.431920    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1844d95da515ebd31c17edcd6b4bf7ac4756432da1f046dba2eae1f36ad5186f\": container with ID starting with 1844d95da515ebd31c17edcd6b4bf7ac4756432da1f046dba2eae1f36ad5186f not found: ID does not exist" containerID="1844d95da515ebd31c17edcd6b4bf7ac4756432da1f046dba2eae1f36ad5186f"
Sep 15 21:12:31.432009 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.431953    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1844d95da515ebd31c17edcd6b4bf7ac4756432da1f046dba2eae1f36ad5186f"} err="failed to get container status \"1844d95da515ebd31c17edcd6b4bf7ac4756432da1f046dba2eae1f36ad5186f\": rpc error: code = NotFound desc = could not find container \"1844d95da515ebd31c17edcd6b4bf7ac4756432da1f046dba2eae1f36ad5186f\": container with ID starting with 1844d95da515ebd31c17edcd6b4bf7ac4756432da1f046dba2eae1f36ad5186f not found: ID does not exist"
Sep 15 21:12:31.432009 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.431975    2558 scope.go:117] "RemoveContainer" containerID="291fbe9fa35a7680e9b8f3cfe7f306998dd1315c4f5f3649917cbc4bcc15d134"
Sep 15 21:12:31.432266 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:12:31.432241    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"291fbe9fa35a7680e9b8f3cfe7f306998dd1315c4f5f3649917cbc4bcc15d134\": container with ID starting with 291fbe9fa35a7680e9b8f3cfe7f306998dd1315c4f5f3649917cbc4bcc15d134 not found: ID does not exist" containerID="291fbe9fa35a7680e9b8f3cfe7f306998dd1315c4f5f3649917cbc4bcc15d134"
Sep 15 21:12:31.432343 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.432272    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"291fbe9fa35a7680e9b8f3cfe7f306998dd1315c4f5f3649917cbc4bcc15d134"} err="failed to get container status \"291fbe9fa35a7680e9b8f3cfe7f306998dd1315c4f5f3649917cbc4bcc15d134\": rpc error: code = NotFound desc = could not find container \"291fbe9fa35a7680e9b8f3cfe7f306998dd1315c4f5f3649917cbc4bcc15d134\": container with ID starting with 291fbe9fa35a7680e9b8f3cfe7f306998dd1315c4f5f3649917cbc4bcc15d134 not found: ID does not exist"
Sep 15 21:12:31.439175 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.439155    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"]
Sep 15 21:12:31.445390 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.445370    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-817c786c-openshift-default-c7596967d-v96bq"]
Sep 15 21:12:31.499145 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.499122    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"]
Sep 15 21:12:31.499292 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:12:31.499273    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5172f59a_2de8_48da_aa0d_91ebb1a2f949.slice/crio-569728c962ee39a458e436af3bd9a47f1acf51f306fd58855d796f2766fb63aa WatchSource:0}: Error finding container 569728c962ee39a458e436af3bd9a47f1acf51f306fd58855d796f2766fb63aa: Status 404 returned error can't find the container with id 569728c962ee39a458e436af3bd9a47f1acf51f306fd58855d796f2766fb63aa
Sep 15 21:12:31.944154 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:31.944124    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6cfd3f7-ee6b-478a-b913-85104242a65d" path="/var/lib/kubelet/pods/b6cfd3f7-ee6b-478a-b913-85104242a65d/volumes"
Sep 15 21:12:32.420466 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:32.420428    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg" event={"ID":"5172f59a-2de8-48da-aa0d-91ebb1a2f949","Type":"ContainerStarted","Data":"6c9fea08ed6bd12435ab4b691f2419f697d424d3edce4a9585581fcf9756b200"}
Sep 15 21:12:32.420920 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:32.420494    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg" event={"ID":"5172f59a-2de8-48da-aa0d-91ebb1a2f949","Type":"ContainerStarted","Data":"569728c962ee39a458e436af3bd9a47f1acf51f306fd58855d796f2766fb63aa"}
Sep 15 21:12:32.420920 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:32.420603    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"
Sep 15 21:12:32.443582 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:32.443536    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg" podStartSLOduration=2.016255564 podStartE2EDuration="2.443520067s" totalImagesPullingTime="427.264503ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:32.44246154 +0000 UTC m=+870.933107271" watchObservedRunningTime="2026-09-15 21:12:32.443520067 +0000 UTC m=+870.934165798"
Sep 15 21:12:38.432950 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:38.432920    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"
Sep 15 21:12:45.533531 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:45.533488    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:12:56.575529 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:56.575493    2558 generic.go:356] "Generic (PLEG): container finished" podID="7a669e06-19ec-4c9c-ab0d-a9e887cbf97f" containerID="6b0311e735fb3e9b61916a8c3bf56190eff27d70feeec34efe36aa020148f26a" exitCode=0
Sep 15 21:12:56.575958 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:56.575562    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg" event={"ID":"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f","Type":"ContainerDied","Data":"6b0311e735fb3e9b61916a8c3bf56190eff27d70feeec34efe36aa020148f26a"}
Sep 15 21:12:56.575958 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:56.575940    2558 scope.go:117] "RemoveContainer" containerID="6b0311e735fb3e9b61916a8c3bf56190eff27d70feeec34efe36aa020148f26a"
Sep 15 21:12:56.577028 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:56.577001    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:12:56.577119 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:56.577085    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:12:56.577158 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:56.577122    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:12:57.582732 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:57.582693    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg" event={"ID":"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f","Type":"ContainerStarted","Data":"a0bf099d2ad1f95a81ad5e6ff2ddaa019053e8afd1f1dfd63fa1ade249b8e7d2"}
Sep 15 21:12:57.769220 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:57.769181    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:57.770521 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:57.770489    2558 patch_prober.go:28] interesting pod/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 15 21:12:57.770684 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:57.770542    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg" podUID="7a669e06-19ec-4c9c-ab0d-a9e887cbf97f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 15 21:12:58.770003 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:58.769962    2558 patch_prober.go:28] interesting pod/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 15 21:12:58.770466 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:58.770031    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg" podUID="7a669e06-19ec-4c9c-ab0d-a9e887cbf97f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 15 21:12:59.401419 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:59.401383    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:12:59.769600 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:59.769567    2558 patch_prober.go:28] interesting pod/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 15 21:12:59.769745 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:12:59.769627    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg" podUID="7a669e06-19ec-4c9c-ab0d-a9e887cbf97f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 15 21:13:00.766074 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:00.758857    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"]
Sep 15 21:13:00.766074 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:00.759164    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg" podUID="5172f59a-2de8-48da-aa0d-91ebb1a2f949" containerName="maas-api" containerID="cri-o://6c9fea08ed6bd12435ab4b691f2419f697d424d3edce4a9585581fcf9756b200" gracePeriod=30
Sep 15 21:13:00.773773 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:00.773389    2558 patch_prober.go:28] interesting pod/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 15 21:13:00.773773 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:00.773446    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg" podUID="7a669e06-19ec-4c9c-ab0d-a9e887cbf97f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 15 21:13:01.022336 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.022267    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"
Sep 15 21:13:01.122037 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.122009    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5172f59a-2de8-48da-aa0d-91ebb1a2f949-openshift-service-ca\") pod \"5172f59a-2de8-48da-aa0d-91ebb1a2f949\" (UID: \"5172f59a-2de8-48da-aa0d-91ebb1a2f949\") "
Sep 15 21:13:01.122362 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.122340    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkl6d\" (UniqueName: \"kubernetes.io/projected/5172f59a-2de8-48da-aa0d-91ebb1a2f949-kube-api-access-zkl6d\") pod \"5172f59a-2de8-48da-aa0d-91ebb1a2f949\" (UID: \"5172f59a-2de8-48da-aa0d-91ebb1a2f949\") "
Sep 15 21:13:01.122419 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.122387    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5172f59a-2de8-48da-aa0d-91ebb1a2f949-maas-api-tls\") pod \"5172f59a-2de8-48da-aa0d-91ebb1a2f949\" (UID: \"5172f59a-2de8-48da-aa0d-91ebb1a2f949\") "
Sep 15 21:13:01.122465 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.122439    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5172f59a-2de8-48da-aa0d-91ebb1a2f949-metrics-certs\") pod \"5172f59a-2de8-48da-aa0d-91ebb1a2f949\" (UID: \"5172f59a-2de8-48da-aa0d-91ebb1a2f949\") "
Sep 15 21:13:01.123001 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.122969    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5172f59a-2de8-48da-aa0d-91ebb1a2f949-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5172f59a-2de8-48da-aa0d-91ebb1a2f949" (UID: "5172f59a-2de8-48da-aa0d-91ebb1a2f949"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:13:01.126123 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.126095    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5172f59a-2de8-48da-aa0d-91ebb1a2f949-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5172f59a-2de8-48da-aa0d-91ebb1a2f949" (UID: "5172f59a-2de8-48da-aa0d-91ebb1a2f949"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:13:01.127840 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.127811    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5172f59a-2de8-48da-aa0d-91ebb1a2f949-kube-api-access-zkl6d" (OuterVolumeSpecName: "kube-api-access-zkl6d") pod "5172f59a-2de8-48da-aa0d-91ebb1a2f949" (UID: "5172f59a-2de8-48da-aa0d-91ebb1a2f949"). InnerVolumeSpecName "kube-api-access-zkl6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:13:01.127942 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.127892    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5172f59a-2de8-48da-aa0d-91ebb1a2f949-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5172f59a-2de8-48da-aa0d-91ebb1a2f949" (UID: "5172f59a-2de8-48da-aa0d-91ebb1a2f949"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:13:01.137406 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.137376    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"]
Sep 15 21:13:01.137633 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.137600    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg" podUID="7a669e06-19ec-4c9c-ab0d-a9e887cbf97f" containerName="istio-proxy" containerID="cri-o://a0bf099d2ad1f95a81ad5e6ff2ddaa019053e8afd1f1dfd63fa1ade249b8e7d2" gracePeriod=30
Sep 15 21:13:01.224024 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.223995    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5172f59a-2de8-48da-aa0d-91ebb1a2f949-openshift-service-ca\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:13:01.224024 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.224021    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkl6d\" (UniqueName: \"kubernetes.io/projected/5172f59a-2de8-48da-aa0d-91ebb1a2f949-kube-api-access-zkl6d\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:13:01.224024 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.224031    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5172f59a-2de8-48da-aa0d-91ebb1a2f949-maas-api-tls\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:13:01.224273 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.224041    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5172f59a-2de8-48da-aa0d-91ebb1a2f949-metrics-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:13:01.607001 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.606967    2558 generic.go:356] "Generic (PLEG): container finished" podID="5172f59a-2de8-48da-aa0d-91ebb1a2f949" containerID="6c9fea08ed6bd12435ab4b691f2419f697d424d3edce4a9585581fcf9756b200" exitCode=0
Sep 15 21:13:01.607001 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.607004    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"
Sep 15 21:13:01.607230 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.607043    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg" event={"ID":"5172f59a-2de8-48da-aa0d-91ebb1a2f949","Type":"ContainerDied","Data":"6c9fea08ed6bd12435ab4b691f2419f697d424d3edce4a9585581fcf9756b200"}
Sep 15 21:13:01.607230 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.607147    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg" event={"ID":"5172f59a-2de8-48da-aa0d-91ebb1a2f949","Type":"ContainerDied","Data":"569728c962ee39a458e436af3bd9a47f1acf51f306fd58855d796f2766fb63aa"}
Sep 15 21:13:01.607230 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.607184    2558 scope.go:117] "RemoveContainer" containerID="6c9fea08ed6bd12435ab4b691f2419f697d424d3edce4a9585581fcf9756b200"
Sep 15 21:13:01.616662 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.616645    2558 scope.go:117] "RemoveContainer" containerID="6c9fea08ed6bd12435ab4b691f2419f697d424d3edce4a9585581fcf9756b200"
Sep 15 21:13:01.616949 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:13:01.616932    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c9fea08ed6bd12435ab4b691f2419f697d424d3edce4a9585581fcf9756b200\": container with ID starting with 6c9fea08ed6bd12435ab4b691f2419f697d424d3edce4a9585581fcf9756b200 not found: ID does not exist" containerID="6c9fea08ed6bd12435ab4b691f2419f697d424d3edce4a9585581fcf9756b200"
Sep 15 21:13:01.616998 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.616958    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9fea08ed6bd12435ab4b691f2419f697d424d3edce4a9585581fcf9756b200"} err="failed to get container status \"6c9fea08ed6bd12435ab4b691f2419f697d424d3edce4a9585581fcf9756b200\": rpc error: code = NotFound desc = could not find container \"6c9fea08ed6bd12435ab4b691f2419f697d424d3edce4a9585581fcf9756b200\": container with ID starting with 6c9fea08ed6bd12435ab4b691f2419f697d424d3edce4a9585581fcf9756b200 not found: ID does not exist"
Sep 15 21:13:01.634042 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.634021    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"]
Sep 15 21:13:01.641222 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.641203    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4194cf76-59dd55476-ggzrg"]
Sep 15 21:13:01.968732 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:01.968620    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5172f59a-2de8-48da-aa0d-91ebb1a2f949" path="/var/lib/kubelet/pods/5172f59a-2de8-48da-aa0d-91ebb1a2f949/volumes"
Sep 15 21:13:07.785433 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.785409    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:13:07.880855 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.880788    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szgrb\" (UniqueName: \"kubernetes.io/projected/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-kube-api-access-szgrb\") pod \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") "
Sep 15 21:13:07.880855 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.880825    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-workload-socket\") pod \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") "
Sep 15 21:13:07.880855 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.880849    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istio-data\") pod \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") "
Sep 15 21:13:07.881454 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.880868    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-credential-socket\") pod \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") "
Sep 15 21:13:07.881454 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.881016    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istio-token\") pod \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") "
Sep 15 21:13:07.881454 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.881078    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istio-podinfo\") pod \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") "
Sep 15 21:13:07.881454 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.881132    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istiod-ca-cert\") pod \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") "
Sep 15 21:13:07.881454 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.881146    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7a669e06-19ec-4c9c-ab0d-a9e887cbf97f" (UID: "7a669e06-19ec-4c9c-ab0d-a9e887cbf97f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:13:07.881454 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.881128    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istio-data" (OuterVolumeSpecName: "istio-data") pod "7a669e06-19ec-4c9c-ab0d-a9e887cbf97f" (UID: "7a669e06-19ec-4c9c-ab0d-a9e887cbf97f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:13:07.881454 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.881167    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-workload-certs\") pod \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") "
Sep 15 21:13:07.881454 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.881181    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7a669e06-19ec-4c9c-ab0d-a9e887cbf97f" (UID: "7a669e06-19ec-4c9c-ab0d-a9e887cbf97f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:13:07.881454 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.881213    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istio-envoy\") pod \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\" (UID: \"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f\") "
Sep 15 21:13:07.881454 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.881424    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7a669e06-19ec-4c9c-ab0d-a9e887cbf97f" (UID: "7a669e06-19ec-4c9c-ab0d-a9e887cbf97f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:13:07.881870 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.881526    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7a669e06-19ec-4c9c-ab0d-a9e887cbf97f" (UID: "7a669e06-19ec-4c9c-ab0d-a9e887cbf97f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:13:07.881870 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.881581    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-workload-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:13:07.881870 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.881602    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istio-data\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:13:07.881870 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.881617    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-credential-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:13:07.881870 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.881630    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-workload-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:13:07.883253 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.883224    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7a669e06-19ec-4c9c-ab0d-a9e887cbf97f" (UID: "7a669e06-19ec-4c9c-ab0d-a9e887cbf97f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:13:07.883352 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.883253    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7a669e06-19ec-4c9c-ab0d-a9e887cbf97f" (UID: "7a669e06-19ec-4c9c-ab0d-a9e887cbf97f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:13:07.883519 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.883496    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istio-token" (OuterVolumeSpecName: "istio-token") pod "7a669e06-19ec-4c9c-ab0d-a9e887cbf97f" (UID: "7a669e06-19ec-4c9c-ab0d-a9e887cbf97f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:13:07.883519 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.883508    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-kube-api-access-szgrb" (OuterVolumeSpecName: "kube-api-access-szgrb") pod "7a669e06-19ec-4c9c-ab0d-a9e887cbf97f" (UID: "7a669e06-19ec-4c9c-ab0d-a9e887cbf97f"). InnerVolumeSpecName "kube-api-access-szgrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:13:07.982556 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.982527    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istio-token\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:13:07.982655 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.982557    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istio-podinfo\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:13:07.982655 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.982571    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istiod-ca-cert\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:13:07.982655 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.982584    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-istio-envoy\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:13:07.982655 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:07.982595    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szgrb\" (UniqueName: \"kubernetes.io/projected/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f-kube-api-access-szgrb\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:13:08.653739 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:08.653703    2558 generic.go:356] "Generic (PLEG): container finished" podID="7a669e06-19ec-4c9c-ab0d-a9e887cbf97f" containerID="a0bf099d2ad1f95a81ad5e6ff2ddaa019053e8afd1f1dfd63fa1ade249b8e7d2" exitCode=0
Sep 15 21:13:08.653905 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:08.653758    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"
Sep 15 21:13:08.653905 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:08.653786    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg" event={"ID":"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f","Type":"ContainerDied","Data":"a0bf099d2ad1f95a81ad5e6ff2ddaa019053e8afd1f1dfd63fa1ade249b8e7d2"}
Sep 15 21:13:08.653905 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:08.653840    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg" event={"ID":"7a669e06-19ec-4c9c-ab0d-a9e887cbf97f","Type":"ContainerDied","Data":"57f00cb942199b68112a9eaea1f06d8df7d1aa451600505b6c675eee935959ea"}
Sep 15 21:13:08.653905 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:08.653861    2558 scope.go:117] "RemoveContainer" containerID="a0bf099d2ad1f95a81ad5e6ff2ddaa019053e8afd1f1dfd63fa1ade249b8e7d2"
Sep 15 21:13:08.662695 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:08.662676    2558 scope.go:117] "RemoveContainer" containerID="6b0311e735fb3e9b61916a8c3bf56190eff27d70feeec34efe36aa020148f26a"
Sep 15 21:13:08.670802 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:08.670784    2558 scope.go:117] "RemoveContainer" containerID="a0bf099d2ad1f95a81ad5e6ff2ddaa019053e8afd1f1dfd63fa1ade249b8e7d2"
Sep 15 21:13:08.671081 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:13:08.671045    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0bf099d2ad1f95a81ad5e6ff2ddaa019053e8afd1f1dfd63fa1ade249b8e7d2\": container with ID starting with a0bf099d2ad1f95a81ad5e6ff2ddaa019053e8afd1f1dfd63fa1ade249b8e7d2 not found: ID does not exist" containerID="a0bf099d2ad1f95a81ad5e6ff2ddaa019053e8afd1f1dfd63fa1ade249b8e7d2"
Sep 15 21:13:08.671169 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:08.671094    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0bf099d2ad1f95a81ad5e6ff2ddaa019053e8afd1f1dfd63fa1ade249b8e7d2"} err="failed to get container status \"a0bf099d2ad1f95a81ad5e6ff2ddaa019053e8afd1f1dfd63fa1ade249b8e7d2\": rpc error: code = NotFound desc = could not find container \"a0bf099d2ad1f95a81ad5e6ff2ddaa019053e8afd1f1dfd63fa1ade249b8e7d2\": container with ID starting with a0bf099d2ad1f95a81ad5e6ff2ddaa019053e8afd1f1dfd63fa1ade249b8e7d2 not found: ID does not exist"
Sep 15 21:13:08.671169 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:08.671116    2558 scope.go:117] "RemoveContainer" containerID="6b0311e735fb3e9b61916a8c3bf56190eff27d70feeec34efe36aa020148f26a"
Sep 15 21:13:08.671399 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:13:08.671379    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b0311e735fb3e9b61916a8c3bf56190eff27d70feeec34efe36aa020148f26a\": container with ID starting with 6b0311e735fb3e9b61916a8c3bf56190eff27d70feeec34efe36aa020148f26a not found: ID does not exist" containerID="6b0311e735fb3e9b61916a8c3bf56190eff27d70feeec34efe36aa020148f26a"
Sep 15 21:13:08.671483 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:08.671404    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b0311e735fb3e9b61916a8c3bf56190eff27d70feeec34efe36aa020148f26a"} err="failed to get container status \"6b0311e735fb3e9b61916a8c3bf56190eff27d70feeec34efe36aa020148f26a\": rpc error: code = NotFound desc = could not find container \"6b0311e735fb3e9b61916a8c3bf56190eff27d70feeec34efe36aa020148f26a\": container with ID starting with 6b0311e735fb3e9b61916a8c3bf56190eff27d70feeec34efe36aa020148f26a not found: ID does not exist"
Sep 15 21:13:08.675160 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:08.675139    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"]
Sep 15 21:13:08.678881 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:08.678861    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-4194cf76-openshift-default-6c75b9cb56-2pjjg"]
Sep 15 21:13:09.500318 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:09.500287    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:13:09.941829 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:09.941798    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a669e06-19ec-4c9c-ab0d-a9e887cbf97f" path="/var/lib/kubelet/pods/7a669e06-19ec-4c9c-ab0d-a9e887cbf97f/volumes"
Sep 15 21:13:27.397473 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.397439    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"]
Sep 15 21:13:27.397930 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.397767    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a669e06-19ec-4c9c-ab0d-a9e887cbf97f" containerName="istio-proxy"
Sep 15 21:13:27.397930 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.397777    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a669e06-19ec-4c9c-ab0d-a9e887cbf97f" containerName="istio-proxy"
Sep 15 21:13:27.397930 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.397787    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5172f59a-2de8-48da-aa0d-91ebb1a2f949" containerName="maas-api"
Sep 15 21:13:27.397930 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.397793    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5172f59a-2de8-48da-aa0d-91ebb1a2f949" containerName="maas-api"
Sep 15 21:13:27.397930 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.397804    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6cfd3f7-ee6b-478a-b913-85104242a65d" containerName="istio-proxy"
Sep 15 21:13:27.397930 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.397810    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6cfd3f7-ee6b-478a-b913-85104242a65d" containerName="istio-proxy"
Sep 15 21:13:27.397930 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.397822    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6cfd3f7-ee6b-478a-b913-85104242a65d" containerName="istio-proxy"
Sep 15 21:13:27.397930 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.397827    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6cfd3f7-ee6b-478a-b913-85104242a65d" containerName="istio-proxy"
Sep 15 21:13:27.397930 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.397879    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6cfd3f7-ee6b-478a-b913-85104242a65d" containerName="istio-proxy"
Sep 15 21:13:27.397930 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.397887    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a669e06-19ec-4c9c-ab0d-a9e887cbf97f" containerName="istio-proxy"
Sep 15 21:13:27.397930 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.397894    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6cfd3f7-ee6b-478a-b913-85104242a65d" containerName="istio-proxy"
Sep 15 21:13:27.397930 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.397899    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5172f59a-2de8-48da-aa0d-91ebb1a2f949" containerName="maas-api"
Sep 15 21:13:27.402229 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.402213    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:13:27.405477 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.405416    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-05731f\"/\"openshift-service-ca.crt\""
Sep 15 21:13:27.405616 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.405595    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-05731f\"/\"kube-root-ca.crt\""
Sep 15 21:13:27.405668 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.405631    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-05731f\"/\"default-dockercfg-jtb2c\""
Sep 15 21:13:27.407128 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.407109    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-05731f\"/\"autores-model-05731f-kserve-self-signed-certs\""
Sep 15 21:13:27.411719 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.411699    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"]
Sep 15 21:13:27.455847 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.455819    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-home\") pod \"autores-model-05731f-kserve-8dff98d79-l2cb6\" (UID: \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\") " pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:13:27.455982 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.455855    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-tmp-dir\") pod \"autores-model-05731f-kserve-8dff98d79-l2cb6\" (UID: \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\") " pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:13:27.455982 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.455884    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-dshm\") pod \"autores-model-05731f-kserve-8dff98d79-l2cb6\" (UID: \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\") " pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:13:27.455982 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.455932    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-model-cache\") pod \"autores-model-05731f-kserve-8dff98d79-l2cb6\" (UID: \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\") " pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:13:27.455982 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.455981    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-tls-certs\") pod \"autores-model-05731f-kserve-8dff98d79-l2cb6\" (UID: \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\") " pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:13:27.456180 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.456036    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvk9w\" (UniqueName: \"kubernetes.io/projected/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-kube-api-access-xvk9w\") pod \"autores-model-05731f-kserve-8dff98d79-l2cb6\" (UID: \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\") " pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:13:27.556725 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.556695    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-tls-certs\") pod \"autores-model-05731f-kserve-8dff98d79-l2cb6\" (UID: \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\") " pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:13:27.556873 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.556756    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvk9w\" (UniqueName: \"kubernetes.io/projected/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-kube-api-access-xvk9w\") pod \"autores-model-05731f-kserve-8dff98d79-l2cb6\" (UID: \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\") " pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:13:27.556873 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.556795    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-home\") pod \"autores-model-05731f-kserve-8dff98d79-l2cb6\" (UID: \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\") " pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:13:27.556873 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.556813    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-tmp-dir\") pod \"autores-model-05731f-kserve-8dff98d79-l2cb6\" (UID: \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\") " pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:13:27.556873 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.556842    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-dshm\") pod \"autores-model-05731f-kserve-8dff98d79-l2cb6\" (UID: \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\") " pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:13:27.556873 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.556870    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-model-cache\") pod \"autores-model-05731f-kserve-8dff98d79-l2cb6\" (UID: \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\") " pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:13:27.557253 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.557230    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-tmp-dir\") pod \"autores-model-05731f-kserve-8dff98d79-l2cb6\" (UID: \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\") " pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:13:27.557321 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.557266    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-home\") pod \"autores-model-05731f-kserve-8dff98d79-l2cb6\" (UID: \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\") " pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:13:27.557321 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.557290    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-model-cache\") pod \"autores-model-05731f-kserve-8dff98d79-l2cb6\" (UID: \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\") " pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:13:27.559144 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.559125    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-dshm\") pod \"autores-model-05731f-kserve-8dff98d79-l2cb6\" (UID: \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\") " pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:13:27.559381 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.559364    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-tls-certs\") pod \"autores-model-05731f-kserve-8dff98d79-l2cb6\" (UID: \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\") " pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:13:27.565375 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.565353    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvk9w\" (UniqueName: \"kubernetes.io/projected/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-kube-api-access-xvk9w\") pod \"autores-model-05731f-kserve-8dff98d79-l2cb6\" (UID: \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\") " pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:13:27.712384 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.712353    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:13:27.983314 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:27.983290    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"]
Sep 15 21:13:27.984394 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:13:27.984357    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda58b6bb8_4f03_477a_9f16_a7faeaba5e2c.slice/crio-928f4f7b3bea5f8ef05525d72d7da881c3707e244bc2e16affc2dbb9d95abe4d WatchSource:0}: Error finding container 928f4f7b3bea5f8ef05525d72d7da881c3707e244bc2e16affc2dbb9d95abe4d: Status 404 returned error can't find the container with id 928f4f7b3bea5f8ef05525d72d7da881c3707e244bc2e16affc2dbb9d95abe4d
Sep 15 21:13:28.762028 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:28.761989    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6" event={"ID":"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c","Type":"ContainerStarted","Data":"ae382785d9b24ce4cd55ad20c3b8a6d6f7e641cf29a267fc9111c1388d3b27cb"}
Sep 15 21:13:28.762426 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:28.762038    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6" event={"ID":"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c","Type":"ContainerStarted","Data":"928f4f7b3bea5f8ef05525d72d7da881c3707e244bc2e16affc2dbb9d95abe4d"}
Sep 15 21:13:28.782572 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:28.782523    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6" podStartSLOduration=1.7825096870000001 podStartE2EDuration="1.782509687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:13:28.780349737 +0000 UTC m=+927.270995470" watchObservedRunningTime="2026-09-15 21:13:28.782509687 +0000 UTC m=+927.273155452"
Sep 15 21:13:34.198294 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:34.198254    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:13:37.713366 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:37.713334    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:13:37.713800 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:37.713381    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:13:37.725862 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:37.725839    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:13:37.828799 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:37.828777    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:13:53.785727 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:13:53.785686    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:14:11.371763 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:11.371725    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"]
Sep 15 21:14:11.372233 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:11.371951    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9" podUID="4c21937f-e198-4bd9-8f77-03f6d9ce455f" containerName="maas-api" containerID="cri-o://75d4f67b1674a2b6775531431f15c48824807a1ba3fb73446f08fe7adf6a68b1" gracePeriod=30
Sep 15 21:14:11.610730 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:11.610706    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"
Sep 15 21:14:11.665271 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:11.665243    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c21937f-e198-4bd9-8f77-03f6d9ce455f-maas-api-tls\") pod \"4c21937f-e198-4bd9-8f77-03f6d9ce455f\" (UID: \"4c21937f-e198-4bd9-8f77-03f6d9ce455f\") "
Sep 15 21:14:11.665427 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:11.665284    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p74dl\" (UniqueName: \"kubernetes.io/projected/4c21937f-e198-4bd9-8f77-03f6d9ce455f-kube-api-access-p74dl\") pod \"4c21937f-e198-4bd9-8f77-03f6d9ce455f\" (UID: \"4c21937f-e198-4bd9-8f77-03f6d9ce455f\") "
Sep 15 21:14:11.665427 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:11.665323    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c21937f-e198-4bd9-8f77-03f6d9ce455f-openshift-service-ca\") pod \"4c21937f-e198-4bd9-8f77-03f6d9ce455f\" (UID: \"4c21937f-e198-4bd9-8f77-03f6d9ce455f\") "
Sep 15 21:14:11.665551 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:11.665489    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c21937f-e198-4bd9-8f77-03f6d9ce455f-metrics-certs\") pod \"4c21937f-e198-4bd9-8f77-03f6d9ce455f\" (UID: \"4c21937f-e198-4bd9-8f77-03f6d9ce455f\") "
Sep 15 21:14:11.665771 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:11.665741    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c21937f-e198-4bd9-8f77-03f6d9ce455f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4c21937f-e198-4bd9-8f77-03f6d9ce455f" (UID: "4c21937f-e198-4bd9-8f77-03f6d9ce455f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:14:11.667424 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:11.667391    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c21937f-e198-4bd9-8f77-03f6d9ce455f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4c21937f-e198-4bd9-8f77-03f6d9ce455f" (UID: "4c21937f-e198-4bd9-8f77-03f6d9ce455f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:11.667544 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:11.667479    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c21937f-e198-4bd9-8f77-03f6d9ce455f-kube-api-access-p74dl" (OuterVolumeSpecName: "kube-api-access-p74dl") pod "4c21937f-e198-4bd9-8f77-03f6d9ce455f" (UID: "4c21937f-e198-4bd9-8f77-03f6d9ce455f"). InnerVolumeSpecName "kube-api-access-p74dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:14:11.667544 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:11.667490    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c21937f-e198-4bd9-8f77-03f6d9ce455f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4c21937f-e198-4bd9-8f77-03f6d9ce455f" (UID: "4c21937f-e198-4bd9-8f77-03f6d9ce455f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:11.766236 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:11.766193    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c21937f-e198-4bd9-8f77-03f6d9ce455f-metrics-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:14:11.766236 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:11.766226    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c21937f-e198-4bd9-8f77-03f6d9ce455f-maas-api-tls\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:14:11.766236 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:11.766236    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p74dl\" (UniqueName: \"kubernetes.io/projected/4c21937f-e198-4bd9-8f77-03f6d9ce455f-kube-api-access-p74dl\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:14:11.766236 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:11.766244    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c21937f-e198-4bd9-8f77-03f6d9ce455f-openshift-service-ca\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:14:12.011240 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:12.011209    2558 generic.go:356] "Generic (PLEG): container finished" podID="4c21937f-e198-4bd9-8f77-03f6d9ce455f" containerID="75d4f67b1674a2b6775531431f15c48824807a1ba3fb73446f08fe7adf6a68b1" exitCode=0
Sep 15 21:14:12.011416 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:12.011256    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"
Sep 15 21:14:12.011416 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:12.011294    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9" event={"ID":"4c21937f-e198-4bd9-8f77-03f6d9ce455f","Type":"ContainerDied","Data":"75d4f67b1674a2b6775531431f15c48824807a1ba3fb73446f08fe7adf6a68b1"}
Sep 15 21:14:12.011416 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:12.011355    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9" event={"ID":"4c21937f-e198-4bd9-8f77-03f6d9ce455f","Type":"ContainerDied","Data":"45782108a9fb3e9397334764519298f5ff7f00f4a50600cd3fd9e9b8e1822515"}
Sep 15 21:14:12.011416 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:12.011379    2558 scope.go:117] "RemoveContainer" containerID="75d4f67b1674a2b6775531431f15c48824807a1ba3fb73446f08fe7adf6a68b1"
Sep 15 21:14:12.020790 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:12.020775    2558 scope.go:117] "RemoveContainer" containerID="75d4f67b1674a2b6775531431f15c48824807a1ba3fb73446f08fe7adf6a68b1"
Sep 15 21:14:12.021067 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:14:12.021024    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d4f67b1674a2b6775531431f15c48824807a1ba3fb73446f08fe7adf6a68b1\": container with ID starting with 75d4f67b1674a2b6775531431f15c48824807a1ba3fb73446f08fe7adf6a68b1 not found: ID does not exist" containerID="75d4f67b1674a2b6775531431f15c48824807a1ba3fb73446f08fe7adf6a68b1"
Sep 15 21:14:12.021123 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:12.021067    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d4f67b1674a2b6775531431f15c48824807a1ba3fb73446f08fe7adf6a68b1"} err="failed to get container status \"75d4f67b1674a2b6775531431f15c48824807a1ba3fb73446f08fe7adf6a68b1\": rpc error: code = NotFound desc = could not find container \"75d4f67b1674a2b6775531431f15c48824807a1ba3fb73446f08fe7adf6a68b1\": container with ID starting with 75d4f67b1674a2b6775531431f15c48824807a1ba3fb73446f08fe7adf6a68b1 not found: ID does not exist"
Sep 15 21:14:12.030514 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:12.030485    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"]
Sep 15 21:14:12.034662 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:12.034640    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-05731f-7d5c66c988-xrgp9"]
Sep 15 21:14:13.942709 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:13.942673    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c21937f-e198-4bd9-8f77-03f6d9ce455f" path="/var/lib/kubelet/pods/4c21937f-e198-4bd9-8f77-03f6d9ce455f/volumes"
Sep 15 21:14:16.265814 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:16.265780    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"]
Sep 15 21:14:16.586373 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:16.586290    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:14:31.285357 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:31.285315    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6" podUID="a58b6bb8-4f03-477a-9f16-a7faeaba5e2c" containerName="main" containerID="cri-o://ae382785d9b24ce4cd55ad20c3b8a6d6f7e641cf29a267fc9111c1388d3b27cb" gracePeriod=45
Sep 15 21:14:31.535616 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:31.535555    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:14:31.639687 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:31.639654    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-model-cache\") pod \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\" (UID: \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\") "
Sep 15 21:14:31.639866 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:31.639694    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvk9w\" (UniqueName: \"kubernetes.io/projected/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-kube-api-access-xvk9w\") pod \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\" (UID: \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\") "
Sep 15 21:14:31.639866 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:31.639722    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-home\") pod \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\" (UID: \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\") "
Sep 15 21:14:31.639866 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:31.639768    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-tls-certs\") pod \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\" (UID: \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\") "
Sep 15 21:14:31.639866 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:31.639826    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-dshm\") pod \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\" (UID: \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\") "
Sep 15 21:14:31.639866 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:31.639862    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-tmp-dir\") pod \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\" (UID: \"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c\") "
Sep 15 21:14:31.640161 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:31.639948    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-model-cache" (OuterVolumeSpecName: "model-cache") pod "a58b6bb8-4f03-477a-9f16-a7faeaba5e2c" (UID: "a58b6bb8-4f03-477a-9f16-a7faeaba5e2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:14:31.640251 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:31.640214    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a58b6bb8-4f03-477a-9f16-a7faeaba5e2c" (UID: "a58b6bb8-4f03-477a-9f16-a7faeaba5e2c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:14:31.640329 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:31.640265    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-model-cache\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:14:31.640578 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:31.640542    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-home" (OuterVolumeSpecName: "home") pod "a58b6bb8-4f03-477a-9f16-a7faeaba5e2c" (UID: "a58b6bb8-4f03-477a-9f16-a7faeaba5e2c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:14:31.642403 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:31.642379    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-kube-api-access-xvk9w" (OuterVolumeSpecName: "kube-api-access-xvk9w") pod "a58b6bb8-4f03-477a-9f16-a7faeaba5e2c" (UID: "a58b6bb8-4f03-477a-9f16-a7faeaba5e2c"). InnerVolumeSpecName "kube-api-access-xvk9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:14:31.642616 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:31.642585    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a58b6bb8-4f03-477a-9f16-a7faeaba5e2c" (UID: "a58b6bb8-4f03-477a-9f16-a7faeaba5e2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:31.642701 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:31.642685    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-dshm" (OuterVolumeSpecName: "dshm") pod "a58b6bb8-4f03-477a-9f16-a7faeaba5e2c" (UID: "a58b6bb8-4f03-477a-9f16-a7faeaba5e2c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:14:31.740828 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:31.740800    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-dshm\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:14:31.740940 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:31.740830    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-tmp-dir\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:14:31.740940 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:31.740847    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvk9w\" (UniqueName: \"kubernetes.io/projected/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-kube-api-access-xvk9w\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:14:31.740940 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:31.740860    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-home\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:14:31.740940 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:31.740872    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c-tls-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:14:32.121664 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:32.121631    2558 generic.go:356] "Generic (PLEG): container finished" podID="a58b6bb8-4f03-477a-9f16-a7faeaba5e2c" containerID="ae382785d9b24ce4cd55ad20c3b8a6d6f7e641cf29a267fc9111c1388d3b27cb" exitCode=0
Sep 15 21:14:32.121664 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:32.121668    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6" event={"ID":"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c","Type":"ContainerDied","Data":"ae382785d9b24ce4cd55ad20c3b8a6d6f7e641cf29a267fc9111c1388d3b27cb"}
Sep 15 21:14:32.121939 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:32.121677    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"
Sep 15 21:14:32.121939 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:32.121695    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6" event={"ID":"a58b6bb8-4f03-477a-9f16-a7faeaba5e2c","Type":"ContainerDied","Data":"928f4f7b3bea5f8ef05525d72d7da881c3707e244bc2e16affc2dbb9d95abe4d"}
Sep 15 21:14:32.121939 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:32.121717    2558 scope.go:117] "RemoveContainer" containerID="ae382785d9b24ce4cd55ad20c3b8a6d6f7e641cf29a267fc9111c1388d3b27cb"
Sep 15 21:14:32.130993 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:32.130967    2558 scope.go:117] "RemoveContainer" containerID="ae382785d9b24ce4cd55ad20c3b8a6d6f7e641cf29a267fc9111c1388d3b27cb"
Sep 15 21:14:32.131289 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:14:32.131261    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae382785d9b24ce4cd55ad20c3b8a6d6f7e641cf29a267fc9111c1388d3b27cb\": container with ID starting with ae382785d9b24ce4cd55ad20c3b8a6d6f7e641cf29a267fc9111c1388d3b27cb not found: ID does not exist" containerID="ae382785d9b24ce4cd55ad20c3b8a6d6f7e641cf29a267fc9111c1388d3b27cb"
Sep 15 21:14:32.131345 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:32.131297    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae382785d9b24ce4cd55ad20c3b8a6d6f7e641cf29a267fc9111c1388d3b27cb"} err="failed to get container status \"ae382785d9b24ce4cd55ad20c3b8a6d6f7e641cf29a267fc9111c1388d3b27cb\": rpc error: code = NotFound desc = could not find container \"ae382785d9b24ce4cd55ad20c3b8a6d6f7e641cf29a267fc9111c1388d3b27cb\": container with ID starting with ae382785d9b24ce4cd55ad20c3b8a6d6f7e641cf29a267fc9111c1388d3b27cb not found: ID does not exist"
Sep 15 21:14:32.141161 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:32.141138    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"]
Sep 15 21:14:32.144871 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:32.144841    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-05731f/autores-model-05731f-kserve-8dff98d79-l2cb6"]
Sep 15 21:14:33.942735 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:33.942689    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a58b6bb8-4f03-477a-9f16-a7faeaba5e2c" path="/var/lib/kubelet/pods/a58b6bb8-4f03-477a-9f16-a7faeaba5e2c/volumes"
Sep 15 21:14:36.158861 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.158820    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"]
Sep 15 21:14:36.159255 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.159169    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a669e06-19ec-4c9c-ab0d-a9e887cbf97f" containerName="istio-proxy"
Sep 15 21:14:36.159255 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.159180    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a669e06-19ec-4c9c-ab0d-a9e887cbf97f" containerName="istio-proxy"
Sep 15 21:14:36.159255 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.159189    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a58b6bb8-4f03-477a-9f16-a7faeaba5e2c" containerName="main"
Sep 15 21:14:36.159255 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.159194    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a58b6bb8-4f03-477a-9f16-a7faeaba5e2c" containerName="main"
Sep 15 21:14:36.159255 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.159206    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c21937f-e198-4bd9-8f77-03f6d9ce455f" containerName="maas-api"
Sep 15 21:14:36.159255 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.159212    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c21937f-e198-4bd9-8f77-03f6d9ce455f" containerName="maas-api"
Sep 15 21:14:36.159483 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.159273    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a58b6bb8-4f03-477a-9f16-a7faeaba5e2c" containerName="main"
Sep 15 21:14:36.159483 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.159281    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c21937f-e198-4bd9-8f77-03f6d9ce455f" containerName="maas-api"
Sep 15 21:14:36.159483 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.159289    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a669e06-19ec-4c9c-ab0d-a9e887cbf97f" containerName="istio-proxy"
Sep 15 21:14:36.163749 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.163729    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.166602 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.166578    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b7100513-openshift-default-dockercfg-hnf8p\""
Sep 15 21:14:36.174191 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.174171    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"]
Sep 15 21:14:36.280571 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.280533    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-istio-data\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.280815 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.280585    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb37284-d74d-460d-810c-d7128bb995f5-istio-token\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.280815 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.280665    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-workload-socket\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.280815 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.280713    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb37284-d74d-460d-810c-d7128bb995f5-istio-podinfo\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.280815 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.280747    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl7rz\" (UniqueName: \"kubernetes.io/projected/ceb37284-d74d-460d-810c-d7128bb995f5-kube-api-access-wl7rz\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.281063 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.280862    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-workload-certs\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.281063 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.280948    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-credential-socket\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.281158 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.281041    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-istio-envoy\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.281158 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.281098    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb37284-d74d-460d-810c-d7128bb995f5-istiod-ca-cert\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.382168 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.382132    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb37284-d74d-460d-810c-d7128bb995f5-istio-token\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.382344 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.382184    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-workload-socket\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.382344 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.382215    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb37284-d74d-460d-810c-d7128bb995f5-istio-podinfo\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.382344 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.382237    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl7rz\" (UniqueName: \"kubernetes.io/projected/ceb37284-d74d-460d-810c-d7128bb995f5-kube-api-access-wl7rz\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.382344 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.382258    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-workload-certs\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.382344 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.382298    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-credential-socket\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.382620 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.382349    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-istio-envoy\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.382620 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.382373    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb37284-d74d-460d-810c-d7128bb995f5-istiod-ca-cert\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.382620 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.382419    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-istio-data\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.382772 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.382730    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-workload-socket\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.382772 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.382745    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-workload-certs\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.382905 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.382881    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-istio-data\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.382992 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.382887    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-credential-socket\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.383334 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.383308    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb37284-d74d-460d-810c-d7128bb995f5-istiod-ca-cert\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.384761 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.384738    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-istio-envoy\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.384931 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.384912    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb37284-d74d-460d-810c-d7128bb995f5-istio-podinfo\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.391143 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.391120    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb37284-d74d-460d-810c-d7128bb995f5-istio-token\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.391235 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.391148    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl7rz\" (UniqueName: \"kubernetes.io/projected/ceb37284-d74d-460d-810c-d7128bb995f5-kube-api-access-wl7rz\") pod \"e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") " pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.475061 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.475034    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:36.813946 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:14:36.813917    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb37284_d74d_460d_810c_d7128bb995f5.slice/crio-5e1a72e48b23a853fa5f578832d38bc95b886c007ea192d3d451fb127aff57bd WatchSource:0}: Error finding container 5e1a72e48b23a853fa5f578832d38bc95b886c007ea192d3d451fb127aff57bd: Status 404 returned error can't find the container with id 5e1a72e48b23a853fa5f578832d38bc95b886c007ea192d3d451fb127aff57bd
Sep 15 21:14:36.815679 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.815654    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"]
Sep 15 21:14:36.816400 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.816365    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:14:36.816473 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.816446    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:14:36.816516 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:36.816494    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:14:37.151918 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:37.151829    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5" event={"ID":"ceb37284-d74d-460d-810c-d7128bb995f5","Type":"ContainerStarted","Data":"9c009600e4ccc2956a59308e73bf8bcae1a4719741d83b99ff3fff12a1ecac09"}
Sep 15 21:14:37.152086 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:37.151928    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5" event={"ID":"ceb37284-d74d-460d-810c-d7128bb995f5","Type":"ContainerStarted","Data":"5e1a72e48b23a853fa5f578832d38bc95b886c007ea192d3d451fb127aff57bd"}
Sep 15 21:14:37.180809 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:37.180762    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5" podStartSLOduration=1.180746869 podStartE2EDuration="1.180746869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:14:37.179746562 +0000 UTC m=+995.670392294" watchObservedRunningTime="2026-09-15 21:14:37.180746869 +0000 UTC m=+995.671392598"
Sep 15 21:14:37.475481 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:37.475450    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:37.480198 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:37.480176    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:38.157473 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:38.157434    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:38.158390 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:38.158368    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:14:43.197378 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:43.197343    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:14:45.632949 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:45.632913    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"]
Sep 15 21:14:45.637207 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:45.637181    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"
Sep 15 21:14:45.640338 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:45.640314    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 21:14:45.640462 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:45.640390    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b7100513-serving-cert\""
Sep 15 21:14:45.641515 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:45.641490    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 21:14:45.641688 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:45.641548    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:14:45.641753 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:45.641714    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8hfhw\""
Sep 15 21:14:45.647335 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:45.647270    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"]
Sep 15 21:14:45.665554 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:45.665533    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84146531-3e8c-4d16-8ce5-ecb596327477-metrics-certs\") pod \"maas-api-e2e-mt-b7100513-56d79b5c99-6js4v\" (UID: \"84146531-3e8c-4d16-8ce5-ecb596327477\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"
Sep 15 21:14:45.665654 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:45.665569    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84146531-3e8c-4d16-8ce5-ecb596327477-maas-api-tls\") pod \"maas-api-e2e-mt-b7100513-56d79b5c99-6js4v\" (UID: \"84146531-3e8c-4d16-8ce5-ecb596327477\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"
Sep 15 21:14:45.665654 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:45.665642    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t88d\" (UniqueName: \"kubernetes.io/projected/84146531-3e8c-4d16-8ce5-ecb596327477-kube-api-access-8t88d\") pod \"maas-api-e2e-mt-b7100513-56d79b5c99-6js4v\" (UID: \"84146531-3e8c-4d16-8ce5-ecb596327477\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"
Sep 15 21:14:45.665740 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:45.665680    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84146531-3e8c-4d16-8ce5-ecb596327477-openshift-service-ca\") pod \"maas-api-e2e-mt-b7100513-56d79b5c99-6js4v\" (UID: \"84146531-3e8c-4d16-8ce5-ecb596327477\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"
Sep 15 21:14:45.766866 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:45.766835    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84146531-3e8c-4d16-8ce5-ecb596327477-openshift-service-ca\") pod \"maas-api-e2e-mt-b7100513-56d79b5c99-6js4v\" (UID: \"84146531-3e8c-4d16-8ce5-ecb596327477\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"
Sep 15 21:14:45.767027 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:45.766886    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84146531-3e8c-4d16-8ce5-ecb596327477-metrics-certs\") pod \"maas-api-e2e-mt-b7100513-56d79b5c99-6js4v\" (UID: \"84146531-3e8c-4d16-8ce5-ecb596327477\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"
Sep 15 21:14:45.767027 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:45.766917    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84146531-3e8c-4d16-8ce5-ecb596327477-maas-api-tls\") pod \"maas-api-e2e-mt-b7100513-56d79b5c99-6js4v\" (UID: \"84146531-3e8c-4d16-8ce5-ecb596327477\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"
Sep 15 21:14:45.767027 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:45.766982    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t88d\" (UniqueName: \"kubernetes.io/projected/84146531-3e8c-4d16-8ce5-ecb596327477-kube-api-access-8t88d\") pod \"maas-api-e2e-mt-b7100513-56d79b5c99-6js4v\" (UID: \"84146531-3e8c-4d16-8ce5-ecb596327477\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"
Sep 15 21:14:45.767167 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:14:45.767088    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-serving-cert: secret "maas-api-e2e-mt-b7100513-serving-cert" not found
Sep 15 21:14:45.767203 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:14:45.767168    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84146531-3e8c-4d16-8ce5-ecb596327477-maas-api-tls podName:84146531-3e8c-4d16-8ce5-ecb596327477 nodeName:}" failed. No retries permitted until 2026-09-15 21:14:46.26714869 +0000 UTC m=+1004.757794408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/84146531-3e8c-4d16-8ce5-ecb596327477-maas-api-tls") pod "maas-api-e2e-mt-b7100513-56d79b5c99-6js4v" (UID: "84146531-3e8c-4d16-8ce5-ecb596327477") : secret "maas-api-e2e-mt-b7100513-serving-cert" not found
Sep 15 21:14:45.767510 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:45.767491    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84146531-3e8c-4d16-8ce5-ecb596327477-openshift-service-ca\") pod \"maas-api-e2e-mt-b7100513-56d79b5c99-6js4v\" (UID: \"84146531-3e8c-4d16-8ce5-ecb596327477\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"
Sep 15 21:14:45.769316 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:45.769298    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84146531-3e8c-4d16-8ce5-ecb596327477-metrics-certs\") pod \"maas-api-e2e-mt-b7100513-56d79b5c99-6js4v\" (UID: \"84146531-3e8c-4d16-8ce5-ecb596327477\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"
Sep 15 21:14:45.776907 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:45.776887    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t88d\" (UniqueName: \"kubernetes.io/projected/84146531-3e8c-4d16-8ce5-ecb596327477-kube-api-access-8t88d\") pod \"maas-api-e2e-mt-b7100513-56d79b5c99-6js4v\" (UID: \"84146531-3e8c-4d16-8ce5-ecb596327477\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"
Sep 15 21:14:46.270834 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:46.270801    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84146531-3e8c-4d16-8ce5-ecb596327477-maas-api-tls\") pod \"maas-api-e2e-mt-b7100513-56d79b5c99-6js4v\" (UID: \"84146531-3e8c-4d16-8ce5-ecb596327477\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"
Sep 15 21:14:46.273344 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:46.273313    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84146531-3e8c-4d16-8ce5-ecb596327477-maas-api-tls\") pod \"maas-api-e2e-mt-b7100513-56d79b5c99-6js4v\" (UID: \"84146531-3e8c-4d16-8ce5-ecb596327477\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"
Sep 15 21:14:46.550671 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:46.550589    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"
Sep 15 21:14:46.820409 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:46.820385    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"]
Sep 15 21:14:46.822730 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:14:46.822705    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84146531_3e8c_4d16_8ce5_ecb596327477.slice/crio-9a87220183aea7c14667e3d4fddffe68ec8737611b2d3163e23c5a9e5e2f4108 WatchSource:0}: Error finding container 9a87220183aea7c14667e3d4fddffe68ec8737611b2d3163e23c5a9e5e2f4108: Status 404 returned error can't find the container with id 9a87220183aea7c14667e3d4fddffe68ec8737611b2d3163e23c5a9e5e2f4108
Sep 15 21:14:46.824431 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:46.824412    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:14:47.207027 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:47.206992    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v" event={"ID":"84146531-3e8c-4d16-8ce5-ecb596327477","Type":"ContainerStarted","Data":"9a87220183aea7c14667e3d4fddffe68ec8737611b2d3163e23c5a9e5e2f4108"}
Sep 15 21:14:48.214098 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:48.214061    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v" event={"ID":"84146531-3e8c-4d16-8ce5-ecb596327477","Type":"ContainerStarted","Data":"2b78bc069ebb09096c9b797a74c7d535aee8cc99310f3dac097b5ca6a54f5648"}
Sep 15 21:14:48.214488 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:48.214126    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"
Sep 15 21:14:48.249823 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:48.249771    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v" podStartSLOduration=2.855489171 podStartE2EDuration="3.249753897s" totalImagesPullingTime="394.264726ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:14:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:14:48.246890118 +0000 UTC m=+1006.737535851" watchObservedRunningTime="2026-09-15 21:14:48.249753897 +0000 UTC m=+1006.740399629"
Sep 15 21:14:54.137731 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:54.137700    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"]
Sep 15 21:14:54.138162 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:54.137949    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89" podUID="4f826e85-24be-4260-be0f-ff7c30ae5f2e" containerName="istio-proxy" containerID="cri-o://6b1bd45e4a9b4828a484a0b5dd973192fcf62b8b5592e4e7cddd33e451f6379a" gracePeriod=30
Sep 15 21:14:54.225324 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:54.225294    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"
Sep 15 21:14:54.842332 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:54.842295    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"]
Sep 15 21:14:54.845964 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:54.845938    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:54.848900 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:54.848877    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-8f08a9-openshift-default-dockercfg-fwtb2\""
Sep 15 21:14:54.857891 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:54.857866    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"]
Sep 15 21:14:54.964713 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:54.964682    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-workload-certs\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:54.964887 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:54.964744    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dede905-5520-4367-a6d0-dda15d3f0f59-istio-podinfo\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:54.964887 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:54.964773    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dede905-5520-4367-a6d0-dda15d3f0f59-istio-token\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:54.964887 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:54.964815    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-credential-socket\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:54.964887 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:54.964835    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmrsf\" (UniqueName: \"kubernetes.io/projected/4dede905-5520-4367-a6d0-dda15d3f0f59-kube-api-access-dmrsf\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:54.964887 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:54.964854    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-istio-envoy\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:54.964887 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:54.964875    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-istio-data\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:54.965229 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:54.964949    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-workload-socket\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:54.965229 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:54.965027    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dede905-5520-4367-a6d0-dda15d3f0f59-istiod-ca-cert\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:55.066209 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:55.066180    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-workload-certs\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:55.066374 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:55.066231    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dede905-5520-4367-a6d0-dda15d3f0f59-istio-podinfo\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:55.066374 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:55.066251    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dede905-5520-4367-a6d0-dda15d3f0f59-istio-token\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:55.066374 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:55.066269    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-credential-socket\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:55.066374 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:55.066295    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmrsf\" (UniqueName: \"kubernetes.io/projected/4dede905-5520-4367-a6d0-dda15d3f0f59-kube-api-access-dmrsf\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:55.066374 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:55.066323    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-istio-envoy\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:55.066374 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:55.066354    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-istio-data\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:55.066728 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:55.066390    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-workload-socket\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:55.066728 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:55.066417    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dede905-5520-4367-a6d0-dda15d3f0f59-istiod-ca-cert\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:55.066852 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:55.066757    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-credential-socket\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:55.066943 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:55.066853    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-istio-data\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:55.067002 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:55.066962    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-workload-socket\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:55.067096 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:55.067004    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-workload-certs\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:55.067340 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:55.067304    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dede905-5520-4367-a6d0-dda15d3f0f59-istiod-ca-cert\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:55.068850 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:55.068828    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-istio-envoy\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:55.068961 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:55.068945    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dede905-5520-4367-a6d0-dda15d3f0f59-istio-podinfo\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:55.075294 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:55.075270    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dede905-5520-4367-a6d0-dda15d3f0f59-istio-token\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:55.075381 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:55.075318    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmrsf\" (UniqueName: \"kubernetes.io/projected/4dede905-5520-4367-a6d0-dda15d3f0f59-kube-api-access-dmrsf\") pod \"e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") " pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:55.157408 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:55.157342    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:55.433863 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:55.433834    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"]
Sep 15 21:14:55.435505 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:14:55.435481    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dede905_5520_4367_a6d0_dda15d3f0f59.slice/crio-71501753a151f9fe47227aca6e58c110dcae6463e2b66960b9f3d747ffaf2448 WatchSource:0}: Error finding container 71501753a151f9fe47227aca6e58c110dcae6463e2b66960b9f3d747ffaf2448: Status 404 returned error can't find the container with id 71501753a151f9fe47227aca6e58c110dcae6463e2b66960b9f3d747ffaf2448
Sep 15 21:14:55.437752 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:55.437719    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:14:55.437853 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:55.437783    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:14:55.437853 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:55.437819    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:14:56.263182 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:56.263148    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt" event={"ID":"4dede905-5520-4367-a6d0-dda15d3f0f59","Type":"ContainerStarted","Data":"0f60f88f25225b2f3f1221b1c5cdbedf239a05d3e827ee83055da27c7273bf15"}
Sep 15 21:14:56.263579 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:56.263200    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt" event={"ID":"4dede905-5520-4367-a6d0-dda15d3f0f59","Type":"ContainerStarted","Data":"71501753a151f9fe47227aca6e58c110dcae6463e2b66960b9f3d747ffaf2448"}
Sep 15 21:14:56.296549 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:56.296504    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt" podStartSLOduration=2.296488985 podStartE2EDuration="2.296488985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:14:56.29303289 +0000 UTC m=+1014.783678660" watchObservedRunningTime="2026-09-15 21:14:56.296488985 +0000 UTC m=+1014.787134717"
Sep 15 21:14:57.157570 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:57.157538    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:57.164234 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:57.164207    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:57.268579 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:57.268549    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:57.269597 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:57.269572    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:14:58.567421 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:58.567382    2558 patch_prober.go:28] interesting pod/e2e-autores-05731f-openshift-default-675d57d64b-gpd89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 21:14:58.567806 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:58.567448    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89" podUID="4f826e85-24be-4260-be0f-ff7c30ae5f2e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 21:14:58.892405 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:58.892322    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"]
Sep 15 21:14:59.285387 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.285350    2558 generic.go:356] "Generic (PLEG): container finished" podID="4f826e85-24be-4260-be0f-ff7c30ae5f2e" containerID="6b1bd45e4a9b4828a484a0b5dd973192fcf62b8b5592e4e7cddd33e451f6379a" exitCode=0
Sep 15 21:14:59.285550 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.285414    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89" event={"ID":"4f826e85-24be-4260-be0f-ff7c30ae5f2e","Type":"ContainerDied","Data":"6b1bd45e4a9b4828a484a0b5dd973192fcf62b8b5592e4e7cddd33e451f6379a"}
Sep 15 21:14:59.285550 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.285498    2558 scope.go:117] "RemoveContainer" containerID="3078e7b4937cefa97ce6caf61725654ebae1972097365c0411e6a8dffa05e3c1"
Sep 15 21:14:59.428701 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.428677    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:14:59.510570 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.510538    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istio-data\") pod \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") "
Sep 15 21:14:59.510570 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.510571    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-workload-certs\") pod \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") "
Sep 15 21:14:59.510802 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.510610    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istio-envoy\") pod \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") "
Sep 15 21:14:59.510802 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.510633    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istiod-ca-cert\") pod \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") "
Sep 15 21:14:59.510802 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.510668    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-credential-socket\") pod \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") "
Sep 15 21:14:59.510802 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.510709    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istio-podinfo\") pod \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") "
Sep 15 21:14:59.510989 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.510835    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istio-token\") pod \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") "
Sep 15 21:14:59.510989 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.510833    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istio-data" (OuterVolumeSpecName: "istio-data") pod "4f826e85-24be-4260-be0f-ff7c30ae5f2e" (UID: "4f826e85-24be-4260-be0f-ff7c30ae5f2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:14:59.510989 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.510873    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4f826e85-24be-4260-be0f-ff7c30ae5f2e" (UID: "4f826e85-24be-4260-be0f-ff7c30ae5f2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:14:59.510989 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.510900    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-workload-socket\") pod \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") "
Sep 15 21:14:59.510989 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.510932    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqwtc\" (UniqueName: \"kubernetes.io/projected/4f826e85-24be-4260-be0f-ff7c30ae5f2e-kube-api-access-sqwtc\") pod \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\" (UID: \"4f826e85-24be-4260-be0f-ff7c30ae5f2e\") "
Sep 15 21:14:59.511286 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.510999    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4f826e85-24be-4260-be0f-ff7c30ae5f2e" (UID: "4f826e85-24be-4260-be0f-ff7c30ae5f2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:14:59.511286 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.511093    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4f826e85-24be-4260-be0f-ff7c30ae5f2e" (UID: "4f826e85-24be-4260-be0f-ff7c30ae5f2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:14:59.511286 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.511175    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4f826e85-24be-4260-be0f-ff7c30ae5f2e" (UID: "4f826e85-24be-4260-be0f-ff7c30ae5f2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:14:59.511418 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.511333    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-workload-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:14:59.511418 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.511349    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istio-data\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:14:59.511418 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.511385    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-workload-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:14:59.511418 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.511398    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istiod-ca-cert\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:14:59.511418 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.511411    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-credential-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:14:59.512903 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.512876    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4f826e85-24be-4260-be0f-ff7c30ae5f2e" (UID: "4f826e85-24be-4260-be0f-ff7c30ae5f2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:14:59.513039 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.513025    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f826e85-24be-4260-be0f-ff7c30ae5f2e-kube-api-access-sqwtc" (OuterVolumeSpecName: "kube-api-access-sqwtc") pod "4f826e85-24be-4260-be0f-ff7c30ae5f2e" (UID: "4f826e85-24be-4260-be0f-ff7c30ae5f2e"). InnerVolumeSpecName "kube-api-access-sqwtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:14:59.513112 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.513088    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istio-token" (OuterVolumeSpecName: "istio-token") pod "4f826e85-24be-4260-be0f-ff7c30ae5f2e" (UID: "4f826e85-24be-4260-be0f-ff7c30ae5f2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:14:59.513162 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.513109    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4f826e85-24be-4260-be0f-ff7c30ae5f2e" (UID: "4f826e85-24be-4260-be0f-ff7c30ae5f2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:14:59.612453 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.612407    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqwtc\" (UniqueName: \"kubernetes.io/projected/4f826e85-24be-4260-be0f-ff7c30ae5f2e-kube-api-access-sqwtc\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:14:59.612453 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.612449    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istio-envoy\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:14:59.612453 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.612458    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istio-podinfo\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:14:59.612453 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:14:59.612468    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f826e85-24be-4260-be0f-ff7c30ae5f2e-istio-token\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:15:00.148261 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.146189    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0df719-29825115-hhfbp"]
Sep 15 21:15:00.148261 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.146766    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f826e85-24be-4260-be0f-ff7c30ae5f2e" containerName="istio-proxy"
Sep 15 21:15:00.148261 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.146789    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f826e85-24be-4260-be0f-ff7c30ae5f2e" containerName="istio-proxy"
Sep 15 21:15:00.148261 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.146804    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f826e85-24be-4260-be0f-ff7c30ae5f2e" containerName="istio-proxy"
Sep 15 21:15:00.148261 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.146813    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f826e85-24be-4260-be0f-ff7c30ae5f2e" containerName="istio-proxy"
Sep 15 21:15:00.148261 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.146919    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f826e85-24be-4260-be0f-ff7c30ae5f2e" containerName="istio-proxy"
Sep 15 21:15:00.148261 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.146932    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f826e85-24be-4260-be0f-ff7c30ae5f2e" containerName="istio-proxy"
Sep 15 21:15:00.150793 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.150762    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-544890-29825115-bzz5t"]
Sep 15 21:15:00.152503 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.152478    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0df719-29825115-hhfbp"
Sep 15 21:15:00.154425 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.154402    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-544890-29825115-bzz5t"
Sep 15 21:15:00.155569 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.155544    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvf2c\""
Sep 15 21:15:00.160254 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.160195    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0df719-29825115-hhfbp"]
Sep 15 21:15:00.163746 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.163728    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-544890-29825115-bzz5t"]
Sep 15 21:15:00.169775 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.169747    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0df719-29825115-hhfbp"
Sep 15 21:15:00.181101 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.174094    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dca193-29825115-zkx9w"]
Sep 15 21:15:00.185333 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.184949    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dca193-29825115-zkx9w"
Sep 15 21:15:00.188484 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.188430    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dca193-29825115-zkx9w"]
Sep 15 21:15:00.207712 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.207688    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-544890-29825115-bzz5t"
Sep 15 21:15:00.215680 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.215524    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dca193-29825115-zkx9w"
Sep 15 21:15:00.304266 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.303995    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89" event={"ID":"4f826e85-24be-4260-be0f-ff7c30ae5f2e","Type":"ContainerDied","Data":"3d87c8e7d653ecac0f797a00ff980fe13b9ad5b71e042371b81ab280c79c9063"}
Sep 15 21:15:00.304266 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.304096    2558 scope.go:117] "RemoveContainer" containerID="6b1bd45e4a9b4828a484a0b5dd973192fcf62b8b5592e4e7cddd33e451f6379a"
Sep 15 21:15:00.304266 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.304122    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"
Sep 15 21:15:00.327449 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.327328    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"]
Sep 15 21:15:00.339747 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.339721    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-05731f-openshift-default-675d57d64b-gpd89"]
Sep 15 21:15:00.465240 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.464534    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0df719-29825115-hhfbp"]
Sep 15 21:15:00.471974 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:15:00.471946    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82838560_39b4_4124_8865_54277ea10b52.slice/crio-a69826f5ba85a6e7e6e247d6d4d07f124029043fe859c274fff24f05009ae4e1 WatchSource:0}: Error finding container a69826f5ba85a6e7e6e247d6d4d07f124029043fe859c274fff24f05009ae4e1: Status 404 returned error can't find the container with id a69826f5ba85a6e7e6e247d6d4d07f124029043fe859c274fff24f05009ae4e1
Sep 15 21:15:00.540184 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.540156    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-544890-29825115-bzz5t"]
Sep 15 21:15:00.540955 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:15:00.540924    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd559e9_c604_4256_97e1_805f47eac94f.slice/crio-dd72d28921b9dacc751c1992ef4fb5aa7de1f1c6bc40fa799d433fc11bed00c8 WatchSource:0}: Error finding container dd72d28921b9dacc751c1992ef4fb5aa7de1f1c6bc40fa799d433fc11bed00c8: Status 404 returned error can't find the container with id dd72d28921b9dacc751c1992ef4fb5aa7de1f1c6bc40fa799d433fc11bed00c8
Sep 15 21:15:00.564952 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:00.564923    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dca193-29825115-zkx9w"]
Sep 15 21:15:00.569069 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:15:00.565338    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80dfa469_11dc_4819_ab15_22b33ec7d8cb.slice/crio-04cd2190c19566e64597d38ad843005c571bea7507472def38e8b0d8e6ee5df4 WatchSource:0}: Error finding container 04cd2190c19566e64597d38ad843005c571bea7507472def38e8b0d8e6ee5df4: Status 404 returned error can't find the container with id 04cd2190c19566e64597d38ad843005c571bea7507472def38e8b0d8e6ee5df4
Sep 15 21:15:01.314384 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:01.314306    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-544890-29825115-bzz5t" event={"ID":"1cd559e9-c604-4256-97e1-805f47eac94f","Type":"ContainerStarted","Data":"dd72d28921b9dacc751c1992ef4fb5aa7de1f1c6bc40fa799d433fc11bed00c8"}
Sep 15 21:15:01.315983 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:01.315942    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dca193-29825115-zkx9w" event={"ID":"80dfa469-11dc-4819-ab15-22b33ec7d8cb","Type":"ContainerStarted","Data":"04cd2190c19566e64597d38ad843005c571bea7507472def38e8b0d8e6ee5df4"}
Sep 15 21:15:01.317522 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:01.317495    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0df719-29825115-hhfbp" event={"ID":"82838560-39b4-4124-8865-54277ea10b52","Type":"ContainerStarted","Data":"a69826f5ba85a6e7e6e247d6d4d07f124029043fe859c274fff24f05009ae4e1"}
Sep 15 21:15:01.968091 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:01.965394    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f826e85-24be-4260-be0f-ff7c30ae5f2e" path="/var/lib/kubelet/pods/4f826e85-24be-4260-be0f-ff7c30ae5f2e/volumes"
Sep 15 21:15:02.080324 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:02.080288    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"]
Sep 15 21:15:02.085988 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:02.085965    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"
Sep 15 21:15:02.091948 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:02.091918    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"]
Sep 15 21:15:02.092385 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:02.092192    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-8f08a9-serving-cert\""
Sep 15 21:15:02.141982 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:02.141815    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-openshift-service-ca\") pod \"maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd\" (UID: \"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"
Sep 15 21:15:02.141982 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:02.141893    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-maas-api-tls\") pod \"maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd\" (UID: \"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"
Sep 15 21:15:02.141982 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:02.141933    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-metrics-certs\") pod \"maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd\" (UID: \"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"
Sep 15 21:15:02.142345 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:02.142153    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9jhw\" (UniqueName: \"kubernetes.io/projected/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-kube-api-access-w9jhw\") pod \"maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd\" (UID: \"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"
Sep 15 21:15:02.262131 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:02.261445    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-metrics-certs\") pod \"maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd\" (UID: \"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"
Sep 15 21:15:02.262131 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:02.261579    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9jhw\" (UniqueName: \"kubernetes.io/projected/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-kube-api-access-w9jhw\") pod \"maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd\" (UID: \"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"
Sep 15 21:15:02.262131 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:02.261617    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-openshift-service-ca\") pod \"maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd\" (UID: \"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"
Sep 15 21:15:02.262131 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:02.261671    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-maas-api-tls\") pod \"maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd\" (UID: \"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"
Sep 15 21:15:02.262131 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:15:02.261808    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-serving-cert: secret "maas-api-e2e-explicit-8f08a9-serving-cert" not found
Sep 15 21:15:02.262131 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:15:02.261877    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-maas-api-tls podName:8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b nodeName:}" failed. No retries permitted until 2026-09-15 21:15:02.761856467 +0000 UTC m=+1021.252502184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-maas-api-tls") pod "maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd" (UID: "8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b") : secret "maas-api-e2e-explicit-8f08a9-serving-cert" not found
Sep 15 21:15:02.267102 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:02.263221    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-openshift-service-ca\") pod \"maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd\" (UID: \"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"
Sep 15 21:15:02.271259 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:02.271234    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-metrics-certs\") pod \"maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd\" (UID: \"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"
Sep 15 21:15:02.277427 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:02.277399    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9jhw\" (UniqueName: \"kubernetes.io/projected/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-kube-api-access-w9jhw\") pod \"maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd\" (UID: \"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"
Sep 15 21:15:02.773071 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:02.772995    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-maas-api-tls\") pod \"maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd\" (UID: \"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"
Sep 15 21:15:02.776243 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:02.776187    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-maas-api-tls\") pod \"maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd\" (UID: \"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"
Sep 15 21:15:03.003420 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:03.003383    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"
Sep 15 21:15:03.727945 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:03.727913    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"]
Sep 15 21:15:03.729653 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:15:03.729627    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d13f43c_bdbc_4ff2_bb0c_e868ab2d737b.slice/crio-b3f4410ef4ca7a867ffad2bdcdd44f559357658529d85029677090c17ee8bbb5 WatchSource:0}: Error finding container b3f4410ef4ca7a867ffad2bdcdd44f559357658529d85029677090c17ee8bbb5: Status 404 returned error can't find the container with id b3f4410ef4ca7a867ffad2bdcdd44f559357658529d85029677090c17ee8bbb5
Sep 15 21:15:04.341625 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:04.341581    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd" event={"ID":"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b","Type":"ContainerStarted","Data":"b3f4410ef4ca7a867ffad2bdcdd44f559357658529d85029677090c17ee8bbb5"}
Sep 15 21:15:04.343454 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:04.343419    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-544890-29825115-bzz5t" event={"ID":"1cd559e9-c604-4256-97e1-805f47eac94f","Type":"ContainerStarted","Data":"fd4b2b87034e27e877478315a4ae9e541b205ec4c79eb52b98cdca61facf1da5"}
Sep 15 21:15:04.345800 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:04.345760    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dca193-29825115-zkx9w" event={"ID":"80dfa469-11dc-4819-ab15-22b33ec7d8cb","Type":"ContainerStarted","Data":"6816a9e0df09a1bcf4f9d768548715777370132f4c347f51ce462484db005b27"}
Sep 15 21:15:04.347735 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:04.347704    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0df719-29825115-hhfbp" event={"ID":"82838560-39b4-4124-8865-54277ea10b52","Type":"ContainerStarted","Data":"2dd6743e99786b4b83cb4c01d23e7e113f3f73e4809cdc20d717c71cfed9731d"}
Sep 15 21:15:04.367128 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:04.367074    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-544890-29825115-bzz5t" podStartSLOduration=1.391694715 podStartE2EDuration="4.367039882s" totalImagesPullingTime="2.975345167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:15:04.363319353 +0000 UTC m=+1022.853965087" watchObservedRunningTime="2026-09-15 21:15:04.367039882 +0000 UTC m=+1022.857685614"
Sep 15 21:15:04.379352 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:04.379311    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0df719-29825115-hhfbp" podStartSLOduration=1.274906114 podStartE2EDuration="4.379298579s" totalImagesPullingTime="3.104392465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:15:04.37846708 +0000 UTC m=+1022.869112814" watchObservedRunningTime="2026-09-15 21:15:04.379298579 +0000 UTC m=+1022.869944311"
Sep 15 21:15:04.396358 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:04.396316    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dca193-29825115-zkx9w" podStartSLOduration=1.387446265 podStartE2EDuration="4.396302507s" totalImagesPullingTime="3.008856242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:15:04.395665151 +0000 UTC m=+1022.886310883" watchObservedRunningTime="2026-09-15 21:15:04.396302507 +0000 UTC m=+1022.886948239"
Sep 15 21:15:05.357693 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:05.357651    2558 generic.go:356] "Generic (PLEG): container finished" podID="80dfa469-11dc-4819-ab15-22b33ec7d8cb" containerID="6816a9e0df09a1bcf4f9d768548715777370132f4c347f51ce462484db005b27" exitCode=0
Sep 15 21:15:05.358201 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:05.357693    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dca193-29825115-zkx9w" event={"ID":"80dfa469-11dc-4819-ab15-22b33ec7d8cb","Type":"ContainerDied","Data":"6816a9e0df09a1bcf4f9d768548715777370132f4c347f51ce462484db005b27"}
Sep 15 21:15:05.359651 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:05.359624    2558 generic.go:356] "Generic (PLEG): container finished" podID="82838560-39b4-4124-8865-54277ea10b52" containerID="2dd6743e99786b4b83cb4c01d23e7e113f3f73e4809cdc20d717c71cfed9731d" exitCode=0
Sep 15 21:15:05.359750 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:05.359727    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0df719-29825115-hhfbp" event={"ID":"82838560-39b4-4124-8865-54277ea10b52","Type":"ContainerDied","Data":"2dd6743e99786b4b83cb4c01d23e7e113f3f73e4809cdc20d717c71cfed9731d"}
Sep 15 21:15:05.361459 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:05.361433    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd" event={"ID":"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b","Type":"ContainerStarted","Data":"1b2c6683162695649de7500bfaf194f9e6021e22fd7314f7743433dce643fa87"}
Sep 15 21:15:05.361568 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:05.361506    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"
Sep 15 21:15:05.363097 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:05.363074    2558 generic.go:356] "Generic (PLEG): container finished" podID="1cd559e9-c604-4256-97e1-805f47eac94f" containerID="fd4b2b87034e27e877478315a4ae9e541b205ec4c79eb52b98cdca61facf1da5" exitCode=0
Sep 15 21:15:05.363212 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:05.363195    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-544890-29825115-bzz5t" event={"ID":"1cd559e9-c604-4256-97e1-805f47eac94f","Type":"ContainerDied","Data":"fd4b2b87034e27e877478315a4ae9e541b205ec4c79eb52b98cdca61facf1da5"}
Sep 15 21:15:05.468089 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:05.468022    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd" podStartSLOduration=1.943514681 podStartE2EDuration="3.468004655s" totalImagesPullingTime="1.524489974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:15:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:15:05.465352028 +0000 UTC m=+1023.955997760" watchObservedRunningTime="2026-09-15 21:15:05.468004655 +0000 UTC m=+1023.958650388"
Sep 15 21:15:05.599994 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:05.599915    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dca193-29825115-zkx9w"]
Sep 15 21:15:06.513270 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:06.513243    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dca193-29825115-zkx9w"
Sep 15 21:15:06.571015 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:06.570992    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-544890-29825115-bzz5t"
Sep 15 21:15:06.575415 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:06.575386    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0df719-29825115-hhfbp"
Sep 15 21:15:07.376558 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:07.376463    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dca193-29825115-zkx9w"
Sep 15 21:15:07.376558 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:07.376480    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dca193-29825115-zkx9w" event={"ID":"80dfa469-11dc-4819-ab15-22b33ec7d8cb","Type":"ContainerDied","Data":"04cd2190c19566e64597d38ad843005c571bea7507472def38e8b0d8e6ee5df4"}
Sep 15 21:15:07.376558 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:07.376527    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04cd2190c19566e64597d38ad843005c571bea7507472def38e8b0d8e6ee5df4"
Sep 15 21:15:07.377936 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:07.377915    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0df719-29825115-hhfbp"
Sep 15 21:15:07.377936 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:07.377922    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0df719-29825115-hhfbp" event={"ID":"82838560-39b4-4124-8865-54277ea10b52","Type":"ContainerDied","Data":"a69826f5ba85a6e7e6e247d6d4d07f124029043fe859c274fff24f05009ae4e1"}
Sep 15 21:15:07.378133 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:07.377962    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a69826f5ba85a6e7e6e247d6d4d07f124029043fe859c274fff24f05009ae4e1"
Sep 15 21:15:07.382820 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:07.382795    2558 generic.go:356] "Generic (PLEG): container finished" podID="ceb37284-d74d-460d-810c-d7128bb995f5" containerID="9c009600e4ccc2956a59308e73bf8bcae1a4719741d83b99ff3fff12a1ecac09" exitCode=0
Sep 15 21:15:07.382949 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:07.382827    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5" event={"ID":"ceb37284-d74d-460d-810c-d7128bb995f5","Type":"ContainerDied","Data":"9c009600e4ccc2956a59308e73bf8bcae1a4719741d83b99ff3fff12a1ecac09"}
Sep 15 21:15:07.383271 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:07.383255    2558 scope.go:117] "RemoveContainer" containerID="9c009600e4ccc2956a59308e73bf8bcae1a4719741d83b99ff3fff12a1ecac09"
Sep 15 21:15:07.384711 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:07.384671    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:15:07.384801 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:07.384750    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-544890-29825115-bzz5t" event={"ID":"1cd559e9-c604-4256-97e1-805f47eac94f","Type":"ContainerDied","Data":"dd72d28921b9dacc751c1992ef4fb5aa7de1f1c6bc40fa799d433fc11bed00c8"}
Sep 15 21:15:07.384801 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:07.384760    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:15:07.384801 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:07.384782    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd72d28921b9dacc751c1992ef4fb5aa7de1f1c6bc40fa799d433fc11bed00c8"
Sep 15 21:15:07.384801 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:07.384787    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-544890-29825115-bzz5t"
Sep 15 21:15:07.385038 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:07.384810    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:15:07.451140 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:07.451113    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dca193-29825115-zkx9w"]
Sep 15 21:15:07.456954 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:07.456930    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dca193-29825115-zkx9w"]
Sep 15 21:15:07.944162 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:07.944134    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80dfa469-11dc-4819-ab15-22b33ec7d8cb" path="/var/lib/kubelet/pods/80dfa469-11dc-4819-ab15-22b33ec7d8cb/volumes"
Sep 15 21:15:08.157920 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:08.157893    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:15:08.393316 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:08.393230    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5" event={"ID":"ceb37284-d74d-460d-810c-d7128bb995f5","Type":"ContainerStarted","Data":"3d1bd5ef60fec938771bcb6959bd273d07a1a9f2ecdb8ce99d60eddb3f5b6745"}
Sep 15 21:15:08.475872 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:08.475840    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:15:08.476976 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:08.476941    2558 patch_prober.go:28] interesting pod/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 15 21:15:08.477110 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:08.476991    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5" podUID="ceb37284-d74d-460d-810c-d7128bb995f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 15 21:15:09.476122 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:09.476080    2558 patch_prober.go:28] interesting pod/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 15 21:15:09.476512 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:09.476152    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5" podUID="ceb37284-d74d-460d-810c-d7128bb995f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 15 21:15:09.498408 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:09.494994    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:15:10.475546 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:10.475514    2558 patch_prober.go:28] interesting pod/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 15 21:15:10.475722 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:10.475571    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5" podUID="ceb37284-d74d-460d-810c-d7128bb995f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 15 21:15:11.374807 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:11.374779    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"
Sep 15 21:15:11.475492 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:11.475454    2558 patch_prober.go:28] interesting pod/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 15 21:15:11.475668 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:11.475509    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5" podUID="ceb37284-d74d-460d-810c-d7128bb995f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 15 21:15:12.476120 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:12.476082    2558 patch_prober.go:28] interesting pod/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 15 21:15:12.476740 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:12.476696    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5" podUID="ceb37284-d74d-460d-810c-d7128bb995f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 15 21:15:13.476209 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:13.476164    2558 patch_prober.go:28] interesting pod/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 15 21:15:13.476709 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:13.476237    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5" podUID="ceb37284-d74d-460d-810c-d7128bb995f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 15 21:15:13.911600 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:13.911507    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6" podUID="f5c14493-efa5-435f-b1bb-c5fd7491f62e" containerName="main" containerID="cri-o://aff9b49f4d7862b275da127a6628fb1f6eae5b532436cfd01ed6490b5ea3774e" gracePeriod=45
Sep 15 21:15:14.303363 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.303308    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:15:14.309252 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.309234    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5c14493-efa5-435f-b1bb-c5fd7491f62e-home\") pod \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\" (UID: \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\") "
Sep 15 21:15:14.309316 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.309298    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5c14493-efa5-435f-b1bb-c5fd7491f62e-dshm\") pod \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\" (UID: \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\") "
Sep 15 21:15:14.309362 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.309317    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5c14493-efa5-435f-b1bb-c5fd7491f62e-model-cache\") pod \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\" (UID: \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\") "
Sep 15 21:15:14.309362 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.309342    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-297br\" (UniqueName: \"kubernetes.io/projected/f5c14493-efa5-435f-b1bb-c5fd7491f62e-kube-api-access-297br\") pod \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\" (UID: \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\") "
Sep 15 21:15:14.309446 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.309378    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5c14493-efa5-435f-b1bb-c5fd7491f62e-tls-certs\") pod \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\" (UID: \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\") "
Sep 15 21:15:14.309494 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.309447    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5c14493-efa5-435f-b1bb-c5fd7491f62e-tmp-dir\") pod \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\" (UID: \"f5c14493-efa5-435f-b1bb-c5fd7491f62e\") "
Sep 15 21:15:14.309550 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.309502    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c14493-efa5-435f-b1bb-c5fd7491f62e-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5c14493-efa5-435f-b1bb-c5fd7491f62e" (UID: "f5c14493-efa5-435f-b1bb-c5fd7491f62e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:15:14.309550 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.309509    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c14493-efa5-435f-b1bb-c5fd7491f62e-home" (OuterVolumeSpecName: "home") pod "f5c14493-efa5-435f-b1bb-c5fd7491f62e" (UID: "f5c14493-efa5-435f-b1bb-c5fd7491f62e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:15:14.309777 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.309759    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5c14493-efa5-435f-b1bb-c5fd7491f62e-home\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:15:14.309842 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.309779    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5c14493-efa5-435f-b1bb-c5fd7491f62e-model-cache\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:15:14.309842 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.309783    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c14493-efa5-435f-b1bb-c5fd7491f62e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5c14493-efa5-435f-b1bb-c5fd7491f62e" (UID: "f5c14493-efa5-435f-b1bb-c5fd7491f62e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:15:14.311450 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.311426    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c14493-efa5-435f-b1bb-c5fd7491f62e-dshm" (OuterVolumeSpecName: "dshm") pod "f5c14493-efa5-435f-b1bb-c5fd7491f62e" (UID: "f5c14493-efa5-435f-b1bb-c5fd7491f62e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:15:14.311592 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.311576    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c14493-efa5-435f-b1bb-c5fd7491f62e-kube-api-access-297br" (OuterVolumeSpecName: "kube-api-access-297br") pod "f5c14493-efa5-435f-b1bb-c5fd7491f62e" (UID: "f5c14493-efa5-435f-b1bb-c5fd7491f62e"). InnerVolumeSpecName "kube-api-access-297br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:15:14.311733 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.311710    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c14493-efa5-435f-b1bb-c5fd7491f62e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5c14493-efa5-435f-b1bb-c5fd7491f62e" (UID: "f5c14493-efa5-435f-b1bb-c5fd7491f62e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:15:14.410760 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.410724    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-297br\" (UniqueName: \"kubernetes.io/projected/f5c14493-efa5-435f-b1bb-c5fd7491f62e-kube-api-access-297br\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:15:14.410760 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.410756    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5c14493-efa5-435f-b1bb-c5fd7491f62e-tls-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:15:14.410760 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.410766    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5c14493-efa5-435f-b1bb-c5fd7491f62e-tmp-dir\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:15:14.411129 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.410774    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5c14493-efa5-435f-b1bb-c5fd7491f62e-dshm\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:15:14.431954 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.431921    2558 generic.go:356] "Generic (PLEG): container finished" podID="f5c14493-efa5-435f-b1bb-c5fd7491f62e" containerID="aff9b49f4d7862b275da127a6628fb1f6eae5b532436cfd01ed6490b5ea3774e" exitCode=0
Sep 15 21:15:14.432158 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.431973    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"
Sep 15 21:15:14.432158 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.432006    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6" event={"ID":"f5c14493-efa5-435f-b1bb-c5fd7491f62e","Type":"ContainerDied","Data":"aff9b49f4d7862b275da127a6628fb1f6eae5b532436cfd01ed6490b5ea3774e"}
Sep 15 21:15:14.432158 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.432061    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6" event={"ID":"f5c14493-efa5-435f-b1bb-c5fd7491f62e","Type":"ContainerDied","Data":"9c67fe84b0a169eaeac62e1343aa333bdda11544783c802df656e23740185b08"}
Sep 15 21:15:14.432158 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.432085    2558 scope.go:117] "RemoveContainer" containerID="aff9b49f4d7862b275da127a6628fb1f6eae5b532436cfd01ed6490b5ea3774e"
Sep 15 21:15:14.441990 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.441974    2558 scope.go:117] "RemoveContainer" containerID="aff9b49f4d7862b275da127a6628fb1f6eae5b532436cfd01ed6490b5ea3774e"
Sep 15 21:15:14.458297 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:15:14.458253    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aff9b49f4d7862b275da127a6628fb1f6eae5b532436cfd01ed6490b5ea3774e\": container with ID starting with aff9b49f4d7862b275da127a6628fb1f6eae5b532436cfd01ed6490b5ea3774e not found: ID does not exist" containerID="aff9b49f4d7862b275da127a6628fb1f6eae5b532436cfd01ed6490b5ea3774e"
Sep 15 21:15:14.458446 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.458308    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff9b49f4d7862b275da127a6628fb1f6eae5b532436cfd01ed6490b5ea3774e"} err="failed to get container status \"aff9b49f4d7862b275da127a6628fb1f6eae5b532436cfd01ed6490b5ea3774e\": rpc error: code = NotFound desc = could not find container \"aff9b49f4d7862b275da127a6628fb1f6eae5b532436cfd01ed6490b5ea3774e\": container with ID starting with aff9b49f4d7862b275da127a6628fb1f6eae5b532436cfd01ed6490b5ea3774e not found: ID does not exist"
Sep 15 21:15:14.478249 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.478211    2558 patch_prober.go:28] interesting pod/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 15 21:15:14.478736 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.478284    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5" podUID="ceb37284-d74d-460d-810c-d7128bb995f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 15 21:15:14.478736 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.478665    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"]
Sep 15 21:15:14.482350 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:14.482323    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-dca193/simulator-w4-dca193-kserve-7db95ff868-rb6s6"]
Sep 15 21:15:15.480005 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:15.479975    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:15:15.480474 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:15.480286    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:15:15.481070 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:15.481030    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:15:15.944937 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:15.944902    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5c14493-efa5-435f-b1bb-c5fd7491f62e" path="/var/lib/kubelet/pods/f5c14493-efa5-435f-b1bb-c5fd7491f62e/volumes"
Sep 15 21:15:20.472116 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:20.472084    2558 generic.go:356] "Generic (PLEG): container finished" podID="4dede905-5520-4367-a6d0-dda15d3f0f59" containerID="0f60f88f25225b2f3f1221b1c5cdbedf239a05d3e827ee83055da27c7273bf15" exitCode=0
Sep 15 21:15:20.472483 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:20.472160    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt" event={"ID":"4dede905-5520-4367-a6d0-dda15d3f0f59","Type":"ContainerDied","Data":"0f60f88f25225b2f3f1221b1c5cdbedf239a05d3e827ee83055da27c7273bf15"}
Sep 15 21:15:20.472542 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:20.472525    2558 scope.go:117] "RemoveContainer" containerID="0f60f88f25225b2f3f1221b1c5cdbedf239a05d3e827ee83055da27c7273bf15"
Sep 15 21:15:20.473590 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:20.473563    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:15:20.473665 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:20.473624    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:15:20.473665 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:20.473654    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:15:21.482528 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:21.482496    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt" event={"ID":"4dede905-5520-4367-a6d0-dda15d3f0f59","Type":"ContainerStarted","Data":"bafeb35bf2bfc3272c63cd61f6aee69529f5ab434189062c1858d16aca3057e7"}
Sep 15 21:15:22.158195 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:22.157765    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:15:22.158402 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:22.158235    2558 patch_prober.go:28] interesting pod/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 15 21:15:22.158402 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:22.158298    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt" podUID="4dede905-5520-4367-a6d0-dda15d3f0f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 15 21:15:23.158490 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:23.158445    2558 patch_prober.go:28] interesting pod/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 15 21:15:23.159026 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:23.158529    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt" podUID="4dede905-5520-4367-a6d0-dda15d3f0f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 15 21:15:23.701526 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:23.701486    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"]
Sep 15 21:15:23.701955 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:23.701919    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v" podUID="84146531-3e8c-4d16-8ce5-ecb596327477" containerName="maas-api" containerID="cri-o://2b78bc069ebb09096c9b797a74c7d535aee8cc99310f3dac097b5ca6a54f5648" gracePeriod=30
Sep 15 21:15:23.984663 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:23.984638    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"
Sep 15 21:15:24.106473 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:24.106436    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84146531-3e8c-4d16-8ce5-ecb596327477-maas-api-tls\") pod \"84146531-3e8c-4d16-8ce5-ecb596327477\" (UID: \"84146531-3e8c-4d16-8ce5-ecb596327477\") "
Sep 15 21:15:24.106666 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:24.106496    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84146531-3e8c-4d16-8ce5-ecb596327477-metrics-certs\") pod \"84146531-3e8c-4d16-8ce5-ecb596327477\" (UID: \"84146531-3e8c-4d16-8ce5-ecb596327477\") "
Sep 15 21:15:24.106666 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:24.106533    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84146531-3e8c-4d16-8ce5-ecb596327477-openshift-service-ca\") pod \"84146531-3e8c-4d16-8ce5-ecb596327477\" (UID: \"84146531-3e8c-4d16-8ce5-ecb596327477\") "
Sep 15 21:15:24.106804 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:24.106708    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8t88d\" (UniqueName: \"kubernetes.io/projected/84146531-3e8c-4d16-8ce5-ecb596327477-kube-api-access-8t88d\") pod \"84146531-3e8c-4d16-8ce5-ecb596327477\" (UID: \"84146531-3e8c-4d16-8ce5-ecb596327477\") "
Sep 15 21:15:24.107083 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:24.106938    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84146531-3e8c-4d16-8ce5-ecb596327477-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "84146531-3e8c-4d16-8ce5-ecb596327477" (UID: "84146531-3e8c-4d16-8ce5-ecb596327477"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:15:24.107083 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:24.107078    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84146531-3e8c-4d16-8ce5-ecb596327477-openshift-service-ca\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:15:24.109126 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:24.109089    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84146531-3e8c-4d16-8ce5-ecb596327477-kube-api-access-8t88d" (OuterVolumeSpecName: "kube-api-access-8t88d") pod "84146531-3e8c-4d16-8ce5-ecb596327477" (UID: "84146531-3e8c-4d16-8ce5-ecb596327477"). InnerVolumeSpecName "kube-api-access-8t88d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:15:24.109230 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:24.109169    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84146531-3e8c-4d16-8ce5-ecb596327477-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "84146531-3e8c-4d16-8ce5-ecb596327477" (UID: "84146531-3e8c-4d16-8ce5-ecb596327477"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:15:24.109363 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:24.109334    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84146531-3e8c-4d16-8ce5-ecb596327477-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "84146531-3e8c-4d16-8ce5-ecb596327477" (UID: "84146531-3e8c-4d16-8ce5-ecb596327477"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:15:24.161075 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:24.159470    2558 patch_prober.go:28] interesting pod/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 15 21:15:24.161075 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:24.159533    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt" podUID="4dede905-5520-4367-a6d0-dda15d3f0f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 15 21:15:24.207829 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:24.207796    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84146531-3e8c-4d16-8ce5-ecb596327477-metrics-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:15:24.207829 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:24.207822    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t88d\" (UniqueName: \"kubernetes.io/projected/84146531-3e8c-4d16-8ce5-ecb596327477-kube-api-access-8t88d\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:15:24.207829 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:24.207832    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84146531-3e8c-4d16-8ce5-ecb596327477-maas-api-tls\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:15:24.499970 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:24.499932    2558 generic.go:356] "Generic (PLEG): container finished" podID="84146531-3e8c-4d16-8ce5-ecb596327477" containerID="2b78bc069ebb09096c9b797a74c7d535aee8cc99310f3dac097b5ca6a54f5648" exitCode=0
Sep 15 21:15:24.500147 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:24.499983    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"
Sep 15 21:15:24.500147 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:24.499992    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v" event={"ID":"84146531-3e8c-4d16-8ce5-ecb596327477","Type":"ContainerDied","Data":"2b78bc069ebb09096c9b797a74c7d535aee8cc99310f3dac097b5ca6a54f5648"}
Sep 15 21:15:24.500147 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:24.500045    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v" event={"ID":"84146531-3e8c-4d16-8ce5-ecb596327477","Type":"ContainerDied","Data":"9a87220183aea7c14667e3d4fddffe68ec8737611b2d3163e23c5a9e5e2f4108"}
Sep 15 21:15:24.500147 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:24.500091    2558 scope.go:117] "RemoveContainer" containerID="2b78bc069ebb09096c9b797a74c7d535aee8cc99310f3dac097b5ca6a54f5648"
Sep 15 21:15:24.509764 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:24.509747    2558 scope.go:117] "RemoveContainer" containerID="2b78bc069ebb09096c9b797a74c7d535aee8cc99310f3dac097b5ca6a54f5648"
Sep 15 21:15:24.510031 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:15:24.510010    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b78bc069ebb09096c9b797a74c7d535aee8cc99310f3dac097b5ca6a54f5648\": container with ID starting with 2b78bc069ebb09096c9b797a74c7d535aee8cc99310f3dac097b5ca6a54f5648 not found: ID does not exist" containerID="2b78bc069ebb09096c9b797a74c7d535aee8cc99310f3dac097b5ca6a54f5648"
Sep 15 21:15:24.510113 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:24.510040    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b78bc069ebb09096c9b797a74c7d535aee8cc99310f3dac097b5ca6a54f5648"} err="failed to get container status \"2b78bc069ebb09096c9b797a74c7d535aee8cc99310f3dac097b5ca6a54f5648\": rpc error: code = NotFound desc = could not find container \"2b78bc069ebb09096c9b797a74c7d535aee8cc99310f3dac097b5ca6a54f5648\": container with ID starting with 2b78bc069ebb09096c9b797a74c7d535aee8cc99310f3dac097b5ca6a54f5648 not found: ID does not exist"
Sep 15 21:15:24.521330 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:24.521307    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"]
Sep 15 21:15:24.525086 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:24.525041    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b7100513-56d79b5c99-6js4v"]
Sep 15 21:15:25.158572 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:25.158526    2558 patch_prober.go:28] interesting pod/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 15 21:15:25.158788 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:25.158599    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt" podUID="4dede905-5520-4367-a6d0-dda15d3f0f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 15 21:15:25.943152 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:25.943109    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84146531-3e8c-4d16-8ce5-ecb596327477" path="/var/lib/kubelet/pods/84146531-3e8c-4d16-8ce5-ecb596327477/volumes"
Sep 15 21:15:26.157848 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:26.157804    2558 patch_prober.go:28] interesting pod/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 15 21:15:26.158044 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:26.157874    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt" podUID="4dede905-5520-4367-a6d0-dda15d3f0f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 15 21:15:27.162732 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:27.162703    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:15:27.163111 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:27.162948    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:15:27.163745 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:27.163730    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:15:30.331169 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:30.331138    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"]
Sep 15 21:15:30.331583 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:30.331333    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5" podUID="ceb37284-d74d-460d-810c-d7128bb995f5" containerName="istio-proxy" containerID="cri-o://3d1bd5ef60fec938771bcb6959bd273d07a1a9f2ecdb8ce99d60eddb3f5b6745" gracePeriod=30
Sep 15 21:15:30.482763 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:30.482730    2558 patch_prober.go:28] interesting pod/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 21:15:30.482926 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:30.482789    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5" podUID="ceb37284-d74d-460d-810c-d7128bb995f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 21:15:35.095809 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.095774    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:15:35.566130 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.566100    2558 generic.go:356] "Generic (PLEG): container finished" podID="ceb37284-d74d-460d-810c-d7128bb995f5" containerID="3d1bd5ef60fec938771bcb6959bd273d07a1a9f2ecdb8ce99d60eddb3f5b6745" exitCode=0
Sep 15 21:15:35.566217 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.566171    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5" event={"ID":"ceb37284-d74d-460d-810c-d7128bb995f5","Type":"ContainerDied","Data":"3d1bd5ef60fec938771bcb6959bd273d07a1a9f2ecdb8ce99d60eddb3f5b6745"}
Sep 15 21:15:35.566263 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.566251    2558 scope.go:117] "RemoveContainer" containerID="9c009600e4ccc2956a59308e73bf8bcae1a4719741d83b99ff3fff12a1ecac09"
Sep 15 21:15:35.588625 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.588606    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:15:35.718842 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.718803    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb37284-d74d-460d-810c-d7128bb995f5-istio-podinfo\") pod \"ceb37284-d74d-460d-810c-d7128bb995f5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") "
Sep 15 21:15:35.719025 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.718899    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-workload-certs\") pod \"ceb37284-d74d-460d-810c-d7128bb995f5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") "
Sep 15 21:15:35.719025 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.718934    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-istio-envoy\") pod \"ceb37284-d74d-460d-810c-d7128bb995f5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") "
Sep 15 21:15:35.719025 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.718989    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-workload-socket\") pod \"ceb37284-d74d-460d-810c-d7128bb995f5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") "
Sep 15 21:15:35.719025 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.719018    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-credential-socket\") pod \"ceb37284-d74d-460d-810c-d7128bb995f5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") "
Sep 15 21:15:35.719273 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.719075    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-istio-data\") pod \"ceb37284-d74d-460d-810c-d7128bb995f5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") "
Sep 15 21:15:35.719273 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.719109    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb37284-d74d-460d-810c-d7128bb995f5-istiod-ca-cert\") pod \"ceb37284-d74d-460d-810c-d7128bb995f5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") "
Sep 15 21:15:35.719273 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.719165    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wl7rz\" (UniqueName: \"kubernetes.io/projected/ceb37284-d74d-460d-810c-d7128bb995f5-kube-api-access-wl7rz\") pod \"ceb37284-d74d-460d-810c-d7128bb995f5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") "
Sep 15 21:15:35.719273 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.719191    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb37284-d74d-460d-810c-d7128bb995f5-istio-token\") pod \"ceb37284-d74d-460d-810c-d7128bb995f5\" (UID: \"ceb37284-d74d-460d-810c-d7128bb995f5\") "
Sep 15 21:15:35.719475 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.719280    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ceb37284-d74d-460d-810c-d7128bb995f5" (UID: "ceb37284-d74d-460d-810c-d7128bb995f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:15:35.719475 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.719402    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ceb37284-d74d-460d-810c-d7128bb995f5" (UID: "ceb37284-d74d-460d-810c-d7128bb995f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:15:35.719475 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.719433    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-istio-data" (OuterVolumeSpecName: "istio-data") pod "ceb37284-d74d-460d-810c-d7128bb995f5" (UID: "ceb37284-d74d-460d-810c-d7128bb995f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:15:35.719632 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.719485    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ceb37284-d74d-460d-810c-d7128bb995f5" (UID: "ceb37284-d74d-460d-810c-d7128bb995f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:15:35.719632 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.719579    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceb37284-d74d-460d-810c-d7128bb995f5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ceb37284-d74d-460d-810c-d7128bb995f5" (UID: "ceb37284-d74d-460d-810c-d7128bb995f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:15:35.719735 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.719640    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-credential-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:15:35.719735 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.719660    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-istio-data\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:15:35.719735 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.719673    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-workload-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:15:35.719735 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.719685    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-workload-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:15:35.721187 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.721163    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ceb37284-d74d-460d-810c-d7128bb995f5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ceb37284-d74d-460d-810c-d7128bb995f5" (UID: "ceb37284-d74d-460d-810c-d7128bb995f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:15:35.721347 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.721331    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ceb37284-d74d-460d-810c-d7128bb995f5" (UID: "ceb37284-d74d-460d-810c-d7128bb995f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:15:35.721347 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.721332    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb37284-d74d-460d-810c-d7128bb995f5-istio-token" (OuterVolumeSpecName: "istio-token") pod "ceb37284-d74d-460d-810c-d7128bb995f5" (UID: "ceb37284-d74d-460d-810c-d7128bb995f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:15:35.721519 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.721499    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb37284-d74d-460d-810c-d7128bb995f5-kube-api-access-wl7rz" (OuterVolumeSpecName: "kube-api-access-wl7rz") pod "ceb37284-d74d-460d-810c-d7128bb995f5" (UID: "ceb37284-d74d-460d-810c-d7128bb995f5"). InnerVolumeSpecName "kube-api-access-wl7rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:15:35.820840 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.820812    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb37284-d74d-460d-810c-d7128bb995f5-istio-envoy\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:15:35.820840 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.820838    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb37284-d74d-460d-810c-d7128bb995f5-istiod-ca-cert\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:15:35.820840 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.820847    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl7rz\" (UniqueName: \"kubernetes.io/projected/ceb37284-d74d-460d-810c-d7128bb995f5-kube-api-access-wl7rz\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:15:35.821084 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.820856    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb37284-d74d-460d-810c-d7128bb995f5-istio-token\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:15:35.821084 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:35.820866    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb37284-d74d-460d-810c-d7128bb995f5-istio-podinfo\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:15:36.572686 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:36.572652    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"
Sep 15 21:15:36.572686 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:36.572664    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5" event={"ID":"ceb37284-d74d-460d-810c-d7128bb995f5","Type":"ContainerDied","Data":"5e1a72e48b23a853fa5f578832d38bc95b886c007ea192d3d451fb127aff57bd"}
Sep 15 21:15:36.573263 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:36.572726    2558 scope.go:117] "RemoveContainer" containerID="3d1bd5ef60fec938771bcb6959bd273d07a1a9f2ecdb8ce99d60eddb3f5b6745"
Sep 15 21:15:36.608755 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:36.608731    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"]
Sep 15 21:15:36.615148 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:36.615119    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b7100513-openshift-default-786948c9f6-hqmq5"]
Sep 15 21:15:37.942010 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:37.941979    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceb37284-d74d-460d-810c-d7128bb995f5" path="/var/lib/kubelet/pods/ceb37284-d74d-460d-810c-d7128bb995f5/volumes"
Sep 15 21:15:45.142795 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.142757    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"]
Sep 15 21:15:45.143207 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.143125    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb37284-d74d-460d-810c-d7128bb995f5" containerName="istio-proxy"
Sep 15 21:15:45.143207 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.143137    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb37284-d74d-460d-810c-d7128bb995f5" containerName="istio-proxy"
Sep 15 21:15:45.143207 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.143150    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb37284-d74d-460d-810c-d7128bb995f5" containerName="istio-proxy"
Sep 15 21:15:45.143207 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.143155    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb37284-d74d-460d-810c-d7128bb995f5" containerName="istio-proxy"
Sep 15 21:15:45.143207 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.143164    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82838560-39b4-4124-8865-54277ea10b52" containerName="cleanup"
Sep 15 21:15:45.143207 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.143170    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="82838560-39b4-4124-8865-54277ea10b52" containerName="cleanup"
Sep 15 21:15:45.143207 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.143181    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80dfa469-11dc-4819-ab15-22b33ec7d8cb" containerName="cleanup"
Sep 15 21:15:45.143207 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.143186    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="80dfa469-11dc-4819-ab15-22b33ec7d8cb" containerName="cleanup"
Sep 15 21:15:45.143207 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.143197    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c14493-efa5-435f-b1bb-c5fd7491f62e" containerName="main"
Sep 15 21:15:45.143207 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.143202    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c14493-efa5-435f-b1bb-c5fd7491f62e" containerName="main"
Sep 15 21:15:45.143207 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.143210    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84146531-3e8c-4d16-8ce5-ecb596327477" containerName="maas-api"
Sep 15 21:15:45.143640 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.143216    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="84146531-3e8c-4d16-8ce5-ecb596327477" containerName="maas-api"
Sep 15 21:15:45.143640 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.143222    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cd559e9-c604-4256-97e1-805f47eac94f" containerName="cleanup"
Sep 15 21:15:45.143640 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.143227    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cd559e9-c604-4256-97e1-805f47eac94f" containerName="cleanup"
Sep 15 21:15:45.143640 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.143278    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceb37284-d74d-460d-810c-d7128bb995f5" containerName="istio-proxy"
Sep 15 21:15:45.143640 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.143284    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="84146531-3e8c-4d16-8ce5-ecb596327477" containerName="maas-api"
Sep 15 21:15:45.143640 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.143290    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="80dfa469-11dc-4819-ab15-22b33ec7d8cb" containerName="cleanup"
Sep 15 21:15:45.143640 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.143296    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cd559e9-c604-4256-97e1-805f47eac94f" containerName="cleanup"
Sep 15 21:15:45.143640 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.143302    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5c14493-efa5-435f-b1bb-c5fd7491f62e" containerName="main"
Sep 15 21:15:45.143640 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.143309    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="82838560-39b4-4124-8865-54277ea10b52" containerName="cleanup"
Sep 15 21:15:45.147694 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.147677    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.151032 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.151010    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-7c59a9-openshift-default-dockercfg-rnr8r\""
Sep 15 21:15:45.164933 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.164907    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"]
Sep 15 21:15:45.308837 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.308802    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-credential-socket\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.308837 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.308843    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23930493-5ede-47bc-a2b8-437761d1f50f-istio-token\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.309078 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.308863    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23930493-5ede-47bc-a2b8-437761d1f50f-istiod-ca-cert\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.309078 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.308938    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-workload-certs\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.309078 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.309006    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23930493-5ede-47bc-a2b8-437761d1f50f-istio-podinfo\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.309188 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.309078    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-workload-socket\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.309188 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.309119    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-istio-envoy\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.309188 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.309152    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtmm9\" (UniqueName: \"kubernetes.io/projected/23930493-5ede-47bc-a2b8-437761d1f50f-kube-api-access-dtmm9\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.309188 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.309182    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-istio-data\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.410017 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.409933    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-istio-data\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.410017 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.409977    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-credential-socket\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.410017 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.410004    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23930493-5ede-47bc-a2b8-437761d1f50f-istio-token\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.410311 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.410027    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23930493-5ede-47bc-a2b8-437761d1f50f-istiod-ca-cert\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.410311 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.410116    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-workload-certs\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.410311 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.410189    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23930493-5ede-47bc-a2b8-437761d1f50f-istio-podinfo\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.410311 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.410231    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-workload-socket\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.410311 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.410287    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-istio-envoy\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.410579 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.410316    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtmm9\" (UniqueName: \"kubernetes.io/projected/23930493-5ede-47bc-a2b8-437761d1f50f-kube-api-access-dtmm9\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.410579 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.410404    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-istio-data\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.410579 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.410435    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-credential-socket\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.410579 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.410495    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-workload-certs\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.410770 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.410732    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-workload-socket\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.411214 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.411195    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23930493-5ede-47bc-a2b8-437761d1f50f-istiod-ca-cert\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.412515 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.412494    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-istio-envoy\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.412834 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.412818    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23930493-5ede-47bc-a2b8-437761d1f50f-istio-podinfo\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.418536 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.418514    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtmm9\" (UniqueName: \"kubernetes.io/projected/23930493-5ede-47bc-a2b8-437761d1f50f-kube-api-access-dtmm9\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.418627 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.418599    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23930493-5ede-47bc-a2b8-437761d1f50f-istio-token\") pod \"e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") " pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.460102 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.460070    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:45.762243 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.762214    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"]
Sep 15 21:15:45.762397 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:15:45.762360    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23930493_5ede_47bc_a2b8_437761d1f50f.slice/crio-2c3dc35e914a9846f461bd086cd516fab5981a2ac71a8233dc7d528bc91d2201 WatchSource:0}: Error finding container 2c3dc35e914a9846f461bd086cd516fab5981a2ac71a8233dc7d528bc91d2201: Status 404 returned error can't find the container with id 2c3dc35e914a9846f461bd086cd516fab5981a2ac71a8233dc7d528bc91d2201
Sep 15 21:15:45.764828 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.764800    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:15:45.764885 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.764860    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:15:45.764930 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:45.764887    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:15:46.634990 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:46.634953    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp" event={"ID":"23930493-5ede-47bc-a2b8-437761d1f50f","Type":"ContainerStarted","Data":"bd9f5f07fd72c58e5e7461548376b0c5dd75d078423cc4521277837767173380"}
Sep 15 21:15:46.635390 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:46.635008    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp" event={"ID":"23930493-5ede-47bc-a2b8-437761d1f50f","Type":"ContainerStarted","Data":"2c3dc35e914a9846f461bd086cd516fab5981a2ac71a8233dc7d528bc91d2201"}
Sep 15 21:15:46.661234 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:46.661185    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp" podStartSLOduration=1.6611659410000001 podStartE2EDuration="1.661165941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:15:46.659609385 +0000 UTC m=+1065.150255117" watchObservedRunningTime="2026-09-15 21:15:46.661165941 +0000 UTC m=+1065.151811739"
Sep 15 21:15:47.460278 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:47.460244    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:47.465245 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:47.465221    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:47.640847 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:47.640813    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:47.641686 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:47.641669    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:15:52.294116 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:52.294080    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75"]
Sep 15 21:15:52.294719 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:52.294701    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceb37284-d74d-460d-810c-d7128bb995f5" containerName="istio-proxy"
Sep 15 21:15:52.298195 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:52.298172    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75"
Sep 15 21:15:52.300830 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:52.300811    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-7c59a9-serving-cert\""
Sep 15 21:15:52.309182 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:52.309157    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75"]
Sep 15 21:15:52.375598 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:52.375569    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75\" (UID: \"20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75"
Sep 15 21:15:52.375726 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:52.375617    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75\" (UID: \"20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75"
Sep 15 21:15:52.375726 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:52.375670    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75\" (UID: \"20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75"
Sep 15 21:15:52.375809 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:52.375744    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwm9t\" (UniqueName: \"kubernetes.io/projected/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b-kube-api-access-lwm9t\") pod \"maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75\" (UID: \"20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75"
Sep 15 21:15:52.477046 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:52.477014    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwm9t\" (UniqueName: \"kubernetes.io/projected/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b-kube-api-access-lwm9t\") pod \"maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75\" (UID: \"20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75"
Sep 15 21:15:52.477238 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:52.477150    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75\" (UID: \"20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75"
Sep 15 21:15:52.477238 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:52.477202    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75\" (UID: \"20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75"
Sep 15 21:15:52.477327 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:52.477234    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75\" (UID: \"20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75"
Sep 15 21:15:52.477859 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:52.477828    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75\" (UID: \"20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75"
Sep 15 21:15:52.479717 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:52.479690    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75\" (UID: \"20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75"
Sep 15 21:15:52.479810 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:52.479735    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75\" (UID: \"20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75"
Sep 15 21:15:52.485323 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:52.485303    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwm9t\" (UniqueName: \"kubernetes.io/projected/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b-kube-api-access-lwm9t\") pod \"maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75\" (UID: \"20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75"
Sep 15 21:15:52.611279 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:52.611211    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75"
Sep 15 21:15:52.890632 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:52.890603    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75"]
Sep 15 21:15:52.891318 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:15:52.891295    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ecac1d_85e4_45b3_844c_2ccb5cc5ed4b.slice/crio-4f20979867f16ef7ebb618d01aa206d30a3b306751bc034e1256adad05800e70 WatchSource:0}: Error finding container 4f20979867f16ef7ebb618d01aa206d30a3b306751bc034e1256adad05800e70: Status 404 returned error can't find the container with id 4f20979867f16ef7ebb618d01aa206d30a3b306751bc034e1256adad05800e70
Sep 15 21:15:53.580758 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.580679    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"]
Sep 15 21:15:53.584131 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.584114    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"
Sep 15 21:15:53.588678 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.588653    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-8f08a9\"/\"default-dockercfg-452hv\""
Sep 15 21:15:53.588814 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.588663    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-8f08a9\"/\"explicit-model-8f08a9-kserve-self-signed-certs\""
Sep 15 21:15:53.588814 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.588726    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-8f08a9\"/\"kube-root-ca.crt\""
Sep 15 21:15:53.588814 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.588663    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-8f08a9\"/\"openshift-service-ca.crt\""
Sep 15 21:15:53.596812 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.596790    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"]
Sep 15 21:15:53.680389 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.680361    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75" event={"ID":"20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b","Type":"ContainerStarted","Data":"43262ec2e8ed54c3adbb9523704f399476c6e4ba0da3c156859e1df1c7bd3142"}
Sep 15 21:15:53.680558 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.680411    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75" event={"ID":"20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b","Type":"ContainerStarted","Data":"4f20979867f16ef7ebb618d01aa206d30a3b306751bc034e1256adad05800e70"}
Sep 15 21:15:53.680558 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.680445    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75"
Sep 15 21:15:53.701235 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.701190    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75" podStartSLOduration=1.3061177019999999 podStartE2EDuration="1.701176415s" totalImagesPullingTime="395.058713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:15:53.698740397 +0000 UTC m=+1072.189386130" watchObservedRunningTime="2026-09-15 21:15:53.701176415 +0000 UTC m=+1072.191822205"
Sep 15 21:15:53.702155 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.702137    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9049ff1f-34ce-4639-be80-14ebff380c3a-dshm\") pod \"explicit-model-8f08a9-kserve-575b87c5d9-cwqs9\" (UID: \"9049ff1f-34ce-4639-be80-14ebff380c3a\") " pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"
Sep 15 21:15:53.702211 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.702187    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9049ff1f-34ce-4639-be80-14ebff380c3a-home\") pod \"explicit-model-8f08a9-kserve-575b87c5d9-cwqs9\" (UID: \"9049ff1f-34ce-4639-be80-14ebff380c3a\") " pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"
Sep 15 21:15:53.702258 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.702209    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9049ff1f-34ce-4639-be80-14ebff380c3a-tls-certs\") pod \"explicit-model-8f08a9-kserve-575b87c5d9-cwqs9\" (UID: \"9049ff1f-34ce-4639-be80-14ebff380c3a\") " pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"
Sep 15 21:15:53.702258 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.702242    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9049ff1f-34ce-4639-be80-14ebff380c3a-tmp-dir\") pod \"explicit-model-8f08a9-kserve-575b87c5d9-cwqs9\" (UID: \"9049ff1f-34ce-4639-be80-14ebff380c3a\") " pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"
Sep 15 21:15:53.702325 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.702270    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9049ff1f-34ce-4639-be80-14ebff380c3a-model-cache\") pod \"explicit-model-8f08a9-kserve-575b87c5d9-cwqs9\" (UID: \"9049ff1f-34ce-4639-be80-14ebff380c3a\") " pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"
Sep 15 21:15:53.702358 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.702343    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwj76\" (UniqueName: \"kubernetes.io/projected/9049ff1f-34ce-4639-be80-14ebff380c3a-kube-api-access-wwj76\") pod \"explicit-model-8f08a9-kserve-575b87c5d9-cwqs9\" (UID: \"9049ff1f-34ce-4639-be80-14ebff380c3a\") " pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"
Sep 15 21:15:53.803074 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.803018    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwj76\" (UniqueName: \"kubernetes.io/projected/9049ff1f-34ce-4639-be80-14ebff380c3a-kube-api-access-wwj76\") pod \"explicit-model-8f08a9-kserve-575b87c5d9-cwqs9\" (UID: \"9049ff1f-34ce-4639-be80-14ebff380c3a\") " pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"
Sep 15 21:15:53.803246 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.803103    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9049ff1f-34ce-4639-be80-14ebff380c3a-dshm\") pod \"explicit-model-8f08a9-kserve-575b87c5d9-cwqs9\" (UID: \"9049ff1f-34ce-4639-be80-14ebff380c3a\") " pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"
Sep 15 21:15:53.803246 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.803171    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9049ff1f-34ce-4639-be80-14ebff380c3a-home\") pod \"explicit-model-8f08a9-kserve-575b87c5d9-cwqs9\" (UID: \"9049ff1f-34ce-4639-be80-14ebff380c3a\") " pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"
Sep 15 21:15:53.803246 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.803192    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9049ff1f-34ce-4639-be80-14ebff380c3a-tls-certs\") pod \"explicit-model-8f08a9-kserve-575b87c5d9-cwqs9\" (UID: \"9049ff1f-34ce-4639-be80-14ebff380c3a\") " pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"
Sep 15 21:15:53.803422 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.803242    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9049ff1f-34ce-4639-be80-14ebff380c3a-tmp-dir\") pod \"explicit-model-8f08a9-kserve-575b87c5d9-cwqs9\" (UID: \"9049ff1f-34ce-4639-be80-14ebff380c3a\") " pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"
Sep 15 21:15:53.803422 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.803306    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9049ff1f-34ce-4639-be80-14ebff380c3a-model-cache\") pod \"explicit-model-8f08a9-kserve-575b87c5d9-cwqs9\" (UID: \"9049ff1f-34ce-4639-be80-14ebff380c3a\") " pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"
Sep 15 21:15:53.803658 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.803635    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9049ff1f-34ce-4639-be80-14ebff380c3a-home\") pod \"explicit-model-8f08a9-kserve-575b87c5d9-cwqs9\" (UID: \"9049ff1f-34ce-4639-be80-14ebff380c3a\") " pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"
Sep 15 21:15:53.803712 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.803681    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9049ff1f-34ce-4639-be80-14ebff380c3a-model-cache\") pod \"explicit-model-8f08a9-kserve-575b87c5d9-cwqs9\" (UID: \"9049ff1f-34ce-4639-be80-14ebff380c3a\") " pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"
Sep 15 21:15:53.803756 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.803705    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9049ff1f-34ce-4639-be80-14ebff380c3a-tmp-dir\") pod \"explicit-model-8f08a9-kserve-575b87c5d9-cwqs9\" (UID: \"9049ff1f-34ce-4639-be80-14ebff380c3a\") " pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"
Sep 15 21:15:53.805374 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.805352    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9049ff1f-34ce-4639-be80-14ebff380c3a-dshm\") pod \"explicit-model-8f08a9-kserve-575b87c5d9-cwqs9\" (UID: \"9049ff1f-34ce-4639-be80-14ebff380c3a\") " pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"
Sep 15 21:15:53.805747 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.805727    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9049ff1f-34ce-4639-be80-14ebff380c3a-tls-certs\") pod \"explicit-model-8f08a9-kserve-575b87c5d9-cwqs9\" (UID: \"9049ff1f-34ce-4639-be80-14ebff380c3a\") " pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"
Sep 15 21:15:53.812199 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.812172    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwj76\" (UniqueName: \"kubernetes.io/projected/9049ff1f-34ce-4639-be80-14ebff380c3a-kube-api-access-wwj76\") pod \"explicit-model-8f08a9-kserve-575b87c5d9-cwqs9\" (UID: \"9049ff1f-34ce-4639-be80-14ebff380c3a\") " pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"
Sep 15 21:15:53.893839 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:53.893772    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"
Sep 15 21:15:54.168200 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:54.168176    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"]
Sep 15 21:15:54.169300 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:15:54.169263    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9049ff1f_34ce_4639_be80_14ebff380c3a.slice/crio-9edf25a81756e1c95813a95ce1140192625e6195207a06398b343069557f8091 WatchSource:0}: Error finding container 9edf25a81756e1c95813a95ce1140192625e6195207a06398b343069557f8091: Status 404 returned error can't find the container with id 9edf25a81756e1c95813a95ce1140192625e6195207a06398b343069557f8091
Sep 15 21:15:54.689383 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:54.689338    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9" event={"ID":"9049ff1f-34ce-4639-be80-14ebff380c3a","Type":"ContainerStarted","Data":"260282ddb79fbad0c8851cfd75af2d39a2ddbf89925657a4fb34dd5fc95ae6a7"}
Sep 15 21:15:54.689783 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:54.689394    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9" event={"ID":"9049ff1f-34ce-4639-be80-14ebff380c3a","Type":"ContainerStarted","Data":"9edf25a81756e1c95813a95ce1140192625e6195207a06398b343069557f8091"}
Sep 15 21:15:54.718275 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:54.718226    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9" podStartSLOduration=1.718211627 podStartE2EDuration="1.718211627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:15:54.716438336 +0000 UTC m=+1073.207084068" watchObservedRunningTime="2026-09-15 21:15:54.718211627 +0000 UTC m=+1073.208857358"
Sep 15 21:15:59.695673 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:15:59.695647    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75"
Sep 15 21:16:01.465428 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:01.465389    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"]
Sep 15 21:16:02.293353 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:02.293310    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:16:03.894180 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:03.894129    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"
Sep 15 21:16:12.812256 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:12.812222    2558 generic.go:356] "Generic (PLEG): container finished" podID="23930493-5ede-47bc-a2b8-437761d1f50f" containerID="bd9f5f07fd72c58e5e7461548376b0c5dd75d078423cc4521277837767173380" exitCode=0
Sep 15 21:16:12.812669 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:12.812293    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp" event={"ID":"23930493-5ede-47bc-a2b8-437761d1f50f","Type":"ContainerDied","Data":"bd9f5f07fd72c58e5e7461548376b0c5dd75d078423cc4521277837767173380"}
Sep 15 21:16:12.812711 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:12.812668    2558 scope.go:117] "RemoveContainer" containerID="bd9f5f07fd72c58e5e7461548376b0c5dd75d078423cc4521277837767173380"
Sep 15 21:16:12.813853 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:12.813829    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:16:12.813911 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:12.813895    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:16:12.813948 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:12.813925    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:16:12.952178 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:12.952145    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"]
Sep 15 21:16:12.952492 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:12.952462    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd" podUID="8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b" containerName="maas-api" containerID="cri-o://1b2c6683162695649de7500bfaf194f9e6021e22fd7314f7743433dce643fa87" gracePeriod=30
Sep 15 21:16:13.222137 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:13.222108    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"
Sep 15 21:16:13.299098 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:13.299029    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-metrics-certs\") pod \"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b\" (UID: \"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b\") "
Sep 15 21:16:13.299098 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:13.299088    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-openshift-service-ca\") pod \"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b\" (UID: \"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b\") "
Sep 15 21:16:13.299361 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:13.299118    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9jhw\" (UniqueName: \"kubernetes.io/projected/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-kube-api-access-w9jhw\") pod \"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b\" (UID: \"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b\") "
Sep 15 21:16:13.299361 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:13.299139    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-maas-api-tls\") pod \"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b\" (UID: \"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b\") "
Sep 15 21:16:13.299501 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:13.299473    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b" (UID: "8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:16:13.301338 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:13.301312    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b" (UID: "8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:16:13.301465 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:13.301316    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-kube-api-access-w9jhw" (OuterVolumeSpecName: "kube-api-access-w9jhw") pod "8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b" (UID: "8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b"). InnerVolumeSpecName "kube-api-access-w9jhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:16:13.301465 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:13.301339    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b" (UID: "8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:16:13.400695 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:13.400604    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-metrics-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:16:13.400695 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:13.400633    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-openshift-service-ca\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:16:13.400695 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:13.400644    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9jhw\" (UniqueName: \"kubernetes.io/projected/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-kube-api-access-w9jhw\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:16:13.400695 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:13.400654    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b-maas-api-tls\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:16:13.819330 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:13.819294    2558 generic.go:356] "Generic (PLEG): container finished" podID="8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b" containerID="1b2c6683162695649de7500bfaf194f9e6021e22fd7314f7743433dce643fa87" exitCode=0
Sep 15 21:16:13.819797 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:13.819340    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"
Sep 15 21:16:13.819797 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:13.819380    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd" event={"ID":"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b","Type":"ContainerDied","Data":"1b2c6683162695649de7500bfaf194f9e6021e22fd7314f7743433dce643fa87"}
Sep 15 21:16:13.819797 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:13.819432    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd" event={"ID":"8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b","Type":"ContainerDied","Data":"b3f4410ef4ca7a867ffad2bdcdd44f559357658529d85029677090c17ee8bbb5"}
Sep 15 21:16:13.819797 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:13.819464    2558 scope.go:117] "RemoveContainer" containerID="1b2c6683162695649de7500bfaf194f9e6021e22fd7314f7743433dce643fa87"
Sep 15 21:16:13.821721 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:13.821697    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp" event={"ID":"23930493-5ede-47bc-a2b8-437761d1f50f","Type":"ContainerStarted","Data":"99e6351ee15eb1b7f49dedfc181b1241bc5ddcd520e435250abea6a1555c1c2f"}
Sep 15 21:16:13.829880 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:13.829864    2558 scope.go:117] "RemoveContainer" containerID="1b2c6683162695649de7500bfaf194f9e6021e22fd7314f7743433dce643fa87"
Sep 15 21:16:13.830157 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:16:13.830136    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b2c6683162695649de7500bfaf194f9e6021e22fd7314f7743433dce643fa87\": container with ID starting with 1b2c6683162695649de7500bfaf194f9e6021e22fd7314f7743433dce643fa87 not found: ID does not exist" containerID="1b2c6683162695649de7500bfaf194f9e6021e22fd7314f7743433dce643fa87"
Sep 15 21:16:13.830222 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:13.830166    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b2c6683162695649de7500bfaf194f9e6021e22fd7314f7743433dce643fa87"} err="failed to get container status \"1b2c6683162695649de7500bfaf194f9e6021e22fd7314f7743433dce643fa87\": rpc error: code = NotFound desc = could not find container \"1b2c6683162695649de7500bfaf194f9e6021e22fd7314f7743433dce643fa87\": container with ID starting with 1b2c6683162695649de7500bfaf194f9e6021e22fd7314f7743433dce643fa87 not found: ID does not exist"
Sep 15 21:16:13.869741 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:13.869686    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"]
Sep 15 21:16:13.872670 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:13.872631    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8f08a9-775f7d546f-ct7gd"]
Sep 15 21:16:13.942978 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:13.942934    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b" path="/var/lib/kubelet/pods/8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b/volumes"
Sep 15 21:16:14.461689 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:14.461658    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:16:14.462068 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:14.462024    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 15 21:16:14.462239 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:14.462211    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp" podUID="23930493-5ede-47bc-a2b8-437761d1f50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 15 21:16:15.461091 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:15.461029    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 15 21:16:15.461664 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:15.461635    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp" podUID="23930493-5ede-47bc-a2b8-437761d1f50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 15 21:16:16.461424 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.461374    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 15 21:16:16.461916 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.461450    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp" podUID="23930493-5ede-47bc-a2b8-437761d1f50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 15 21:16:16.487783 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.487745    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9" podUID="9049ff1f-34ce-4639-be80-14ebff380c3a" containerName="main" containerID="cri-o://260282ddb79fbad0c8851cfd75af2d39a2ddbf89925657a4fb34dd5fc95ae6a7" gracePeriod=45
Sep 15 21:16:16.571190 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.571135    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf"]
Sep 15 21:16:16.571712 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.571692    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b" containerName="maas-api"
Sep 15 21:16:16.571843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.571717    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b" containerName="maas-api"
Sep 15 21:16:16.571843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.571826    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d13f43c-bdbc-4ff2-bb0c-e868ab2d737b" containerName="maas-api"
Sep 15 21:16:16.587477 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.587445    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf"
Sep 15 21:16:16.590656 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.590629    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf"]
Sep 15 21:16:16.592038 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.591973    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-2c7774-serving-cert\""
Sep 15 21:16:16.735814 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.735730    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzmdk\" (UniqueName: \"kubernetes.io/projected/113066b8-2eb4-4d91-9c53-9451dd6067c9-kube-api-access-bzmdk\") pod \"maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf\" (UID: \"113066b8-2eb4-4d91-9c53-9451dd6067c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf"
Sep 15 21:16:16.735814 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.735805    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/113066b8-2eb4-4d91-9c53-9451dd6067c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf\" (UID: \"113066b8-2eb4-4d91-9c53-9451dd6067c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf"
Sep 15 21:16:16.736010 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.735842    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/113066b8-2eb4-4d91-9c53-9451dd6067c9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf\" (UID: \"113066b8-2eb4-4d91-9c53-9451dd6067c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf"
Sep 15 21:16:16.736010 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.735885    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/113066b8-2eb4-4d91-9c53-9451dd6067c9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf\" (UID: \"113066b8-2eb4-4d91-9c53-9451dd6067c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf"
Sep 15 21:16:16.789428 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.789398    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"
Sep 15 21:16:16.837012 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.836979    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzmdk\" (UniqueName: \"kubernetes.io/projected/113066b8-2eb4-4d91-9c53-9451dd6067c9-kube-api-access-bzmdk\") pod \"maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf\" (UID: \"113066b8-2eb4-4d91-9c53-9451dd6067c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf"
Sep 15 21:16:16.837185 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.837037    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/113066b8-2eb4-4d91-9c53-9451dd6067c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf\" (UID: \"113066b8-2eb4-4d91-9c53-9451dd6067c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf"
Sep 15 21:16:16.837185 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.837102    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/113066b8-2eb4-4d91-9c53-9451dd6067c9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf\" (UID: \"113066b8-2eb4-4d91-9c53-9451dd6067c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf"
Sep 15 21:16:16.837185 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.837137    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/113066b8-2eb4-4d91-9c53-9451dd6067c9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf\" (UID: \"113066b8-2eb4-4d91-9c53-9451dd6067c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf"
Sep 15 21:16:16.837874 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.837850    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/113066b8-2eb4-4d91-9c53-9451dd6067c9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf\" (UID: \"113066b8-2eb4-4d91-9c53-9451dd6067c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf"
Sep 15 21:16:16.839575 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.839544    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/113066b8-2eb4-4d91-9c53-9451dd6067c9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf\" (UID: \"113066b8-2eb4-4d91-9c53-9451dd6067c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf"
Sep 15 21:16:16.839887 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.839866    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/113066b8-2eb4-4d91-9c53-9451dd6067c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf\" (UID: \"113066b8-2eb4-4d91-9c53-9451dd6067c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf"
Sep 15 21:16:16.868107 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.868038    2558 generic.go:356] "Generic (PLEG): container finished" podID="9049ff1f-34ce-4639-be80-14ebff380c3a" containerID="260282ddb79fbad0c8851cfd75af2d39a2ddbf89925657a4fb34dd5fc95ae6a7" exitCode=0
Sep 15 21:16:16.868107 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.868110    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"
Sep 15 21:16:16.868335 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.868167    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9" event={"ID":"9049ff1f-34ce-4639-be80-14ebff380c3a","Type":"ContainerDied","Data":"260282ddb79fbad0c8851cfd75af2d39a2ddbf89925657a4fb34dd5fc95ae6a7"}
Sep 15 21:16:16.868335 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.868202    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9" event={"ID":"9049ff1f-34ce-4639-be80-14ebff380c3a","Type":"ContainerDied","Data":"9edf25a81756e1c95813a95ce1140192625e6195207a06398b343069557f8091"}
Sep 15 21:16:16.868335 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.868223    2558 scope.go:117] "RemoveContainer" containerID="260282ddb79fbad0c8851cfd75af2d39a2ddbf89925657a4fb34dd5fc95ae6a7"
Sep 15 21:16:16.871005 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.870977    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzmdk\" (UniqueName: \"kubernetes.io/projected/113066b8-2eb4-4d91-9c53-9451dd6067c9-kube-api-access-bzmdk\") pod \"maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf\" (UID: \"113066b8-2eb4-4d91-9c53-9451dd6067c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf"
Sep 15 21:16:16.878204 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.878185    2558 scope.go:117] "RemoveContainer" containerID="260282ddb79fbad0c8851cfd75af2d39a2ddbf89925657a4fb34dd5fc95ae6a7"
Sep 15 21:16:16.878494 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:16:16.878474    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"260282ddb79fbad0c8851cfd75af2d39a2ddbf89925657a4fb34dd5fc95ae6a7\": container with ID starting with 260282ddb79fbad0c8851cfd75af2d39a2ddbf89925657a4fb34dd5fc95ae6a7 not found: ID does not exist" containerID="260282ddb79fbad0c8851cfd75af2d39a2ddbf89925657a4fb34dd5fc95ae6a7"
Sep 15 21:16:16.878542 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.878505    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"260282ddb79fbad0c8851cfd75af2d39a2ddbf89925657a4fb34dd5fc95ae6a7"} err="failed to get container status \"260282ddb79fbad0c8851cfd75af2d39a2ddbf89925657a4fb34dd5fc95ae6a7\": rpc error: code = NotFound desc = could not find container \"260282ddb79fbad0c8851cfd75af2d39a2ddbf89925657a4fb34dd5fc95ae6a7\": container with ID starting with 260282ddb79fbad0c8851cfd75af2d39a2ddbf89925657a4fb34dd5fc95ae6a7 not found: ID does not exist"
Sep 15 21:16:16.900738 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.900714    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf"
Sep 15 21:16:16.937826 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.937797    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9049ff1f-34ce-4639-be80-14ebff380c3a-dshm\") pod \"9049ff1f-34ce-4639-be80-14ebff380c3a\" (UID: \"9049ff1f-34ce-4639-be80-14ebff380c3a\") "
Sep 15 21:16:16.937991 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.937839    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9049ff1f-34ce-4639-be80-14ebff380c3a-tmp-dir\") pod \"9049ff1f-34ce-4639-be80-14ebff380c3a\" (UID: \"9049ff1f-34ce-4639-be80-14ebff380c3a\") "
Sep 15 21:16:16.937991 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.937880    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwj76\" (UniqueName: \"kubernetes.io/projected/9049ff1f-34ce-4639-be80-14ebff380c3a-kube-api-access-wwj76\") pod \"9049ff1f-34ce-4639-be80-14ebff380c3a\" (UID: \"9049ff1f-34ce-4639-be80-14ebff380c3a\") "
Sep 15 21:16:16.937991 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.937925    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9049ff1f-34ce-4639-be80-14ebff380c3a-model-cache\") pod \"9049ff1f-34ce-4639-be80-14ebff380c3a\" (UID: \"9049ff1f-34ce-4639-be80-14ebff380c3a\") "
Sep 15 21:16:16.937991 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.937977    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9049ff1f-34ce-4639-be80-14ebff380c3a-tls-certs\") pod \"9049ff1f-34ce-4639-be80-14ebff380c3a\" (UID: \"9049ff1f-34ce-4639-be80-14ebff380c3a\") "
Sep 15 21:16:16.938217 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.938074    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9049ff1f-34ce-4639-be80-14ebff380c3a-home\") pod \"9049ff1f-34ce-4639-be80-14ebff380c3a\" (UID: \"9049ff1f-34ce-4639-be80-14ebff380c3a\") "
Sep 15 21:16:16.938789 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.938577    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9049ff1f-34ce-4639-be80-14ebff380c3a-home" (OuterVolumeSpecName: "home") pod "9049ff1f-34ce-4639-be80-14ebff380c3a" (UID: "9049ff1f-34ce-4639-be80-14ebff380c3a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:16:16.938909 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.938881    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9049ff1f-34ce-4639-be80-14ebff380c3a-model-cache" (OuterVolumeSpecName: "model-cache") pod "9049ff1f-34ce-4639-be80-14ebff380c3a" (UID: "9049ff1f-34ce-4639-be80-14ebff380c3a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:16:16.939005 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.938978    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9049ff1f-34ce-4639-be80-14ebff380c3a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9049ff1f-34ce-4639-be80-14ebff380c3a" (UID: "9049ff1f-34ce-4639-be80-14ebff380c3a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:16:16.940754 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.940731    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9049ff1f-34ce-4639-be80-14ebff380c3a-dshm" (OuterVolumeSpecName: "dshm") pod "9049ff1f-34ce-4639-be80-14ebff380c3a" (UID: "9049ff1f-34ce-4639-be80-14ebff380c3a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:16:16.940754 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.940745    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9049ff1f-34ce-4639-be80-14ebff380c3a-kube-api-access-wwj76" (OuterVolumeSpecName: "kube-api-access-wwj76") pod "9049ff1f-34ce-4639-be80-14ebff380c3a" (UID: "9049ff1f-34ce-4639-be80-14ebff380c3a"). InnerVolumeSpecName "kube-api-access-wwj76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:16:16.941214 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:16.941194    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9049ff1f-34ce-4639-be80-14ebff380c3a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9049ff1f-34ce-4639-be80-14ebff380c3a" (UID: "9049ff1f-34ce-4639-be80-14ebff380c3a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:16:17.039937 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:17.039857    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9049ff1f-34ce-4639-be80-14ebff380c3a-dshm\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:16:17.039937 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:17.039890    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9049ff1f-34ce-4639-be80-14ebff380c3a-tmp-dir\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:16:17.039937 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:17.039900    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwj76\" (UniqueName: \"kubernetes.io/projected/9049ff1f-34ce-4639-be80-14ebff380c3a-kube-api-access-wwj76\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:16:17.039937 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:17.039909    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9049ff1f-34ce-4639-be80-14ebff380c3a-model-cache\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:16:17.039937 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:17.039917    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9049ff1f-34ce-4639-be80-14ebff380c3a-tls-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:16:17.039937 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:17.039924    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9049ff1f-34ce-4639-be80-14ebff380c3a-home\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:16:17.197738 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:17.197712    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf"]
Sep 15 21:16:17.198752 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:16:17.198727    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod113066b8_2eb4_4d91_9c53_9451dd6067c9.slice/crio-071808cf8aecc343d478b83ee7953fc537b6578cbd64908103e0edf5c4ee135a WatchSource:0}: Error finding container 071808cf8aecc343d478b83ee7953fc537b6578cbd64908103e0edf5c4ee135a: Status 404 returned error can't find the container with id 071808cf8aecc343d478b83ee7953fc537b6578cbd64908103e0edf5c4ee135a
Sep 15 21:16:17.205767 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:17.205746    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"]
Sep 15 21:16:17.209397 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:17.209372    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-8f08a9/explicit-model-8f08a9-kserve-575b87c5d9-cwqs9"]
Sep 15 21:16:17.460998 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:17.460532    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 15 21:16:17.460998 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:17.460608    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp" podUID="23930493-5ede-47bc-a2b8-437761d1f50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 15 21:16:17.640790 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:17.640764    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:16:17.878043 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:17.877953    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf" event={"ID":"113066b8-2eb4-4d91-9c53-9451dd6067c9","Type":"ContainerStarted","Data":"e9065a7a4814b9b0a827ce906fd233e9d3a917cd675a13ddcab35177777c21a2"}
Sep 15 21:16:17.878043 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:17.878001    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf" event={"ID":"113066b8-2eb4-4d91-9c53-9451dd6067c9","Type":"ContainerStarted","Data":"071808cf8aecc343d478b83ee7953fc537b6578cbd64908103e0edf5c4ee135a"}
Sep 15 21:16:17.878275 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:17.878080    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf"
Sep 15 21:16:17.905897 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:17.905846    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf" podStartSLOduration=1.507238393 podStartE2EDuration="1.905831164s" totalImagesPullingTime="398.592771ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:17.902098748 +0000 UTC m=+1096.392744481" watchObservedRunningTime="2026-09-15 21:16:17.905831164 +0000 UTC m=+1096.396476895"
Sep 15 21:16:17.943389 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:17.943354    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9049ff1f-34ce-4639-be80-14ebff380c3a" path="/var/lib/kubelet/pods/9049ff1f-34ce-4639-be80-14ebff380c3a/volumes"
Sep 15 21:16:18.460551 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:18.460507    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 15 21:16:18.460758 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:18.460575    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp" podUID="23930493-5ede-47bc-a2b8-437761d1f50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 15 21:16:19.464463 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:19.464435    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:16:19.465440 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:19.465424    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:16:20.312919 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:20.312883    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"]
Sep 15 21:16:20.313150 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:20.313126    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt" podUID="4dede905-5520-4367-a6d0-dda15d3f0f59" containerName="istio-proxy" containerID="cri-o://bafeb35bf2bfc3272c63cd61f6aee69529f5ab434189062c1858d16aca3057e7" gracePeriod=30
Sep 15 21:16:21.030007 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.029970    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"]
Sep 15 21:16:21.030391 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.030362    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9049ff1f-34ce-4639-be80-14ebff380c3a" containerName="main"
Sep 15 21:16:21.030391 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.030374    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9049ff1f-34ce-4639-be80-14ebff380c3a" containerName="main"
Sep 15 21:16:21.030483 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.030437    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9049ff1f-34ce-4639-be80-14ebff380c3a" containerName="main"
Sep 15 21:16:21.033463 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.033446    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.036527 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.036498    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-6587cd-openshift-default-dockercfg-4tmlf\""
Sep 15 21:16:21.045424 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.045398    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"]
Sep 15 21:16:21.185907 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.185872    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istio-podinfo\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.185907 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.185910    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istio-token\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.186190 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.185935    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-workload-socket\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.186190 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.185950    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istio-data\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.186190 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.186022    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-credential-socket\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.186190 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.186082    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc8hg\" (UniqueName: \"kubernetes.io/projected/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-kube-api-access-gc8hg\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.186190 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.186117    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-workload-certs\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.186190 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.186182    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istiod-ca-cert\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.186420 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.186205    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istio-envoy\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.287740 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.287634    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istio-podinfo\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.287740 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.287704    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istio-token\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.287961 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.287743    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-workload-socket\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.287961 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.287768    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istio-data\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.287961 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.287799    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-credential-socket\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.287961 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.287834    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc8hg\" (UniqueName: \"kubernetes.io/projected/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-kube-api-access-gc8hg\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.287961 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.287891    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-workload-certs\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.288281 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.287970    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istiod-ca-cert\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.288281 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.287996    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istio-envoy\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.289560 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.288716    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-credential-socket\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.289560 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.288741    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istio-data\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.289560 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.289094    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-workload-certs\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.289560 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.289354    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-workload-socket\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.289560 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.289478    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istiod-ca-cert\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.290921 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.290895    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istio-envoy\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.291069 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.290993    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istio-podinfo\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.297506 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.297480    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc8hg\" (UniqueName: \"kubernetes.io/projected/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-kube-api-access-gc8hg\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.297831 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.297800    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istio-token\") pod \"e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") " pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.344811 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.344779    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:21.626652 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.626618    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"]
Sep 15 21:16:21.627208 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:16:21.627177    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0a00a2_0c0b_48b5_bd27_62168fc3a274.slice/crio-cbd2438332f3fa53ab2b77479da25b0d19871446e17817a9499461cd83532f17 WatchSource:0}: Error finding container cbd2438332f3fa53ab2b77479da25b0d19871446e17817a9499461cd83532f17: Status 404 returned error can't find the container with id cbd2438332f3fa53ab2b77479da25b0d19871446e17817a9499461cd83532f17
Sep 15 21:16:21.629653 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.629616    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:16:21.629738 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.629688    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:16:21.629738 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.629717    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:16:21.906125 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.906022    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr" event={"ID":"4a0a00a2-0c0b-48b5-bd27-62168fc3a274","Type":"ContainerStarted","Data":"5c9a116f9a855caca5356854145460eb68049066d05032d1d80d71d144eb527a"}
Sep 15 21:16:21.906125 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.906097    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr" event={"ID":"4a0a00a2-0c0b-48b5-bd27-62168fc3a274","Type":"ContainerStarted","Data":"cbd2438332f3fa53ab2b77479da25b0d19871446e17817a9499461cd83532f17"}
Sep 15 21:16:21.932933 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:21.932881    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr" podStartSLOduration=0.932867283 podStartE2EDuration="932.867283ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:21.92956777 +0000 UTC m=+1100.420213501" watchObservedRunningTime="2026-09-15 21:16:21.932867283 +0000 UTC m=+1100.423513060"
Sep 15 21:16:22.345995 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:22.345959    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:22.351159 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:22.351136    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:22.912355 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:22.912322    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:22.913368 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:22.913346    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:16:23.891451 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:23.891425    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf"
Sep 15 21:16:25.560975 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.560948    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:16:25.731783 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.731751    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dede905-5520-4367-a6d0-dda15d3f0f59-istiod-ca-cert\") pod \"4dede905-5520-4367-a6d0-dda15d3f0f59\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") "
Sep 15 21:16:25.731951 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.731822    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dede905-5520-4367-a6d0-dda15d3f0f59-istio-podinfo\") pod \"4dede905-5520-4367-a6d0-dda15d3f0f59\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") "
Sep 15 21:16:25.731951 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.731871    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-credential-socket\") pod \"4dede905-5520-4367-a6d0-dda15d3f0f59\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") "
Sep 15 21:16:25.731951 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.731891    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmrsf\" (UniqueName: \"kubernetes.io/projected/4dede905-5520-4367-a6d0-dda15d3f0f59-kube-api-access-dmrsf\") pod \"4dede905-5520-4367-a6d0-dda15d3f0f59\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") "
Sep 15 21:16:25.731951 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.731937    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-istio-data\") pod \"4dede905-5520-4367-a6d0-dda15d3f0f59\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") "
Sep 15 21:16:25.732167 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.731962    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-workload-socket\") pod \"4dede905-5520-4367-a6d0-dda15d3f0f59\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") "
Sep 15 21:16:25.732167 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.731991    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-istio-envoy\") pod \"4dede905-5520-4367-a6d0-dda15d3f0f59\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") "
Sep 15 21:16:25.732167 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.732023    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-workload-certs\") pod \"4dede905-5520-4367-a6d0-dda15d3f0f59\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") "
Sep 15 21:16:25.732167 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.732083    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dede905-5520-4367-a6d0-dda15d3f0f59-istio-token\") pod \"4dede905-5520-4367-a6d0-dda15d3f0f59\" (UID: \"4dede905-5520-4367-a6d0-dda15d3f0f59\") "
Sep 15 21:16:25.732368 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.732214    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4dede905-5520-4367-a6d0-dda15d3f0f59" (UID: "4dede905-5520-4367-a6d0-dda15d3f0f59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:16:25.732368 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.732259    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-istio-data" (OuterVolumeSpecName: "istio-data") pod "4dede905-5520-4367-a6d0-dda15d3f0f59" (UID: "4dede905-5520-4367-a6d0-dda15d3f0f59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:16:25.732368 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.732300    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dede905-5520-4367-a6d0-dda15d3f0f59-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4dede905-5520-4367-a6d0-dda15d3f0f59" (UID: "4dede905-5520-4367-a6d0-dda15d3f0f59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:16:25.732561 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.732533    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dede905-5520-4367-a6d0-dda15d3f0f59-istiod-ca-cert\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:16:25.732630 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.732590    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-credential-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:16:25.732630 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.732606    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-istio-data\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:16:25.732630 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.732535    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4dede905-5520-4367-a6d0-dda15d3f0f59" (UID: "4dede905-5520-4367-a6d0-dda15d3f0f59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:16:25.732866 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.732833    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4dede905-5520-4367-a6d0-dda15d3f0f59" (UID: "4dede905-5520-4367-a6d0-dda15d3f0f59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:16:25.734175 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.734148    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4dede905-5520-4367-a6d0-dda15d3f0f59-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4dede905-5520-4367-a6d0-dda15d3f0f59" (UID: "4dede905-5520-4367-a6d0-dda15d3f0f59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:16:25.734365 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.734343    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dede905-5520-4367-a6d0-dda15d3f0f59-istio-token" (OuterVolumeSpecName: "istio-token") pod "4dede905-5520-4367-a6d0-dda15d3f0f59" (UID: "4dede905-5520-4367-a6d0-dda15d3f0f59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:16:25.734720 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.734695    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4dede905-5520-4367-a6d0-dda15d3f0f59" (UID: "4dede905-5520-4367-a6d0-dda15d3f0f59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:16:25.734808 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.734721    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dede905-5520-4367-a6d0-dda15d3f0f59-kube-api-access-dmrsf" (OuterVolumeSpecName: "kube-api-access-dmrsf") pod "4dede905-5520-4367-a6d0-dda15d3f0f59" (UID: "4dede905-5520-4367-a6d0-dda15d3f0f59"). InnerVolumeSpecName "kube-api-access-dmrsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:16:25.833429 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.833386    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dede905-5520-4367-a6d0-dda15d3f0f59-istio-podinfo\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:16:25.833429 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.833426    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmrsf\" (UniqueName: \"kubernetes.io/projected/4dede905-5520-4367-a6d0-dda15d3f0f59-kube-api-access-dmrsf\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:16:25.833429 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.833438    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-workload-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:16:25.833429 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.833447    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-istio-envoy\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:16:25.833689 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.833455    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dede905-5520-4367-a6d0-dda15d3f0f59-workload-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:16:25.833689 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.833463    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dede905-5520-4367-a6d0-dda15d3f0f59-istio-token\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:16:25.935451 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.935421    2558 generic.go:356] "Generic (PLEG): container finished" podID="4dede905-5520-4367-a6d0-dda15d3f0f59" containerID="bafeb35bf2bfc3272c63cd61f6aee69529f5ab434189062c1858d16aca3057e7" exitCode=0
Sep 15 21:16:25.935593 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.935468    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"
Sep 15 21:16:25.935593 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.935513    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt" event={"ID":"4dede905-5520-4367-a6d0-dda15d3f0f59","Type":"ContainerDied","Data":"bafeb35bf2bfc3272c63cd61f6aee69529f5ab434189062c1858d16aca3057e7"}
Sep 15 21:16:25.935593 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.935572    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt" event={"ID":"4dede905-5520-4367-a6d0-dda15d3f0f59","Type":"ContainerDied","Data":"71501753a151f9fe47227aca6e58c110dcae6463e2b66960b9f3d747ffaf2448"}
Sep 15 21:16:25.935773 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.935601    2558 scope.go:117] "RemoveContainer" containerID="bafeb35bf2bfc3272c63cd61f6aee69529f5ab434189062c1858d16aca3057e7"
Sep 15 21:16:25.947195 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.947175    2558 scope.go:117] "RemoveContainer" containerID="0f60f88f25225b2f3f1221b1c5cdbedf239a05d3e827ee83055da27c7273bf15"
Sep 15 21:16:25.957447 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.957428    2558 scope.go:117] "RemoveContainer" containerID="bafeb35bf2bfc3272c63cd61f6aee69529f5ab434189062c1858d16aca3057e7"
Sep 15 21:16:25.957719 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:16:25.957700    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bafeb35bf2bfc3272c63cd61f6aee69529f5ab434189062c1858d16aca3057e7\": container with ID starting with bafeb35bf2bfc3272c63cd61f6aee69529f5ab434189062c1858d16aca3057e7 not found: ID does not exist" containerID="bafeb35bf2bfc3272c63cd61f6aee69529f5ab434189062c1858d16aca3057e7"
Sep 15 21:16:25.957788 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.957726    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bafeb35bf2bfc3272c63cd61f6aee69529f5ab434189062c1858d16aca3057e7"} err="failed to get container status \"bafeb35bf2bfc3272c63cd61f6aee69529f5ab434189062c1858d16aca3057e7\": rpc error: code = NotFound desc = could not find container \"bafeb35bf2bfc3272c63cd61f6aee69529f5ab434189062c1858d16aca3057e7\": container with ID starting with bafeb35bf2bfc3272c63cd61f6aee69529f5ab434189062c1858d16aca3057e7 not found: ID does not exist"
Sep 15 21:16:25.957788 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.957744    2558 scope.go:117] "RemoveContainer" containerID="0f60f88f25225b2f3f1221b1c5cdbedf239a05d3e827ee83055da27c7273bf15"
Sep 15 21:16:25.957984 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:16:25.957967    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f60f88f25225b2f3f1221b1c5cdbedf239a05d3e827ee83055da27c7273bf15\": container with ID starting with 0f60f88f25225b2f3f1221b1c5cdbedf239a05d3e827ee83055da27c7273bf15 not found: ID does not exist" containerID="0f60f88f25225b2f3f1221b1c5cdbedf239a05d3e827ee83055da27c7273bf15"
Sep 15 21:16:25.958025 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.957991    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f60f88f25225b2f3f1221b1c5cdbedf239a05d3e827ee83055da27c7273bf15"} err="failed to get container status \"0f60f88f25225b2f3f1221b1c5cdbedf239a05d3e827ee83055da27c7273bf15\": rpc error: code = NotFound desc = could not find container \"0f60f88f25225b2f3f1221b1c5cdbedf239a05d3e827ee83055da27c7273bf15\": container with ID starting with 0f60f88f25225b2f3f1221b1c5cdbedf239a05d3e827ee83055da27c7273bf15 not found: ID does not exist"
Sep 15 21:16:25.972034 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.972012    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"]
Sep 15 21:16:25.987826 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:25.987781    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-8f08a9-openshift-default-85cf7dc756-tc2dt"]
Sep 15 21:16:27.942227 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:27.942198    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dede905-5520-4367-a6d0-dda15d3f0f59" path="/var/lib/kubelet/pods/4dede905-5520-4367-a6d0-dda15d3f0f59/volumes"
Sep 15 21:16:28.877634 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:28.877598    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"]
Sep 15 21:16:28.877949 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:28.877937    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dede905-5520-4367-a6d0-dda15d3f0f59" containerName="istio-proxy"
Sep 15 21:16:28.877990 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:28.877951    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dede905-5520-4367-a6d0-dda15d3f0f59" containerName="istio-proxy"
Sep 15 21:16:28.878031 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:28.878022    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dede905-5520-4367-a6d0-dda15d3f0f59" containerName="istio-proxy"
Sep 15 21:16:28.878031 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:28.878029    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dede905-5520-4367-a6d0-dda15d3f0f59" containerName="istio-proxy"
Sep 15 21:16:28.882794 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:28.882775    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:16:28.885791 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:28.885768    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-6587cd\"/\"openshift-service-ca.crt\""
Sep 15 21:16:28.885937 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:28.885884    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-6587cd\"/\"kube-root-ca.crt\""
Sep 15 21:16:28.888869 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:28.888029    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-6587cd\"/\"default-dockercfg-fzns9\""
Sep 15 21:16:28.889698 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:28.888916    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-6587cd\"/\"noten-model-6587cd-kserve-self-signed-certs\""
Sep 15 21:16:28.896335 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:28.896291    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"]
Sep 15 21:16:28.958167 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:28.958138    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39794398-9d10-4ef6-ad6e-2f85d3f490b0-tls-certs\") pod \"noten-model-6587cd-kserve-5449c57ddc-586cr\" (UID: \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\") " pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:16:28.958519 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:28.958199    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39794398-9d10-4ef6-ad6e-2f85d3f490b0-dshm\") pod \"noten-model-6587cd-kserve-5449c57ddc-586cr\" (UID: \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\") " pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:16:28.958519 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:28.958225    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39794398-9d10-4ef6-ad6e-2f85d3f490b0-model-cache\") pod \"noten-model-6587cd-kserve-5449c57ddc-586cr\" (UID: \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\") " pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:16:28.958519 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:28.958329    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39794398-9d10-4ef6-ad6e-2f85d3f490b0-tmp-dir\") pod \"noten-model-6587cd-kserve-5449c57ddc-586cr\" (UID: \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\") " pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:16:28.958519 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:28.958363    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwzjv\" (UniqueName: \"kubernetes.io/projected/39794398-9d10-4ef6-ad6e-2f85d3f490b0-kube-api-access-fwzjv\") pod \"noten-model-6587cd-kserve-5449c57ddc-586cr\" (UID: \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\") " pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:16:28.958519 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:28.958383    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39794398-9d10-4ef6-ad6e-2f85d3f490b0-home\") pod \"noten-model-6587cd-kserve-5449c57ddc-586cr\" (UID: \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\") " pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:16:29.058846 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:29.058815    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39794398-9d10-4ef6-ad6e-2f85d3f490b0-dshm\") pod \"noten-model-6587cd-kserve-5449c57ddc-586cr\" (UID: \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\") " pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:16:29.059075 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:29.058866    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39794398-9d10-4ef6-ad6e-2f85d3f490b0-model-cache\") pod \"noten-model-6587cd-kserve-5449c57ddc-586cr\" (UID: \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\") " pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:16:29.059075 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:29.058982    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39794398-9d10-4ef6-ad6e-2f85d3f490b0-tmp-dir\") pod \"noten-model-6587cd-kserve-5449c57ddc-586cr\" (UID: \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\") " pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:16:29.059075 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:29.059011    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwzjv\" (UniqueName: \"kubernetes.io/projected/39794398-9d10-4ef6-ad6e-2f85d3f490b0-kube-api-access-fwzjv\") pod \"noten-model-6587cd-kserve-5449c57ddc-586cr\" (UID: \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\") " pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:16:29.059075 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:29.059040    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39794398-9d10-4ef6-ad6e-2f85d3f490b0-home\") pod \"noten-model-6587cd-kserve-5449c57ddc-586cr\" (UID: \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\") " pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:16:29.059311 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:29.059102    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39794398-9d10-4ef6-ad6e-2f85d3f490b0-tls-certs\") pod \"noten-model-6587cd-kserve-5449c57ddc-586cr\" (UID: \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\") " pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:16:29.059311 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:29.059300    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39794398-9d10-4ef6-ad6e-2f85d3f490b0-model-cache\") pod \"noten-model-6587cd-kserve-5449c57ddc-586cr\" (UID: \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\") " pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:16:29.059421 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:29.059349    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39794398-9d10-4ef6-ad6e-2f85d3f490b0-tmp-dir\") pod \"noten-model-6587cd-kserve-5449c57ddc-586cr\" (UID: \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\") " pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:16:29.059421 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:29.059395    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39794398-9d10-4ef6-ad6e-2f85d3f490b0-home\") pod \"noten-model-6587cd-kserve-5449c57ddc-586cr\" (UID: \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\") " pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:16:29.061109 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:29.061091    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39794398-9d10-4ef6-ad6e-2f85d3f490b0-dshm\") pod \"noten-model-6587cd-kserve-5449c57ddc-586cr\" (UID: \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\") " pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:16:29.061586 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:29.061565    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39794398-9d10-4ef6-ad6e-2f85d3f490b0-tls-certs\") pod \"noten-model-6587cd-kserve-5449c57ddc-586cr\" (UID: \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\") " pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:16:29.067338 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:29.067313    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwzjv\" (UniqueName: \"kubernetes.io/projected/39794398-9d10-4ef6-ad6e-2f85d3f490b0-kube-api-access-fwzjv\") pod \"noten-model-6587cd-kserve-5449c57ddc-586cr\" (UID: \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\") " pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:16:29.202816 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:29.202785    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:16:29.485427 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:16:29.485394    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39794398_9d10_4ef6_ad6e_2f85d3f490b0.slice/crio-2c0723d124563b340cac418b7a3b2f619562d411c69b799761326f7bc7b791e3 WatchSource:0}: Error finding container 2c0723d124563b340cac418b7a3b2f619562d411c69b799761326f7bc7b791e3: Status 404 returned error can't find the container with id 2c0723d124563b340cac418b7a3b2f619562d411c69b799761326f7bc7b791e3
Sep 15 21:16:29.485891 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:29.485867    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"]
Sep 15 21:16:29.958334 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:29.958306    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr" event={"ID":"39794398-9d10-4ef6-ad6e-2f85d3f490b0","Type":"ContainerStarted","Data":"d9ae45284008fd6ab5c3644f0b6f1afd226c4215529319bbbb966fcc7f989a45"}
Sep 15 21:16:29.958334 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:29.958344    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr" event={"ID":"39794398-9d10-4ef6-ad6e-2f85d3f490b0","Type":"ContainerStarted","Data":"2c0723d124563b340cac418b7a3b2f619562d411c69b799761326f7bc7b791e3"}
Sep 15 21:16:29.978848 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:29.978806    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr" podStartSLOduration=1.978789642 podStartE2EDuration="1.978789642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:29.975150897 +0000 UTC m=+1108.465796636" watchObservedRunningTime="2026-09-15 21:16:29.978789642 +0000 UTC m=+1108.469435373"
Sep 15 21:16:31.793263 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:31.793231    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:16:39.203876 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:39.203842    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:16:39.204279 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:39.203903    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:16:39.216931 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:39.216906    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:16:40.024933 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:40.024905    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:16:43.323114 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:43.323072    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7925-5bfbed"]
Sep 15 21:16:43.323649 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:43.323613    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dede905-5520-4367-a6d0-dda15d3f0f59" containerName="istio-proxy"
Sep 15 21:16:43.323649 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:43.323634    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dede905-5520-4367-a6d0-dda15d3f0f59" containerName="istio-proxy"
Sep 15 21:16:43.327042 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:43.327021    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7925-5bfbed"
Sep 15 21:16:43.331246 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:43.331224    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-r6zqd\""
Sep 15 21:16:43.346151 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:43.346127    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7925-5bfbed"]
Sep 15 21:16:43.392914 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:43.392885    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzb9b\" (UniqueName: \"kubernetes.io/projected/3a220618-efc8-4e2f-bade-48089f1edea8-kube-api-access-tzb9b\") pod \"test-curl-7925-5bfbed\" (UID: \"3a220618-efc8-4e2f-bade-48089f1edea8\") " pod="opendatahub/test-curl-7925-5bfbed"
Sep 15 21:16:43.493881 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:43.493854    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzb9b\" (UniqueName: \"kubernetes.io/projected/3a220618-efc8-4e2f-bade-48089f1edea8-kube-api-access-tzb9b\") pod \"test-curl-7925-5bfbed\" (UID: \"3a220618-efc8-4e2f-bade-48089f1edea8\") " pod="opendatahub/test-curl-7925-5bfbed"
Sep 15 21:16:43.511426 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:43.511398    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzb9b\" (UniqueName: \"kubernetes.io/projected/3a220618-efc8-4e2f-bade-48089f1edea8-kube-api-access-tzb9b\") pod \"test-curl-7925-5bfbed\" (UID: \"3a220618-efc8-4e2f-bade-48089f1edea8\") " pod="opendatahub/test-curl-7925-5bfbed"
Sep 15 21:16:43.636816 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:43.636744    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7925-5bfbed"
Sep 15 21:16:43.908254 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:43.908227    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7925-5bfbed"]
Sep 15 21:16:43.908770 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:16:43.908746    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a220618_efc8_4e2f_bade_48089f1edea8.slice/crio-7c2e752e9caeb7432593c8b21a3285586e1fcb45e1f0a54c92707c039d866f30 WatchSource:0}: Error finding container 7c2e752e9caeb7432593c8b21a3285586e1fcb45e1f0a54c92707c039d866f30: Status 404 returned error can't find the container with id 7c2e752e9caeb7432593c8b21a3285586e1fcb45e1f0a54c92707c039d866f30
Sep 15 21:16:44.038886 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:44.038851    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7925-5bfbed" event={"ID":"3a220618-efc8-4e2f-bade-48089f1edea8","Type":"ContainerStarted","Data":"4eac84350665cdd92c4acb6c6a9863031332423c0dbb80878fdaa4486daaefb3"}
Sep 15 21:16:44.039098 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:44.038900    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7925-5bfbed" event={"ID":"3a220618-efc8-4e2f-bade-48089f1edea8","Type":"ContainerStarted","Data":"7c2e752e9caeb7432593c8b21a3285586e1fcb45e1f0a54c92707c039d866f30"}
Sep 15 21:16:44.056612 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:44.056566    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7925-5bfbed" podStartSLOduration=1.056549865 podStartE2EDuration="1.056549865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:44.055082996 +0000 UTC m=+1122.545728726" watchObservedRunningTime="2026-09-15 21:16:44.056549865 +0000 UTC m=+1122.547195597"
Sep 15 21:16:44.716263 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:44.716230    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7925-5bfbed"]
Sep 15 21:16:44.719744 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:44.719716    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7925-5bfbed"]
Sep 15 21:16:44.865007 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:44.864920    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7925-ed206d"]
Sep 15 21:16:44.865341 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:44.865327    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a220618-efc8-4e2f-bade-48089f1edea8" containerName="test-curl-7925-5bfbed"
Sep 15 21:16:44.865385 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:44.865342    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a220618-efc8-4e2f-bade-48089f1edea8" containerName="test-curl-7925-5bfbed"
Sep 15 21:16:44.865421 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:44.865411    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a220618-efc8-4e2f-bade-48089f1edea8" containerName="test-curl-7925-5bfbed"
Sep 15 21:16:44.868618 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:44.868602    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7925-ed206d"
Sep 15 21:16:44.890878 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:44.890856    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7925-ed206d"]
Sep 15 21:16:44.907125 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:44.907096    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jj8r\" (UniqueName: \"kubernetes.io/projected/3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab-kube-api-access-6jj8r\") pod \"test-curl-7925-ed206d\" (UID: \"3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab\") " pod="opendatahub/test-curl-7925-ed206d"
Sep 15 21:16:45.008306 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:45.008272    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jj8r\" (UniqueName: \"kubernetes.io/projected/3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab-kube-api-access-6jj8r\") pod \"test-curl-7925-ed206d\" (UID: \"3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab\") " pod="opendatahub/test-curl-7925-ed206d"
Sep 15 21:16:45.021487 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:45.021462    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jj8r\" (UniqueName: \"kubernetes.io/projected/3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab-kube-api-access-6jj8r\") pod \"test-curl-7925-ed206d\" (UID: \"3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab\") " pod="opendatahub/test-curl-7925-ed206d"
Sep 15 21:16:45.044837 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:16:45.044816    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7925-5bfbed" container="test-curl-7925-5bfbed"
Sep 15 21:16:45.177457 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:45.177424    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7925-ed206d"
Sep 15 21:16:45.460715 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:45.460674    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7925-ed206d"]
Sep 15 21:16:45.460857 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:16:45.460685    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e4a227e_8d0e_4de0_ba9e_eab1411ce9ab.slice/crio-635fcad76235b883fbaf5ae26fe54d4cf2c884a78d14b48fe5efaa5bcf2e299f WatchSource:0}: Error finding container 635fcad76235b883fbaf5ae26fe54d4cf2c884a78d14b48fe5efaa5bcf2e299f: Status 404 returned error can't find the container with id 635fcad76235b883fbaf5ae26fe54d4cf2c884a78d14b48fe5efaa5bcf2e299f
Sep 15 21:16:46.051993 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:46.051951    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7925-ed206d" event={"ID":"3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab","Type":"ContainerStarted","Data":"de7a392498598a464f43ed305b37e46f092f08dec65da56bc2462910c6062bfb"}
Sep 15 21:16:46.051993 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:46.051989    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7925-5bfbed" podUID="3a220618-efc8-4e2f-bade-48089f1edea8" containerName="test-curl-7925-5bfbed" containerID="cri-o://4eac84350665cdd92c4acb6c6a9863031332423c0dbb80878fdaa4486daaefb3" gracePeriod=2
Sep 15 21:16:46.052536 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:46.052000    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7925-ed206d" event={"ID":"3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab","Type":"ContainerStarted","Data":"635fcad76235b883fbaf5ae26fe54d4cf2c884a78d14b48fe5efaa5bcf2e299f"}
Sep 15 21:16:46.827241 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:46.827134    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7925-ed206d" podStartSLOduration=2.827110972 podStartE2EDuration="2.827110972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:46.082182593 +0000 UTC m=+1124.572828326" watchObservedRunningTime="2026-09-15 21:16:46.827110972 +0000 UTC m=+1125.317756704"
Sep 15 21:16:46.827872 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:46.827853    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7925-ed206d"]
Sep 15 21:16:46.842962 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:46.842932    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7925-ed206d"]
Sep 15 21:16:46.977006 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:46.976977    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7925-eb465e"]
Sep 15 21:16:46.977346 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:46.977333    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab" containerName="test-curl-7925-ed206d"
Sep 15 21:16:46.977392 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:46.977348    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab" containerName="test-curl-7925-ed206d"
Sep 15 21:16:46.977438 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:46.977429    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab" containerName="test-curl-7925-ed206d"
Sep 15 21:16:46.980492 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:46.980475    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7925-eb465e"
Sep 15 21:16:47.000296 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:47.000272    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7925-eb465e"]
Sep 15 21:16:47.026726 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:47.026696    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxqpx\" (UniqueName: \"kubernetes.io/projected/4284cde2-925f-44c7-996d-8ac9a4f53ec1-kube-api-access-zxqpx\") pod \"test-curl-7925-eb465e\" (UID: \"4284cde2-925f-44c7-996d-8ac9a4f53ec1\") " pod="opendatahub/test-curl-7925-eb465e"
Sep 15 21:16:47.058280 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:16:47.058254    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7925-ed206d" container="test-curl-7925-ed206d"
Sep 15 21:16:47.127336 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:47.127275    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxqpx\" (UniqueName: \"kubernetes.io/projected/4284cde2-925f-44c7-996d-8ac9a4f53ec1-kube-api-access-zxqpx\") pod \"test-curl-7925-eb465e\" (UID: \"4284cde2-925f-44c7-996d-8ac9a4f53ec1\") " pod="opendatahub/test-curl-7925-eb465e"
Sep 15 21:16:47.145293 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:47.145265    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxqpx\" (UniqueName: \"kubernetes.io/projected/4284cde2-925f-44c7-996d-8ac9a4f53ec1-kube-api-access-zxqpx\") pod \"test-curl-7925-eb465e\" (UID: \"4284cde2-925f-44c7-996d-8ac9a4f53ec1\") " pod="opendatahub/test-curl-7925-eb465e"
Sep 15 21:16:47.289676 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:47.289642    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7925-eb465e"
Sep 15 21:16:47.573157 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:47.573129    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7925-eb465e"]
Sep 15 21:16:47.575299 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:16:47.575274    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4284cde2_925f_44c7_996d_8ac9a4f53ec1.slice/crio-38eb3aee4c2b9a9a54d2d9256222be64617b42637b9add3ac44a759150ac9f9b WatchSource:0}: Error finding container 38eb3aee4c2b9a9a54d2d9256222be64617b42637b9add3ac44a759150ac9f9b: Status 404 returned error can't find the container with id 38eb3aee4c2b9a9a54d2d9256222be64617b42637b9add3ac44a759150ac9f9b
Sep 15 21:16:48.065226 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:48.065196    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7925-eb465e" event={"ID":"4284cde2-925f-44c7-996d-8ac9a4f53ec1","Type":"ContainerStarted","Data":"8a00f1ceddb9a32a1c05ad4c34391043d025ee966ab1900156ab2375231bb512"}
Sep 15 21:16:48.065503 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:48.065240    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7925-eb465e" event={"ID":"4284cde2-925f-44c7-996d-8ac9a4f53ec1","Type":"ContainerStarted","Data":"38eb3aee4c2b9a9a54d2d9256222be64617b42637b9add3ac44a759150ac9f9b"}
Sep 15 21:16:48.065503 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:48.065341    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7925-ed206d" podUID="3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab" containerName="test-curl-7925-ed206d" containerID="cri-o://de7a392498598a464f43ed305b37e46f092f08dec65da56bc2462910c6062bfb" gracePeriod=2
Sep 15 21:16:48.083662 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:48.083615    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7925-eb465e" podStartSLOduration=2.083600474 podStartE2EDuration="2.083600474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:48.083290705 +0000 UTC m=+1126.573936438" watchObservedRunningTime="2026-09-15 21:16:48.083600474 +0000 UTC m=+1126.574246226"
Sep 15 21:16:48.229229 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:48.229206    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7925-5bfbed"
Sep 15 21:16:48.338280 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:48.338212    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzb9b\" (UniqueName: \"kubernetes.io/projected/3a220618-efc8-4e2f-bade-48089f1edea8-kube-api-access-tzb9b\") pod \"3a220618-efc8-4e2f-bade-48089f1edea8\" (UID: \"3a220618-efc8-4e2f-bade-48089f1edea8\") "
Sep 15 21:16:48.340003 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:48.339982    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a220618-efc8-4e2f-bade-48089f1edea8-kube-api-access-tzb9b" (OuterVolumeSpecName: "kube-api-access-tzb9b") pod "3a220618-efc8-4e2f-bade-48089f1edea8" (UID: "3a220618-efc8-4e2f-bade-48089f1edea8"). InnerVolumeSpecName "kube-api-access-tzb9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:16:48.439550 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:48.439515    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzb9b\" (UniqueName: \"kubernetes.io/projected/3a220618-efc8-4e2f-bade-48089f1edea8-kube-api-access-tzb9b\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:16:48.651131 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:48.651034    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7925-eb465e"]
Sep 15 21:16:48.658106 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:48.658077    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7925-eb465e"]
Sep 15 21:16:48.776443 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:48.776406    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7925-4adbe2"]
Sep 15 21:16:48.776813 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:48.776799    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4284cde2-925f-44c7-996d-8ac9a4f53ec1" containerName="test-curl-7925-eb465e"
Sep 15 21:16:48.776856 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:48.776815    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4284cde2-925f-44c7-996d-8ac9a4f53ec1" containerName="test-curl-7925-eb465e"
Sep 15 21:16:48.776903 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:48.776894    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4284cde2-925f-44c7-996d-8ac9a4f53ec1" containerName="test-curl-7925-eb465e"
Sep 15 21:16:48.780088 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:48.780071    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7925-4adbe2"
Sep 15 21:16:48.788600 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:48.788576    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7925-4adbe2"]
Sep 15 21:16:48.843202 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:48.843171    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f2lt\" (UniqueName: \"kubernetes.io/projected/ff2325fa-63de-41e0-808e-9a01a1406377-kube-api-access-8f2lt\") pod \"test-curl-7925-4adbe2\" (UID: \"ff2325fa-63de-41e0-808e-9a01a1406377\") " pod="opendatahub/test-curl-7925-4adbe2"
Sep 15 21:16:48.944549 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:48.944519    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f2lt\" (UniqueName: \"kubernetes.io/projected/ff2325fa-63de-41e0-808e-9a01a1406377-kube-api-access-8f2lt\") pod \"test-curl-7925-4adbe2\" (UID: \"ff2325fa-63de-41e0-808e-9a01a1406377\") " pod="opendatahub/test-curl-7925-4adbe2"
Sep 15 21:16:48.957284 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:48.957252    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f2lt\" (UniqueName: \"kubernetes.io/projected/ff2325fa-63de-41e0-808e-9a01a1406377-kube-api-access-8f2lt\") pod \"test-curl-7925-4adbe2\" (UID: \"ff2325fa-63de-41e0-808e-9a01a1406377\") " pod="opendatahub/test-curl-7925-4adbe2"
Sep 15 21:16:49.071927 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:49.071896    2558 generic.go:356] "Generic (PLEG): container finished" podID="3a220618-efc8-4e2f-bade-48089f1edea8" containerID="4eac84350665cdd92c4acb6c6a9863031332423c0dbb80878fdaa4486daaefb3" exitCode=137
Sep 15 21:16:49.072337 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:49.071939    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7925-5bfbed"
Sep 15 21:16:49.072337 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:49.072010    2558 scope.go:117] "RemoveContainer" containerID="4eac84350665cdd92c4acb6c6a9863031332423c0dbb80878fdaa4486daaefb3"
Sep 15 21:16:49.072337 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:16:49.072239    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7925-eb465e" container="test-curl-7925-eb465e"
Sep 15 21:16:49.081977 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:49.081962    2558 scope.go:117] "RemoveContainer" containerID="4eac84350665cdd92c4acb6c6a9863031332423c0dbb80878fdaa4486daaefb3"
Sep 15 21:16:49.082227 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:16:49.082211    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eac84350665cdd92c4acb6c6a9863031332423c0dbb80878fdaa4486daaefb3\": container with ID starting with 4eac84350665cdd92c4acb6c6a9863031332423c0dbb80878fdaa4486daaefb3 not found: ID does not exist" containerID="4eac84350665cdd92c4acb6c6a9863031332423c0dbb80878fdaa4486daaefb3"
Sep 15 21:16:49.082294 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:49.082233    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eac84350665cdd92c4acb6c6a9863031332423c0dbb80878fdaa4486daaefb3"} err="failed to get container status \"4eac84350665cdd92c4acb6c6a9863031332423c0dbb80878fdaa4486daaefb3\": rpc error: code = NotFound desc = could not find container \"4eac84350665cdd92c4acb6c6a9863031332423c0dbb80878fdaa4486daaefb3\": container with ID starting with 4eac84350665cdd92c4acb6c6a9863031332423c0dbb80878fdaa4486daaefb3 not found: ID does not exist"
Sep 15 21:16:49.084373 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:16:49.084336    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7925-5bfbed\" is forbidden: User \"system:node:ip-10-0-142-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-207.ec2.internal' and this object" podUID="3a220618-efc8-4e2f-bade-48089f1edea8" pod="opendatahub/test-curl-7925-5bfbed"
Sep 15 21:16:49.090515 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:49.090501    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7925-4adbe2"
Sep 15 21:16:49.372006 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:49.371976    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7925-4adbe2"]
Sep 15 21:16:49.372413 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:16:49.372389    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff2325fa_63de_41e0_808e_9a01a1406377.slice/crio-6c5555c856830746c3ee8c2306636ccff69dc5a7040fe398fd325f4bad4f6f5b WatchSource:0}: Error finding container 6c5555c856830746c3ee8c2306636ccff69dc5a7040fe398fd325f4bad4f6f5b: Status 404 returned error can't find the container with id 6c5555c856830746c3ee8c2306636ccff69dc5a7040fe398fd325f4bad4f6f5b
Sep 15 21:16:49.943072 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:49.943021    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a220618-efc8-4e2f-bade-48089f1edea8" path="/var/lib/kubelet/pods/3a220618-efc8-4e2f-bade-48089f1edea8/volumes"
Sep 15 21:16:50.080827 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:50.080800    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7925-4adbe2" event={"ID":"ff2325fa-63de-41e0-808e-9a01a1406377","Type":"ContainerStarted","Data":"80d71bc8e7bc9d77f064c471f83ad3abed1960b1a33b0524924a6343646caa7b"}
Sep 15 21:16:50.081176 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:50.080839    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7925-4adbe2" event={"ID":"ff2325fa-63de-41e0-808e-9a01a1406377","Type":"ContainerStarted","Data":"6c5555c856830746c3ee8c2306636ccff69dc5a7040fe398fd325f4bad4f6f5b"}
Sep 15 21:16:50.081176 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:50.080888    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7925-eb465e" podUID="4284cde2-925f-44c7-996d-8ac9a4f53ec1" containerName="test-curl-7925-eb465e" containerID="cri-o://8a00f1ceddb9a32a1c05ad4c34391043d025ee966ab1900156ab2375231bb512" gracePeriod=2
Sep 15 21:16:50.096394 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:50.096338    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7925-4adbe2" podStartSLOduration=2.0963219889999998 podStartE2EDuration="2.096321989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:50.09611192 +0000 UTC m=+1128.586757653" watchObservedRunningTime="2026-09-15 21:16:50.096321989 +0000 UTC m=+1128.586967724"
Sep 15 21:16:50.248880 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:50.248857    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7925-ed206d"
Sep 15 21:16:50.358485 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:50.358449    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jj8r\" (UniqueName: \"kubernetes.io/projected/3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab-kube-api-access-6jj8r\") pod \"3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab\" (UID: \"3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab\") "
Sep 15 21:16:50.360245 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:50.360217    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab-kube-api-access-6jj8r" (OuterVolumeSpecName: "kube-api-access-6jj8r") pod "3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab" (UID: "3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab"). InnerVolumeSpecName "kube-api-access-6jj8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:16:50.459616 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:50.459590    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jj8r\" (UniqueName: \"kubernetes.io/projected/3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab-kube-api-access-6jj8r\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:16:50.666706 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:50.666676    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7925-4adbe2"]
Sep 15 21:16:50.671397 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:50.671369    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7925-4adbe2"]
Sep 15 21:16:51.088251 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.088164    2558 generic.go:356] "Generic (PLEG): container finished" podID="3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab" containerID="de7a392498598a464f43ed305b37e46f092f08dec65da56bc2462910c6062bfb" exitCode=137
Sep 15 21:16:51.088251 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.088210    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7925-ed206d"
Sep 15 21:16:51.088709 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.088258    2558 scope.go:117] "RemoveContainer" containerID="de7a392498598a464f43ed305b37e46f092f08dec65da56bc2462910c6062bfb"
Sep 15 21:16:51.088709 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:16:51.088652    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7925-4adbe2" container="test-curl-7925-4adbe2"
Sep 15 21:16:51.097644 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.097621    2558 scope.go:117] "RemoveContainer" containerID="de7a392498598a464f43ed305b37e46f092f08dec65da56bc2462910c6062bfb"
Sep 15 21:16:51.097904 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:16:51.097886    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de7a392498598a464f43ed305b37e46f092f08dec65da56bc2462910c6062bfb\": container with ID starting with de7a392498598a464f43ed305b37e46f092f08dec65da56bc2462910c6062bfb not found: ID does not exist" containerID="de7a392498598a464f43ed305b37e46f092f08dec65da56bc2462910c6062bfb"
Sep 15 21:16:51.097963 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.097914    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de7a392498598a464f43ed305b37e46f092f08dec65da56bc2462910c6062bfb"} err="failed to get container status \"de7a392498598a464f43ed305b37e46f092f08dec65da56bc2462910c6062bfb\": rpc error: code = NotFound desc = could not find container \"de7a392498598a464f43ed305b37e46f092f08dec65da56bc2462910c6062bfb\": container with ID starting with de7a392498598a464f43ed305b37e46f092f08dec65da56bc2462910c6062bfb not found: ID does not exist"
Sep 15 21:16:51.098881 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:16:51.098862    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7925-ed206d\" is forbidden: User \"system:node:ip-10-0-142-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-207.ec2.internal' and this object" podUID="3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab" pod="opendatahub/test-curl-7925-ed206d"
Sep 15 21:16:51.523556 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.523524    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"]
Sep 15 21:16:51.523903 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.523891    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff2325fa-63de-41e0-808e-9a01a1406377" containerName="test-curl-7925-4adbe2"
Sep 15 21:16:51.523958 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.523905    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff2325fa-63de-41e0-808e-9a01a1406377" containerName="test-curl-7925-4adbe2"
Sep 15 21:16:51.523994 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.523975    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff2325fa-63de-41e0-808e-9a01a1406377" containerName="test-curl-7925-4adbe2"
Sep 15 21:16:51.528400 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.528381    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.531111 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:16:51.531087    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7925-ed206d\" is forbidden: User \"system:node:ip-10-0-142-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-207.ec2.internal' and this object" podUID="3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab" pod="opendatahub/test-curl-7925-ed206d"
Sep 15 21:16:51.531454 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.531439    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-56c6b012-openshift-default-dockercfg-gt55h\""
Sep 15 21:16:51.542004 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.541981    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"]
Sep 15 21:16:51.672063 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.672014    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istio-podinfo\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.672237 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.672076    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istio-token\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.672237 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.672154    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istiod-ca-cert\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.672325 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.672256    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-workload-certs\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.672325 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.672291    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-credential-socket\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.672325 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.672320    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istio-data\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.672423 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.672369    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istio-envoy\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.672423 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.672388    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-workload-socket\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.672423 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.672408    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngldk\" (UniqueName: \"kubernetes.io/projected/9d40317b-6afb-425c-a2d1-0afcbd5d8020-kube-api-access-ngldk\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.773269 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.773235    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istio-podinfo\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.773269 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.773270    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istio-token\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.773605 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.773293    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istiod-ca-cert\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.773605 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.773341    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-workload-certs\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.773605 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.773360    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-credential-socket\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.773605 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.773379    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istio-data\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.773605 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.773421    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istio-envoy\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.773605 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.773487    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-workload-socket\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.773605 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.773528    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngldk\" (UniqueName: \"kubernetes.io/projected/9d40317b-6afb-425c-a2d1-0afcbd5d8020-kube-api-access-ngldk\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.773968 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.773859    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-workload-certs\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.773968 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.773929    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-credential-socket\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.774159 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.774134    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-workload-socket\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.774278 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.774261    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istio-data\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.774317 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.774265    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istiod-ca-cert\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.775534 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.775513    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istio-envoy\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.775786 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.775767    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istio-podinfo\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.783247 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.783226    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istio-token\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.783567 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.783548    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngldk\" (UniqueName: \"kubernetes.io/projected/9d40317b-6afb-425c-a2d1-0afcbd5d8020-kube-api-access-ngldk\") pod \"e2e-mt-56c6b012-openshift-default-798965d66d-m265c\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") " pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.838955 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.838918    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:51.943147 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:16:51.943114    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7925-ed206d\" is forbidden: User \"system:node:ip-10-0-142-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-207.ec2.internal' and this object" podUID="3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab" pod="opendatahub/test-curl-7925-ed206d"
Sep 15 21:16:51.943622 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:51.943598    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab" path="/var/lib/kubelet/pods/3e4a227e-8d0e-4de0-ba9e-eab1411ce9ab/volumes"
Sep 15 21:16:52.095637 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:52.095576    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7925-4adbe2" podUID="ff2325fa-63de-41e0-808e-9a01a1406377" containerName="test-curl-7925-4adbe2" containerID="cri-o://80d71bc8e7bc9d77f064c471f83ad3abed1960b1a33b0524924a6343646caa7b" gracePeriod=2
Sep 15 21:16:52.125832 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:52.125806    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"]
Sep 15 21:16:52.150120 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:16:52.150092    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d40317b_6afb_425c_a2d1_0afcbd5d8020.slice/crio-dbd9d461b30123fa57877ba88ce1a0c5c96d720f1cbb8a32513894cfa07496cc WatchSource:0}: Error finding container dbd9d461b30123fa57877ba88ce1a0c5c96d720f1cbb8a32513894cfa07496cc: Status 404 returned error can't find the container with id dbd9d461b30123fa57877ba88ce1a0c5c96d720f1cbb8a32513894cfa07496cc
Sep 15 21:16:52.152600 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:52.152562    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:16:52.152685 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:52.152635    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:16:52.152685 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:52.152664    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:16:52.257516 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:52.257486    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7925-eb465e"
Sep 15 21:16:52.379296 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:52.379196    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxqpx\" (UniqueName: \"kubernetes.io/projected/4284cde2-925f-44c7-996d-8ac9a4f53ec1-kube-api-access-zxqpx\") pod \"4284cde2-925f-44c7-996d-8ac9a4f53ec1\" (UID: \"4284cde2-925f-44c7-996d-8ac9a4f53ec1\") "
Sep 15 21:16:52.381240 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:52.381216    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4284cde2-925f-44c7-996d-8ac9a4f53ec1-kube-api-access-zxqpx" (OuterVolumeSpecName: "kube-api-access-zxqpx") pod "4284cde2-925f-44c7-996d-8ac9a4f53ec1" (UID: "4284cde2-925f-44c7-996d-8ac9a4f53ec1"). InnerVolumeSpecName "kube-api-access-zxqpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:16:52.480724 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:52.480688    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxqpx\" (UniqueName: \"kubernetes.io/projected/4284cde2-925f-44c7-996d-8ac9a4f53ec1-kube-api-access-zxqpx\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:16:53.103158 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:53.103043    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c" event={"ID":"9d40317b-6afb-425c-a2d1-0afcbd5d8020","Type":"ContainerStarted","Data":"c47bcaea236f1e9465bed3a75f5742baad0e4e9fa54e54f20fb6f849841d164d"}
Sep 15 21:16:53.103158 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:53.103119    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c" event={"ID":"9d40317b-6afb-425c-a2d1-0afcbd5d8020","Type":"ContainerStarted","Data":"dbd9d461b30123fa57877ba88ce1a0c5c96d720f1cbb8a32513894cfa07496cc"}
Sep 15 21:16:53.104361 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:53.104328    2558 generic.go:356] "Generic (PLEG): container finished" podID="4284cde2-925f-44c7-996d-8ac9a4f53ec1" containerID="8a00f1ceddb9a32a1c05ad4c34391043d025ee966ab1900156ab2375231bb512" exitCode=137
Sep 15 21:16:53.104477 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:53.104364    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7925-eb465e"
Sep 15 21:16:53.104477 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:53.104423    2558 scope.go:117] "RemoveContainer" containerID="8a00f1ceddb9a32a1c05ad4c34391043d025ee966ab1900156ab2375231bb512"
Sep 15 21:16:53.113829 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:53.113814    2558 scope.go:117] "RemoveContainer" containerID="8a00f1ceddb9a32a1c05ad4c34391043d025ee966ab1900156ab2375231bb512"
Sep 15 21:16:53.114142 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:16:53.114119    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a00f1ceddb9a32a1c05ad4c34391043d025ee966ab1900156ab2375231bb512\": container with ID starting with 8a00f1ceddb9a32a1c05ad4c34391043d025ee966ab1900156ab2375231bb512 not found: ID does not exist" containerID="8a00f1ceddb9a32a1c05ad4c34391043d025ee966ab1900156ab2375231bb512"
Sep 15 21:16:53.114239 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:53.114148    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a00f1ceddb9a32a1c05ad4c34391043d025ee966ab1900156ab2375231bb512"} err="failed to get container status \"8a00f1ceddb9a32a1c05ad4c34391043d025ee966ab1900156ab2375231bb512\": rpc error: code = NotFound desc = could not find container \"8a00f1ceddb9a32a1c05ad4c34391043d025ee966ab1900156ab2375231bb512\": container with ID starting with 8a00f1ceddb9a32a1c05ad4c34391043d025ee966ab1900156ab2375231bb512 not found: ID does not exist"
Sep 15 21:16:53.127219 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:53.127180    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c" podStartSLOduration=2.127167265 podStartE2EDuration="2.127167265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:53.126120629 +0000 UTC m=+1131.616766362" watchObservedRunningTime="2026-09-15 21:16:53.127167265 +0000 UTC m=+1131.617812996"
Sep 15 21:16:53.128470 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:16:53.128446    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7925-eb465e\" is forbidden: User \"system:node:ip-10-0-142-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-207.ec2.internal' and this object" podUID="4284cde2-925f-44c7-996d-8ac9a4f53ec1" pod="opendatahub/test-curl-7925-eb465e"
Sep 15 21:16:53.839279 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:53.839244    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:53.844316 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:53.844291    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:53.847176 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:16:53.847149    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7925-eb465e\" is forbidden: User \"system:node:ip-10-0-142-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-207.ec2.internal' and this object" podUID="4284cde2-925f-44c7-996d-8ac9a4f53ec1" pod="opendatahub/test-curl-7925-eb465e"
Sep 15 21:16:53.943490 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:53.943452    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4284cde2-925f-44c7-996d-8ac9a4f53ec1" path="/var/lib/kubelet/pods/4284cde2-925f-44c7-996d-8ac9a4f53ec1/volumes"
Sep 15 21:16:54.111672 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:54.111650    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:54.112717 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:54.112654    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:16:54.240665 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:54.240642    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7925-4adbe2"
Sep 15 21:16:54.397908 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:54.397818    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8f2lt\" (UniqueName: \"kubernetes.io/projected/ff2325fa-63de-41e0-808e-9a01a1406377-kube-api-access-8f2lt\") pod \"ff2325fa-63de-41e0-808e-9a01a1406377\" (UID: \"ff2325fa-63de-41e0-808e-9a01a1406377\") "
Sep 15 21:16:54.400002 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:54.399971    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2325fa-63de-41e0-808e-9a01a1406377-kube-api-access-8f2lt" (OuterVolumeSpecName: "kube-api-access-8f2lt") pod "ff2325fa-63de-41e0-808e-9a01a1406377" (UID: "ff2325fa-63de-41e0-808e-9a01a1406377"). InnerVolumeSpecName "kube-api-access-8f2lt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:16:54.499330 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:54.499301    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f2lt\" (UniqueName: \"kubernetes.io/projected/ff2325fa-63de-41e0-808e-9a01a1406377-kube-api-access-8f2lt\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:16:55.118089 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:55.118031    2558 generic.go:356] "Generic (PLEG): container finished" podID="ff2325fa-63de-41e0-808e-9a01a1406377" containerID="80d71bc8e7bc9d77f064c471f83ad3abed1960b1a33b0524924a6343646caa7b" exitCode=137
Sep 15 21:16:55.118089 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:55.118084    2558 scope.go:117] "RemoveContainer" containerID="80d71bc8e7bc9d77f064c471f83ad3abed1960b1a33b0524924a6343646caa7b"
Sep 15 21:16:55.118568 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:55.118106    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7925-4adbe2"
Sep 15 21:16:55.127279 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:55.127264    2558 scope.go:117] "RemoveContainer" containerID="80d71bc8e7bc9d77f064c471f83ad3abed1960b1a33b0524924a6343646caa7b"
Sep 15 21:16:55.127540 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:16:55.127523    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d71bc8e7bc9d77f064c471f83ad3abed1960b1a33b0524924a6343646caa7b\": container with ID starting with 80d71bc8e7bc9d77f064c471f83ad3abed1960b1a33b0524924a6343646caa7b not found: ID does not exist" containerID="80d71bc8e7bc9d77f064c471f83ad3abed1960b1a33b0524924a6343646caa7b"
Sep 15 21:16:55.127592 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:55.127549    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d71bc8e7bc9d77f064c471f83ad3abed1960b1a33b0524924a6343646caa7b"} err="failed to get container status \"80d71bc8e7bc9d77f064c471f83ad3abed1960b1a33b0524924a6343646caa7b\": rpc error: code = NotFound desc = could not find container \"80d71bc8e7bc9d77f064c471f83ad3abed1960b1a33b0524924a6343646caa7b\": container with ID starting with 80d71bc8e7bc9d77f064c471f83ad3abed1960b1a33b0524924a6343646caa7b not found: ID does not exist"
Sep 15 21:16:55.128929 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:16:55.128910    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7925-4adbe2\" is forbidden: User \"system:node:ip-10-0-142-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-207.ec2.internal' and this object" podUID="ff2325fa-63de-41e0-808e-9a01a1406377" pod="opendatahub/test-curl-7925-4adbe2"
Sep 15 21:16:55.942736 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:55.942701    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff2325fa-63de-41e0-808e-9a01a1406377" path="/var/lib/kubelet/pods/ff2325fa-63de-41e0-808e-9a01a1406377/volumes"
Sep 15 21:16:58.598735 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:16:58.598698    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:17:03.802996 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:03.802962    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"]
Sep 15 21:17:03.803443 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:03.803180    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr" podUID="4a0a00a2-0c0b-48b5-bd27-62168fc3a274" containerName="istio-proxy" containerID="cri-o://5c9a116f9a855caca5356854145460eb68049066d05032d1d80d71d144eb527a" gracePeriod=30
Sep 15 21:17:07.914952 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:07.914916    2558 patch_prober.go:28] interesting pod/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 21:17:07.915366 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:07.914973    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr" podUID="4a0a00a2-0c0b-48b5-bd27-62168fc3a274" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 21:17:09.052492 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.052465    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:17:09.124983 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.124914    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"]
Sep 15 21:17:09.198300 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.198244    2558 generic.go:356] "Generic (PLEG): container finished" podID="4a0a00a2-0c0b-48b5-bd27-62168fc3a274" containerID="5c9a116f9a855caca5356854145460eb68049066d05032d1d80d71d144eb527a" exitCode=0
Sep 15 21:17:09.198300 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.198288    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"
Sep 15 21:17:09.198572 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.198310    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr" event={"ID":"4a0a00a2-0c0b-48b5-bd27-62168fc3a274","Type":"ContainerDied","Data":"5c9a116f9a855caca5356854145460eb68049066d05032d1d80d71d144eb527a"}
Sep 15 21:17:09.198572 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.198381    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr" event={"ID":"4a0a00a2-0c0b-48b5-bd27-62168fc3a274","Type":"ContainerDied","Data":"cbd2438332f3fa53ab2b77479da25b0d19871446e17817a9499461cd83532f17"}
Sep 15 21:17:09.198572 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.198413    2558 scope.go:117] "RemoveContainer" containerID="5c9a116f9a855caca5356854145460eb68049066d05032d1d80d71d144eb527a"
Sep 15 21:17:09.207869 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.207760    2558 scope.go:117] "RemoveContainer" containerID="5c9a116f9a855caca5356854145460eb68049066d05032d1d80d71d144eb527a"
Sep 15 21:17:09.208127 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:17:09.208105    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c9a116f9a855caca5356854145460eb68049066d05032d1d80d71d144eb527a\": container with ID starting with 5c9a116f9a855caca5356854145460eb68049066d05032d1d80d71d144eb527a not found: ID does not exist" containerID="5c9a116f9a855caca5356854145460eb68049066d05032d1d80d71d144eb527a"
Sep 15 21:17:09.208210 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.208134    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c9a116f9a855caca5356854145460eb68049066d05032d1d80d71d144eb527a"} err="failed to get container status \"5c9a116f9a855caca5356854145460eb68049066d05032d1d80d71d144eb527a\": rpc error: code = NotFound desc = could not find container \"5c9a116f9a855caca5356854145460eb68049066d05032d1d80d71d144eb527a\": container with ID starting with 5c9a116f9a855caca5356854145460eb68049066d05032d1d80d71d144eb527a not found: ID does not exist"
Sep 15 21:17:09.234465 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.234446    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-workload-socket\") pod \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") "
Sep 15 21:17:09.234584 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.234487    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-credential-socket\") pod \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") "
Sep 15 21:17:09.234584 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.234534    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istio-token\") pod \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") "
Sep 15 21:17:09.234670 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.234589    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gc8hg\" (UniqueName: \"kubernetes.io/projected/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-kube-api-access-gc8hg\") pod \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") "
Sep 15 21:17:09.234670 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.234650    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istio-podinfo\") pod \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") "
Sep 15 21:17:09.234763 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.234689    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istio-envoy\") pod \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") "
Sep 15 21:17:09.234763 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.234719    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istio-data\") pod \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") "
Sep 15 21:17:09.234763 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.234733    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4a0a00a2-0c0b-48b5-bd27-62168fc3a274" (UID: "4a0a00a2-0c0b-48b5-bd27-62168fc3a274"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:17:09.234919 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.234774    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istiod-ca-cert\") pod \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") "
Sep 15 21:17:09.234919 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.234821    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-workload-certs\") pod \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\" (UID: \"4a0a00a2-0c0b-48b5-bd27-62168fc3a274\") "
Sep 15 21:17:09.234919 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.234840    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4a0a00a2-0c0b-48b5-bd27-62168fc3a274" (UID: "4a0a00a2-0c0b-48b5-bd27-62168fc3a274"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:17:09.235124 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.234962    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istio-data" (OuterVolumeSpecName: "istio-data") pod "4a0a00a2-0c0b-48b5-bd27-62168fc3a274" (UID: "4a0a00a2-0c0b-48b5-bd27-62168fc3a274"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:17:09.235205 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.235184    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4a0a00a2-0c0b-48b5-bd27-62168fc3a274" (UID: "4a0a00a2-0c0b-48b5-bd27-62168fc3a274"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:17:09.235274 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.235218    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4a0a00a2-0c0b-48b5-bd27-62168fc3a274" (UID: "4a0a00a2-0c0b-48b5-bd27-62168fc3a274"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:17:09.235274 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.235261    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istio-data\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:17:09.235379 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.235278    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-workload-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:17:09.235379 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.235287    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-credential-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:17:09.236606 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.236584    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istio-token" (OuterVolumeSpecName: "istio-token") pod "4a0a00a2-0c0b-48b5-bd27-62168fc3a274" (UID: "4a0a00a2-0c0b-48b5-bd27-62168fc3a274"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:17:09.236810 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.236661    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-kube-api-access-gc8hg" (OuterVolumeSpecName: "kube-api-access-gc8hg") pod "4a0a00a2-0c0b-48b5-bd27-62168fc3a274" (UID: "4a0a00a2-0c0b-48b5-bd27-62168fc3a274"). InnerVolumeSpecName "kube-api-access-gc8hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:17:09.237186 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.237165    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4a0a00a2-0c0b-48b5-bd27-62168fc3a274" (UID: "4a0a00a2-0c0b-48b5-bd27-62168fc3a274"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:17:09.237265 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.237208    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4a0a00a2-0c0b-48b5-bd27-62168fc3a274" (UID: "4a0a00a2-0c0b-48b5-bd27-62168fc3a274"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:17:09.336734 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.336705    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-workload-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:17:09.336734 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.336729    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istio-token\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:17:09.336734 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.336739    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc8hg\" (UniqueName: \"kubernetes.io/projected/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-kube-api-access-gc8hg\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:17:09.336944 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.336748    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istio-podinfo\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:17:09.336944 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.336756    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istio-envoy\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:17:09.336944 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.336763    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0a00a2-0c0b-48b5-bd27-62168fc3a274-istiod-ca-cert\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:17:09.524548 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.524520    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"]
Sep 15 21:17:09.532344 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.532321    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-6587cd-openshift-default-7db8bd6774-psrcr"]
Sep 15 21:17:09.846941 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.846851    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"]
Sep 15 21:17:09.942883 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:09.942845    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a0a00a2-0c0b-48b5-bd27-62168fc3a274" path="/var/lib/kubelet/pods/4a0a00a2-0c0b-48b5-bd27-62168fc3a274/volumes"
Sep 15 21:17:22.286624 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:22.286593    2558 generic.go:356] "Generic (PLEG): container finished" podID="9d40317b-6afb-425c-a2d1-0afcbd5d8020" containerID="c47bcaea236f1e9465bed3a75f5742baad0e4e9fa54e54f20fb6f849841d164d" exitCode=0
Sep 15 21:17:22.287018 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:22.286670    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c" event={"ID":"9d40317b-6afb-425c-a2d1-0afcbd5d8020","Type":"ContainerDied","Data":"c47bcaea236f1e9465bed3a75f5742baad0e4e9fa54e54f20fb6f849841d164d"}
Sep 15 21:17:22.287141 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:22.287096    2558 scope.go:117] "RemoveContainer" containerID="c47bcaea236f1e9465bed3a75f5742baad0e4e9fa54e54f20fb6f849841d164d"
Sep 15 21:17:22.288220 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:22.288189    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:17:22.288277 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:22.288255    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:17:22.288313 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:22.288286    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:17:23.296987 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:23.296952    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c" event={"ID":"9d40317b-6afb-425c-a2d1-0afcbd5d8020","Type":"ContainerStarted","Data":"ecaf6e00447d74fa16c101479e86459827dcea56dbfab5e501e65159c548208f"}
Sep 15 21:17:23.839664 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:23.839626    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:17:23.840563 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:23.840534    2558 patch_prober.go:28] interesting pod/e2e-mt-56c6b012-openshift-default-798965d66d-m265c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 15 21:17:23.840685 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:23.840590    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c" podUID="9d40317b-6afb-425c-a2d1-0afcbd5d8020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 15 21:17:24.111778 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.111665    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:17:24.148808 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.148752    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr" podUID="39794398-9d10-4ef6-ad6e-2f85d3f490b0" containerName="main" containerID="cri-o://d9ae45284008fd6ab5c3644f0b6f1afd226c4215529319bbbb966fcc7f989a45" gracePeriod=45
Sep 15 21:17:24.304783 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.304747    2558 generic.go:356] "Generic (PLEG): container finished" podID="39794398-9d10-4ef6-ad6e-2f85d3f490b0" containerID="d9ae45284008fd6ab5c3644f0b6f1afd226c4215529319bbbb966fcc7f989a45" exitCode=0
Sep 15 21:17:24.304783 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.304781    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr" event={"ID":"39794398-9d10-4ef6-ad6e-2f85d3f490b0","Type":"ContainerDied","Data":"d9ae45284008fd6ab5c3644f0b6f1afd226c4215529319bbbb966fcc7f989a45"}
Sep 15 21:17:24.418012 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.417989    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:17:24.475787 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.475753    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39794398-9d10-4ef6-ad6e-2f85d3f490b0-tls-certs\") pod \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\" (UID: \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\") "
Sep 15 21:17:24.475957 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.475812    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39794398-9d10-4ef6-ad6e-2f85d3f490b0-model-cache\") pod \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\" (UID: \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\") "
Sep 15 21:17:24.475957 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.475837    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39794398-9d10-4ef6-ad6e-2f85d3f490b0-dshm\") pod \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\" (UID: \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\") "
Sep 15 21:17:24.475957 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.475869    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39794398-9d10-4ef6-ad6e-2f85d3f490b0-home\") pod \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\" (UID: \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\") "
Sep 15 21:17:24.475957 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.475915    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwzjv\" (UniqueName: \"kubernetes.io/projected/39794398-9d10-4ef6-ad6e-2f85d3f490b0-kube-api-access-fwzjv\") pod \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\" (UID: \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\") "
Sep 15 21:17:24.475957 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.475941    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39794398-9d10-4ef6-ad6e-2f85d3f490b0-tmp-dir\") pod \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\" (UID: \"39794398-9d10-4ef6-ad6e-2f85d3f490b0\") "
Sep 15 21:17:24.476253 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.476119    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39794398-9d10-4ef6-ad6e-2f85d3f490b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "39794398-9d10-4ef6-ad6e-2f85d3f490b0" (UID: "39794398-9d10-4ef6-ad6e-2f85d3f490b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:17:24.476253 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.476131    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39794398-9d10-4ef6-ad6e-2f85d3f490b0-home" (OuterVolumeSpecName: "home") pod "39794398-9d10-4ef6-ad6e-2f85d3f490b0" (UID: "39794398-9d10-4ef6-ad6e-2f85d3f490b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:17:24.476367 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.476309    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39794398-9d10-4ef6-ad6e-2f85d3f490b0-home\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:17:24.476367 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.476333    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39794398-9d10-4ef6-ad6e-2f85d3f490b0-model-cache\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:17:24.476367 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.476300    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39794398-9d10-4ef6-ad6e-2f85d3f490b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "39794398-9d10-4ef6-ad6e-2f85d3f490b0" (UID: "39794398-9d10-4ef6-ad6e-2f85d3f490b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:17:24.478154 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.478128    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39794398-9d10-4ef6-ad6e-2f85d3f490b0-dshm" (OuterVolumeSpecName: "dshm") pod "39794398-9d10-4ef6-ad6e-2f85d3f490b0" (UID: "39794398-9d10-4ef6-ad6e-2f85d3f490b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:17:24.478268 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.478165    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39794398-9d10-4ef6-ad6e-2f85d3f490b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "39794398-9d10-4ef6-ad6e-2f85d3f490b0" (UID: "39794398-9d10-4ef6-ad6e-2f85d3f490b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:17:24.478268 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.478224    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39794398-9d10-4ef6-ad6e-2f85d3f490b0-kube-api-access-fwzjv" (OuterVolumeSpecName: "kube-api-access-fwzjv") pod "39794398-9d10-4ef6-ad6e-2f85d3f490b0" (UID: "39794398-9d10-4ef6-ad6e-2f85d3f490b0"). InnerVolumeSpecName "kube-api-access-fwzjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:17:24.577349 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.577313    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39794398-9d10-4ef6-ad6e-2f85d3f490b0-tls-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:17:24.577349 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.577340    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39794398-9d10-4ef6-ad6e-2f85d3f490b0-dshm\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:17:24.577349 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.577349    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwzjv\" (UniqueName: \"kubernetes.io/projected/39794398-9d10-4ef6-ad6e-2f85d3f490b0-kube-api-access-fwzjv\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:17:24.577349 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.577359    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39794398-9d10-4ef6-ad6e-2f85d3f490b0-tmp-dir\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:17:24.839869 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.839836    2558 patch_prober.go:28] interesting pod/e2e-mt-56c6b012-openshift-default-798965d66d-m265c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 15 21:17:24.840069 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.839890    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c" podUID="9d40317b-6afb-425c-a2d1-0afcbd5d8020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 15 21:17:24.868461 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:24.868295    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82" podUID="fc9e1da3-e02a-4262-8bba-d9a4863737d4" containerName="main" containerID="cri-o://4a43473f568925bdad43e09ca111ef75bc945fdc85a861d55f1f8e1de930137b" gracePeriod=45
Sep 15 21:17:25.140444 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.140418    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:17:25.182389 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.182365    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc9e1da3-e02a-4262-8bba-d9a4863737d4-dshm\") pod \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\" (UID: \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\") "
Sep 15 21:17:25.182520 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.182402    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2t9v\" (UniqueName: \"kubernetes.io/projected/fc9e1da3-e02a-4262-8bba-d9a4863737d4-kube-api-access-p2t9v\") pod \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\" (UID: \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\") "
Sep 15 21:17:25.182520 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.182437    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc9e1da3-e02a-4262-8bba-d9a4863737d4-home\") pod \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\" (UID: \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\") "
Sep 15 21:17:25.182520 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.182454    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc9e1da3-e02a-4262-8bba-d9a4863737d4-tmp-dir\") pod \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\" (UID: \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\") "
Sep 15 21:17:25.182636 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.182568    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc9e1da3-e02a-4262-8bba-d9a4863737d4-tls-certs\") pod \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\" (UID: \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\") "
Sep 15 21:17:25.182636 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.182598    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc9e1da3-e02a-4262-8bba-d9a4863737d4-model-cache\") pod \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\" (UID: \"fc9e1da3-e02a-4262-8bba-d9a4863737d4\") "
Sep 15 21:17:25.182722 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.182642    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc9e1da3-e02a-4262-8bba-d9a4863737d4-home" (OuterVolumeSpecName: "home") pod "fc9e1da3-e02a-4262-8bba-d9a4863737d4" (UID: "fc9e1da3-e02a-4262-8bba-d9a4863737d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:17:25.182782 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.182724    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc9e1da3-e02a-4262-8bba-d9a4863737d4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc9e1da3-e02a-4262-8bba-d9a4863737d4" (UID: "fc9e1da3-e02a-4262-8bba-d9a4863737d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:17:25.182836 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.182798    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc9e1da3-e02a-4262-8bba-d9a4863737d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc9e1da3-e02a-4262-8bba-d9a4863737d4" (UID: "fc9e1da3-e02a-4262-8bba-d9a4863737d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:17:25.182982 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.182964    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc9e1da3-e02a-4262-8bba-d9a4863737d4-home\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:17:25.182982 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.182982    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc9e1da3-e02a-4262-8bba-d9a4863737d4-tmp-dir\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:17:25.183165 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.182996    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc9e1da3-e02a-4262-8bba-d9a4863737d4-model-cache\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:17:25.184735 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.184708    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc9e1da3-e02a-4262-8bba-d9a4863737d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc9e1da3-e02a-4262-8bba-d9a4863737d4" (UID: "fc9e1da3-e02a-4262-8bba-d9a4863737d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:17:25.184962 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.184942    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc9e1da3-e02a-4262-8bba-d9a4863737d4-dshm" (OuterVolumeSpecName: "dshm") pod "fc9e1da3-e02a-4262-8bba-d9a4863737d4" (UID: "fc9e1da3-e02a-4262-8bba-d9a4863737d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:17:25.185035 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.184974    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc9e1da3-e02a-4262-8bba-d9a4863737d4-kube-api-access-p2t9v" (OuterVolumeSpecName: "kube-api-access-p2t9v") pod "fc9e1da3-e02a-4262-8bba-d9a4863737d4" (UID: "fc9e1da3-e02a-4262-8bba-d9a4863737d4"). InnerVolumeSpecName "kube-api-access-p2t9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:17:25.283789 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.283754    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc9e1da3-e02a-4262-8bba-d9a4863737d4-tls-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:17:25.283789 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.283784    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc9e1da3-e02a-4262-8bba-d9a4863737d4-dshm\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:17:25.283789 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.283793    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2t9v\" (UniqueName: \"kubernetes.io/projected/fc9e1da3-e02a-4262-8bba-d9a4863737d4-kube-api-access-p2t9v\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:17:25.312484 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.312453    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"
Sep 15 21:17:25.312900 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.312491    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr" event={"ID":"39794398-9d10-4ef6-ad6e-2f85d3f490b0","Type":"ContainerDied","Data":"2c0723d124563b340cac418b7a3b2f619562d411c69b799761326f7bc7b791e3"}
Sep 15 21:17:25.312900 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.312546    2558 scope.go:117] "RemoveContainer" containerID="d9ae45284008fd6ab5c3644f0b6f1afd226c4215529319bbbb966fcc7f989a45"
Sep 15 21:17:25.314020 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.313997    2558 generic.go:356] "Generic (PLEG): container finished" podID="fc9e1da3-e02a-4262-8bba-d9a4863737d4" containerID="4a43473f568925bdad43e09ca111ef75bc945fdc85a861d55f1f8e1de930137b" exitCode=0
Sep 15 21:17:25.314131 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.314033    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"
Sep 15 21:17:25.314131 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.314092    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82" event={"ID":"fc9e1da3-e02a-4262-8bba-d9a4863737d4","Type":"ContainerDied","Data":"4a43473f568925bdad43e09ca111ef75bc945fdc85a861d55f1f8e1de930137b"}
Sep 15 21:17:25.314224 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.314133    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82" event={"ID":"fc9e1da3-e02a-4262-8bba-d9a4863737d4","Type":"ContainerDied","Data":"ab3996684ccf54300389afe41754a88ec8e32abeb076267f459060f2e406ea0c"}
Sep 15 21:17:25.323230 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.323213    2558 scope.go:117] "RemoveContainer" containerID="4a43473f568925bdad43e09ca111ef75bc945fdc85a861d55f1f8e1de930137b"
Sep 15 21:17:25.335000 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.334984    2558 scope.go:117] "RemoveContainer" containerID="4a43473f568925bdad43e09ca111ef75bc945fdc85a861d55f1f8e1de930137b"
Sep 15 21:17:25.335270 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:17:25.335252    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a43473f568925bdad43e09ca111ef75bc945fdc85a861d55f1f8e1de930137b\": container with ID starting with 4a43473f568925bdad43e09ca111ef75bc945fdc85a861d55f1f8e1de930137b not found: ID does not exist" containerID="4a43473f568925bdad43e09ca111ef75bc945fdc85a861d55f1f8e1de930137b"
Sep 15 21:17:25.335326 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.335278    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a43473f568925bdad43e09ca111ef75bc945fdc85a861d55f1f8e1de930137b"} err="failed to get container status \"4a43473f568925bdad43e09ca111ef75bc945fdc85a861d55f1f8e1de930137b\": rpc error: code = NotFound desc = could not find container \"4a43473f568925bdad43e09ca111ef75bc945fdc85a861d55f1f8e1de930137b\": container with ID starting with 4a43473f568925bdad43e09ca111ef75bc945fdc85a861d55f1f8e1de930137b not found: ID does not exist"
Sep 15 21:17:25.378483 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.378447    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"]
Sep 15 21:17:25.390750 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.390679    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0df719/test-model-0df719-kserve-7795895b8c-hnz82"]
Sep 15 21:17:25.419177 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.419151    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"]
Sep 15 21:17:25.435028 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.435001    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-6587cd/noten-model-6587cd-kserve-5449c57ddc-586cr"]
Sep 15 21:17:25.840010 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.839971    2558 patch_prober.go:28] interesting pod/e2e-mt-56c6b012-openshift-default-798965d66d-m265c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 15 21:17:25.840209 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.840040    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c" podUID="9d40317b-6afb-425c-a2d1-0afcbd5d8020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 15 21:17:25.943735 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.943690    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39794398-9d10-4ef6-ad6e-2f85d3f490b0" path="/var/lib/kubelet/pods/39794398-9d10-4ef6-ad6e-2f85d3f490b0/volumes"
Sep 15 21:17:25.944350 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:25.944323    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc9e1da3-e02a-4262-8bba-d9a4863737d4" path="/var/lib/kubelet/pods/fc9e1da3-e02a-4262-8bba-d9a4863737d4/volumes"
Sep 15 21:17:26.791893 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:26.791858    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:17:26.839416 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:26.839384    2558 patch_prober.go:28] interesting pod/e2e-mt-56c6b012-openshift-default-798965d66d-m265c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 15 21:17:26.839565 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:26.839443    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c" podUID="9d40317b-6afb-425c-a2d1-0afcbd5d8020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 15 21:17:27.839992 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:27.839956    2558 patch_prober.go:28] interesting pod/e2e-mt-56c6b012-openshift-default-798965d66d-m265c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 15 21:17:27.840403 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:27.840024    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c" podUID="9d40317b-6afb-425c-a2d1-0afcbd5d8020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 15 21:17:28.839713 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:28.839673    2558 patch_prober.go:28] interesting pod/e2e-mt-56c6b012-openshift-default-798965d66d-m265c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 15 21:17:28.839922 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:28.839741    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c" podUID="9d40317b-6afb-425c-a2d1-0afcbd5d8020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 15 21:17:29.844684 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:29.844653    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:17:29.845726 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:29.845703    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:17:40.634987 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:40.634902    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0df719-29825115-hhfbp"]
Sep 15 21:17:40.637634 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:40.637612    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0df719-29825115-hhfbp"]
Sep 15 21:17:41.941802 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:41.941770    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82838560-39b4-4124-8865-54277ea10b52" path="/var/lib/kubelet/pods/82838560-39b4-4124-8865-54277ea10b52/volumes"
Sep 15 21:17:47.695498 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:17:47.695462    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:18:05.582833 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:05.582797    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"]
Sep 15 21:18:05.583310 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:05.582993    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst" podUID="5de574df-a42f-4b09-ab0f-ba38a26d6968" containerName="istio-proxy" containerID="cri-o://66ea917bd61bbfe9cfca81e86c0ab4a97eeb24abfafd0c5d9c1f95877a7ce3a4" gracePeriod=30
Sep 15 21:18:06.372519 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.372485    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"]
Sep 15 21:18:06.372861 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.372848    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc9e1da3-e02a-4262-8bba-d9a4863737d4" containerName="main"
Sep 15 21:18:06.372923 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.372863    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc9e1da3-e02a-4262-8bba-d9a4863737d4" containerName="main"
Sep 15 21:18:06.372923 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.372873    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a0a00a2-0c0b-48b5-bd27-62168fc3a274" containerName="istio-proxy"
Sep 15 21:18:06.372923 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.372879    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a0a00a2-0c0b-48b5-bd27-62168fc3a274" containerName="istio-proxy"
Sep 15 21:18:06.372923 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.372889    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39794398-9d10-4ef6-ad6e-2f85d3f490b0" containerName="main"
Sep 15 21:18:06.372923 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.372894    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="39794398-9d10-4ef6-ad6e-2f85d3f490b0" containerName="main"
Sep 15 21:18:06.373117 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.372966    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="39794398-9d10-4ef6-ad6e-2f85d3f490b0" containerName="main"
Sep 15 21:18:06.373117 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.372975    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc9e1da3-e02a-4262-8bba-d9a4863737d4" containerName="main"
Sep 15 21:18:06.373117 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.372981    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a0a00a2-0c0b-48b5-bd27-62168fc3a274" containerName="istio-proxy"
Sep 15 21:18:06.377302 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.377284    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.380024 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.380002    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-2f34ef-openshift-default-dockercfg-259kc\""
Sep 15 21:18:06.390671 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.390647    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"]
Sep 15 21:18:06.475141 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.475106    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30f4dc5-6a5e-4474-be39-088181771d7c-istiod-ca-cert\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.475333 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.475163    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9qrq\" (UniqueName: \"kubernetes.io/projected/a30f4dc5-6a5e-4474-be39-088181771d7c-kube-api-access-q9qrq\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.475333 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.475193    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a30f4dc5-6a5e-4474-be39-088181771d7c-istio-token\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.475333 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.475228    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-credential-socket\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.475333 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.475302    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-istio-data\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.475505 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.475387    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30f4dc5-6a5e-4474-be39-088181771d7c-istio-podinfo\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.475505 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.475436    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-istio-envoy\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.475589 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.475530    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-workload-socket\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.475641 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.475603    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-workload-certs\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.576443 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.576413    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-workload-certs\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.576632 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.576452    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30f4dc5-6a5e-4474-be39-088181771d7c-istiod-ca-cert\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.576632 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.576479    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9qrq\" (UniqueName: \"kubernetes.io/projected/a30f4dc5-6a5e-4474-be39-088181771d7c-kube-api-access-q9qrq\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.576632 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.576498    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a30f4dc5-6a5e-4474-be39-088181771d7c-istio-token\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.576632 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.576517    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-credential-socket\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.576632 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.576535    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-istio-data\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.576632 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.576576    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30f4dc5-6a5e-4474-be39-088181771d7c-istio-podinfo\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.576632 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.576615    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-istio-envoy\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.576957 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.576657    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-workload-socket\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.577085 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.577063    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-istio-data\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.577166 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.577121    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-workload-certs\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.577224 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.577183    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-credential-socket\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.577297 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.577280    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-workload-socket\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.577344 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.577307    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30f4dc5-6a5e-4474-be39-088181771d7c-istiod-ca-cert\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.579550 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.579528    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-istio-envoy\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.579677 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.579659    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30f4dc5-6a5e-4474-be39-088181771d7c-istio-podinfo\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.585155 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.585134    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a30f4dc5-6a5e-4474-be39-088181771d7c-istio-token\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.585449 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.585393    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9qrq\" (UniqueName: \"kubernetes.io/projected/a30f4dc5-6a5e-4474-be39-088181771d7c-kube-api-access-q9qrq\") pod \"e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") " pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.687735 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.687712    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:06.967520 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.967492    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"]
Sep 15 21:18:06.968402 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:18:06.968375    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda30f4dc5_6a5e_4474_be39_088181771d7c.slice/crio-39991ed6ef435582880b0f730286c9088702838089320d93a61fc864a10e8eaf WatchSource:0}: Error finding container 39991ed6ef435582880b0f730286c9088702838089320d93a61fc864a10e8eaf: Status 404 returned error can't find the container with id 39991ed6ef435582880b0f730286c9088702838089320d93a61fc864a10e8eaf
Sep 15 21:18:06.971103 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.971069    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:18:06.971223 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.971144    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:18:06.971223 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:06.971189    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:18:07.194356 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:07.194321    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:18:07.548845 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:07.548810    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh" event={"ID":"a30f4dc5-6a5e-4474-be39-088181771d7c","Type":"ContainerStarted","Data":"0a40e8340d3e4f187668145c31919714194172ff8fac9df231af79ace4e73258"}
Sep 15 21:18:07.549027 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:07.548897    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh" event={"ID":"a30f4dc5-6a5e-4474-be39-088181771d7c","Type":"ContainerStarted","Data":"39991ed6ef435582880b0f730286c9088702838089320d93a61fc864a10e8eaf"}
Sep 15 21:18:07.575290 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:07.575237    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh" podStartSLOduration=1.575218523 podStartE2EDuration="1.575218523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:07.574407025 +0000 UTC m=+1206.065052756" watchObservedRunningTime="2026-09-15 21:18:07.575218523 +0000 UTC m=+1206.065864253"
Sep 15 21:18:07.688160 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:07.688114    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:08.693948 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:08.693915    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:09.559937 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:09.559899    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:09.560770 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:09.560749    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:10.833337 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:10.833314    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:18:10.920526 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:10.920497    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-credential-socket\") pod \"5de574df-a42f-4b09-ab0f-ba38a26d6968\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") "
Sep 15 21:18:10.920526 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:10.920528    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-workload-certs\") pod \"5de574df-a42f-4b09-ab0f-ba38a26d6968\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") "
Sep 15 21:18:10.920778 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:10.920598    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-workload-socket\") pod \"5de574df-a42f-4b09-ab0f-ba38a26d6968\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") "
Sep 15 21:18:10.920778 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:10.920626    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfbqq\" (UniqueName: \"kubernetes.io/projected/5de574df-a42f-4b09-ab0f-ba38a26d6968-kube-api-access-qfbqq\") pod \"5de574df-a42f-4b09-ab0f-ba38a26d6968\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") "
Sep 15 21:18:10.920778 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:10.920663    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-istio-envoy\") pod \"5de574df-a42f-4b09-ab0f-ba38a26d6968\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") "
Sep 15 21:18:10.920778 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:10.920683    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de574df-a42f-4b09-ab0f-ba38a26d6968-istio-token\") pod \"5de574df-a42f-4b09-ab0f-ba38a26d6968\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") "
Sep 15 21:18:10.920778 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:10.920705    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-istio-data\") pod \"5de574df-a42f-4b09-ab0f-ba38a26d6968\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") "
Sep 15 21:18:10.920778 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:10.920728    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de574df-a42f-4b09-ab0f-ba38a26d6968-istio-podinfo\") pod \"5de574df-a42f-4b09-ab0f-ba38a26d6968\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") "
Sep 15 21:18:10.920778 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:10.920754    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de574df-a42f-4b09-ab0f-ba38a26d6968-istiod-ca-cert\") pod \"5de574df-a42f-4b09-ab0f-ba38a26d6968\" (UID: \"5de574df-a42f-4b09-ab0f-ba38a26d6968\") "
Sep 15 21:18:10.921184 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:10.920844    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5de574df-a42f-4b09-ab0f-ba38a26d6968" (UID: "5de574df-a42f-4b09-ab0f-ba38a26d6968"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:18:10.921184 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:10.920920    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5de574df-a42f-4b09-ab0f-ba38a26d6968" (UID: "5de574df-a42f-4b09-ab0f-ba38a26d6968"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:18:10.921184 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:10.920933    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5de574df-a42f-4b09-ab0f-ba38a26d6968" (UID: "5de574df-a42f-4b09-ab0f-ba38a26d6968"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:18:10.921184 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:10.921128    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-workload-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:18:10.921184 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:10.921144    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-credential-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:18:10.921184 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:10.921153    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-workload-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:18:10.921420 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:10.921222    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-istio-data" (OuterVolumeSpecName: "istio-data") pod "5de574df-a42f-4b09-ab0f-ba38a26d6968" (UID: "5de574df-a42f-4b09-ab0f-ba38a26d6968"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:18:10.921420 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:10.921255    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5de574df-a42f-4b09-ab0f-ba38a26d6968-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5de574df-a42f-4b09-ab0f-ba38a26d6968" (UID: "5de574df-a42f-4b09-ab0f-ba38a26d6968"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:18:10.923065 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:10.923033    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5de574df-a42f-4b09-ab0f-ba38a26d6968" (UID: "5de574df-a42f-4b09-ab0f-ba38a26d6968"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:18:10.923065 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:10.923039    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de574df-a42f-4b09-ab0f-ba38a26d6968-istio-token" (OuterVolumeSpecName: "istio-token") pod "5de574df-a42f-4b09-ab0f-ba38a26d6968" (UID: "5de574df-a42f-4b09-ab0f-ba38a26d6968"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:18:10.923355 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:10.923336    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5de574df-a42f-4b09-ab0f-ba38a26d6968-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5de574df-a42f-4b09-ab0f-ba38a26d6968" (UID: "5de574df-a42f-4b09-ab0f-ba38a26d6968"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:18:10.923429 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:10.923337    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de574df-a42f-4b09-ab0f-ba38a26d6968-kube-api-access-qfbqq" (OuterVolumeSpecName: "kube-api-access-qfbqq") pod "5de574df-a42f-4b09-ab0f-ba38a26d6968" (UID: "5de574df-a42f-4b09-ab0f-ba38a26d6968"). InnerVolumeSpecName "kube-api-access-qfbqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:18:11.022569 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:11.022537    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfbqq\" (UniqueName: \"kubernetes.io/projected/5de574df-a42f-4b09-ab0f-ba38a26d6968-kube-api-access-qfbqq\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:18:11.022569 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:11.022566    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-istio-envoy\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:18:11.022766 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:11.022581    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de574df-a42f-4b09-ab0f-ba38a26d6968-istio-token\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:18:11.022766 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:11.022594    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de574df-a42f-4b09-ab0f-ba38a26d6968-istio-data\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:18:11.022766 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:11.022606    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de574df-a42f-4b09-ab0f-ba38a26d6968-istio-podinfo\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:18:11.022766 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:11.022617    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de574df-a42f-4b09-ab0f-ba38a26d6968-istiod-ca-cert\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:18:11.572423 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:11.572391    2558 generic.go:356] "Generic (PLEG): container finished" podID="5de574df-a42f-4b09-ab0f-ba38a26d6968" containerID="66ea917bd61bbfe9cfca81e86c0ab4a97eeb24abfafd0c5d9c1f95877a7ce3a4" exitCode=0
Sep 15 21:18:11.572613 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:11.572439    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"
Sep 15 21:18:11.572613 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:11.572473    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst" event={"ID":"5de574df-a42f-4b09-ab0f-ba38a26d6968","Type":"ContainerDied","Data":"66ea917bd61bbfe9cfca81e86c0ab4a97eeb24abfafd0c5d9c1f95877a7ce3a4"}
Sep 15 21:18:11.572613 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:11.572532    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst" event={"ID":"5de574df-a42f-4b09-ab0f-ba38a26d6968","Type":"ContainerDied","Data":"12c9c3e2c03092da5e194989037a8c2aba77ec86415fb5b8bd26d032905baa1d"}
Sep 15 21:18:11.572613 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:11.572551    2558 scope.go:117] "RemoveContainer" containerID="66ea917bd61bbfe9cfca81e86c0ab4a97eeb24abfafd0c5d9c1f95877a7ce3a4"
Sep 15 21:18:11.582219 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:11.582203    2558 scope.go:117] "RemoveContainer" containerID="868e6be7774c9b2972671d0c08024db5f6ec63ef2962b6c4392e48d1abac2cbb"
Sep 15 21:18:11.590508 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:11.590489    2558 scope.go:117] "RemoveContainer" containerID="66ea917bd61bbfe9cfca81e86c0ab4a97eeb24abfafd0c5d9c1f95877a7ce3a4"
Sep 15 21:18:11.590791 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:18:11.590771    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ea917bd61bbfe9cfca81e86c0ab4a97eeb24abfafd0c5d9c1f95877a7ce3a4\": container with ID starting with 66ea917bd61bbfe9cfca81e86c0ab4a97eeb24abfafd0c5d9c1f95877a7ce3a4 not found: ID does not exist" containerID="66ea917bd61bbfe9cfca81e86c0ab4a97eeb24abfafd0c5d9c1f95877a7ce3a4"
Sep 15 21:18:11.590863 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:11.590797    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ea917bd61bbfe9cfca81e86c0ab4a97eeb24abfafd0c5d9c1f95877a7ce3a4"} err="failed to get container status \"66ea917bd61bbfe9cfca81e86c0ab4a97eeb24abfafd0c5d9c1f95877a7ce3a4\": rpc error: code = NotFound desc = could not find container \"66ea917bd61bbfe9cfca81e86c0ab4a97eeb24abfafd0c5d9c1f95877a7ce3a4\": container with ID starting with 66ea917bd61bbfe9cfca81e86c0ab4a97eeb24abfafd0c5d9c1f95877a7ce3a4 not found: ID does not exist"
Sep 15 21:18:11.590863 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:11.590813    2558 scope.go:117] "RemoveContainer" containerID="868e6be7774c9b2972671d0c08024db5f6ec63ef2962b6c4392e48d1abac2cbb"
Sep 15 21:18:11.591041 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:18:11.591027    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"868e6be7774c9b2972671d0c08024db5f6ec63ef2962b6c4392e48d1abac2cbb\": container with ID starting with 868e6be7774c9b2972671d0c08024db5f6ec63ef2962b6c4392e48d1abac2cbb not found: ID does not exist" containerID="868e6be7774c9b2972671d0c08024db5f6ec63ef2962b6c4392e48d1abac2cbb"
Sep 15 21:18:11.591148 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:11.591044    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868e6be7774c9b2972671d0c08024db5f6ec63ef2962b6c4392e48d1abac2cbb"} err="failed to get container status \"868e6be7774c9b2972671d0c08024db5f6ec63ef2962b6c4392e48d1abac2cbb\": rpc error: code = NotFound desc = could not find container \"868e6be7774c9b2972671d0c08024db5f6ec63ef2962b6c4392e48d1abac2cbb\": container with ID starting with 868e6be7774c9b2972671d0c08024db5f6ec63ef2962b6c4392e48d1abac2cbb not found: ID does not exist"
Sep 15 21:18:11.595904 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:11.595880    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"]
Sep 15 21:18:11.601484 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:11.601462    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-253ca1-openshift-default-547d79bd86-fqhst"]
Sep 15 21:18:11.948329 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:11.948298    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5de574df-a42f-4b09-ab0f-ba38a26d6968" path="/var/lib/kubelet/pods/5de574df-a42f-4b09-ab0f-ba38a26d6968/volumes"
Sep 15 21:18:13.252237 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:13.252203    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"]
Sep 15 21:18:13.253354 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:13.253327    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5de574df-a42f-4b09-ab0f-ba38a26d6968" containerName="istio-proxy"
Sep 15 21:18:13.253515 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:13.253501    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5de574df-a42f-4b09-ab0f-ba38a26d6968" containerName="istio-proxy"
Sep 15 21:18:13.253632 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:13.253619    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5de574df-a42f-4b09-ab0f-ba38a26d6968" containerName="istio-proxy"
Sep 15 21:18:13.253721 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:13.253710    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5de574df-a42f-4b09-ab0f-ba38a26d6968" containerName="istio-proxy"
Sep 15 21:18:13.253930 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:13.253915    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5de574df-a42f-4b09-ab0f-ba38a26d6968" containerName="istio-proxy"
Sep 15 21:18:13.254027 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:13.254017    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5de574df-a42f-4b09-ab0f-ba38a26d6968" containerName="istio-proxy"
Sep 15 21:18:13.258677 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:13.258656    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"
Sep 15 21:18:13.261612 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:13.261588    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-2f34ef-serving-cert\""
Sep 15 21:18:13.268862 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:13.268837    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"]
Sep 15 21:18:13.344498 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:13.344462    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff08845a-385f-42be-8231-eb324d9faeb1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg\" (UID: \"ff08845a-385f-42be-8231-eb324d9faeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"
Sep 15 21:18:13.344655 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:13.344505    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h4wz\" (UniqueName: \"kubernetes.io/projected/ff08845a-385f-42be-8231-eb324d9faeb1-kube-api-access-9h4wz\") pod \"maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg\" (UID: \"ff08845a-385f-42be-8231-eb324d9faeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"
Sep 15 21:18:13.344655 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:13.344564    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff08845a-385f-42be-8231-eb324d9faeb1-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg\" (UID: \"ff08845a-385f-42be-8231-eb324d9faeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"
Sep 15 21:18:13.344748 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:13.344666    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff08845a-385f-42be-8231-eb324d9faeb1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg\" (UID: \"ff08845a-385f-42be-8231-eb324d9faeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"
Sep 15 21:18:13.445963 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:13.445925    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h4wz\" (UniqueName: \"kubernetes.io/projected/ff08845a-385f-42be-8231-eb324d9faeb1-kube-api-access-9h4wz\") pod \"maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg\" (UID: \"ff08845a-385f-42be-8231-eb324d9faeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"
Sep 15 21:18:13.445963 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:13.445968    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff08845a-385f-42be-8231-eb324d9faeb1-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg\" (UID: \"ff08845a-385f-42be-8231-eb324d9faeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"
Sep 15 21:18:13.446241 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:13.446023    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff08845a-385f-42be-8231-eb324d9faeb1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg\" (UID: \"ff08845a-385f-42be-8231-eb324d9faeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"
Sep 15 21:18:13.446241 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:13.446112    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff08845a-385f-42be-8231-eb324d9faeb1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg\" (UID: \"ff08845a-385f-42be-8231-eb324d9faeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"
Sep 15 21:18:13.446241 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:18:13.446194    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-serving-cert: secret "maas-api-e2e-shared-a-w3-2f34ef-serving-cert" not found
Sep 15 21:18:13.446388 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:18:13.446250    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff08845a-385f-42be-8231-eb324d9faeb1-maas-api-tls podName:ff08845a-385f-42be-8231-eb324d9faeb1 nodeName:}" failed. No retries permitted until 2026-09-15 21:18:13.946235963 +0000 UTC m=+1212.436881686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ff08845a-385f-42be-8231-eb324d9faeb1-maas-api-tls") pod "maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg" (UID: "ff08845a-385f-42be-8231-eb324d9faeb1") : secret "maas-api-e2e-shared-a-w3-2f34ef-serving-cert" not found
Sep 15 21:18:13.446748 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:13.446727    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff08845a-385f-42be-8231-eb324d9faeb1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg\" (UID: \"ff08845a-385f-42be-8231-eb324d9faeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"
Sep 15 21:18:13.448633 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:13.448610    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff08845a-385f-42be-8231-eb324d9faeb1-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg\" (UID: \"ff08845a-385f-42be-8231-eb324d9faeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"
Sep 15 21:18:13.458846 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:13.458822    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h4wz\" (UniqueName: \"kubernetes.io/projected/ff08845a-385f-42be-8231-eb324d9faeb1-kube-api-access-9h4wz\") pod \"maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg\" (UID: \"ff08845a-385f-42be-8231-eb324d9faeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"
Sep 15 21:18:13.951177 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:13.951147    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff08845a-385f-42be-8231-eb324d9faeb1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg\" (UID: \"ff08845a-385f-42be-8231-eb324d9faeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"
Sep 15 21:18:13.953506 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:13.953480    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff08845a-385f-42be-8231-eb324d9faeb1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg\" (UID: \"ff08845a-385f-42be-8231-eb324d9faeb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"
Sep 15 21:18:14.170080 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:14.170024    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"
Sep 15 21:18:14.441794 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:14.441767    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"]
Sep 15 21:18:14.442865 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:18:14.442842    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff08845a_385f_42be_8231_eb324d9faeb1.slice/crio-f37efd9559289631ee00cba9926dcdaa3984f5466943e76af1535c67c0132ca2 WatchSource:0}: Error finding container f37efd9559289631ee00cba9926dcdaa3984f5466943e76af1535c67c0132ca2: Status 404 returned error can't find the container with id f37efd9559289631ee00cba9926dcdaa3984f5466943e76af1535c67c0132ca2
Sep 15 21:18:14.592481 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:14.592397    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg" event={"ID":"ff08845a-385f-42be-8231-eb324d9faeb1","Type":"ContainerStarted","Data":"f37efd9559289631ee00cba9926dcdaa3984f5466943e76af1535c67c0132ca2"}
Sep 15 21:18:15.599727 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:15.599692    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg" event={"ID":"ff08845a-385f-42be-8231-eb324d9faeb1","Type":"ContainerStarted","Data":"053f5333bc0133f5200c4c022d2181a019bfe9a033031d1d73760063a2130cbd"}
Sep 15 21:18:15.600136 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:15.599757    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"
Sep 15 21:18:15.623294 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:15.623251    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg" podStartSLOduration=2.2173566510000002 podStartE2EDuration="2.623236955s" totalImagesPullingTime="405.880304ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:15.621087953 +0000 UTC m=+1214.111733692" watchObservedRunningTime="2026-09-15 21:18:15.623236955 +0000 UTC m=+1214.113882690"
Sep 15 21:18:21.610296 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:21.610267    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"
Sep 15 21:18:21.798552 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:21.798517    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:18:22.183426 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.183379    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"]
Sep 15 21:18:22.187195 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.187168    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:18:22.190249 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.190221    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-56c6b012\"/\"kube-root-ca.crt\""
Sep 15 21:18:22.190388 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.190302    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-56c6b012\"/\"e2e-lifecycle-model-56c6b012-kserve-self-signed-certs\""
Sep 15 21:18:22.191445 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.191421    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-56c6b012\"/\"default-dockercfg-h4vcm\""
Sep 15 21:18:22.191545 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.191472    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-56c6b012\"/\"openshift-service-ca.crt\""
Sep 15 21:18:22.196813 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.196791    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"]
Sep 15 21:18:22.326416 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.326387    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6d05190-d607-48a5-98f7-dd41758df7dc-tmp-dir\") pod \"e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq\" (UID: \"a6d05190-d607-48a5-98f7-dd41758df7dc\") " pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:18:22.326612 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.326432    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6d05190-d607-48a5-98f7-dd41758df7dc-dshm\") pod \"e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq\" (UID: \"a6d05190-d607-48a5-98f7-dd41758df7dc\") " pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:18:22.326612 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.326511    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6d05190-d607-48a5-98f7-dd41758df7dc-model-cache\") pod \"e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq\" (UID: \"a6d05190-d607-48a5-98f7-dd41758df7dc\") " pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:18:22.326612 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.326552    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znhdb\" (UniqueName: \"kubernetes.io/projected/a6d05190-d607-48a5-98f7-dd41758df7dc-kube-api-access-znhdb\") pod \"e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq\" (UID: \"a6d05190-d607-48a5-98f7-dd41758df7dc\") " pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:18:22.326612 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.326590    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6d05190-d607-48a5-98f7-dd41758df7dc-home\") pod \"e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq\" (UID: \"a6d05190-d607-48a5-98f7-dd41758df7dc\") " pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:18:22.326812 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.326630    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6d05190-d607-48a5-98f7-dd41758df7dc-tls-certs\") pod \"e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq\" (UID: \"a6d05190-d607-48a5-98f7-dd41758df7dc\") " pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:18:22.427772 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.427716    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6d05190-d607-48a5-98f7-dd41758df7dc-home\") pod \"e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq\" (UID: \"a6d05190-d607-48a5-98f7-dd41758df7dc\") " pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:18:22.427941 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.427786    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6d05190-d607-48a5-98f7-dd41758df7dc-tls-certs\") pod \"e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq\" (UID: \"a6d05190-d607-48a5-98f7-dd41758df7dc\") " pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:18:22.427941 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.427865    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6d05190-d607-48a5-98f7-dd41758df7dc-tmp-dir\") pod \"e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq\" (UID: \"a6d05190-d607-48a5-98f7-dd41758df7dc\") " pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:18:22.427941 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.427900    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6d05190-d607-48a5-98f7-dd41758df7dc-dshm\") pod \"e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq\" (UID: \"a6d05190-d607-48a5-98f7-dd41758df7dc\") " pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:18:22.427941 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.427926    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6d05190-d607-48a5-98f7-dd41758df7dc-model-cache\") pod \"e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq\" (UID: \"a6d05190-d607-48a5-98f7-dd41758df7dc\") " pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:18:22.428188 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.427943    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znhdb\" (UniqueName: \"kubernetes.io/projected/a6d05190-d607-48a5-98f7-dd41758df7dc-kube-api-access-znhdb\") pod \"e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq\" (UID: \"a6d05190-d607-48a5-98f7-dd41758df7dc\") " pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:18:22.428250 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.428185    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6d05190-d607-48a5-98f7-dd41758df7dc-home\") pod \"e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq\" (UID: \"a6d05190-d607-48a5-98f7-dd41758df7dc\") " pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:18:22.428250 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.428223    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6d05190-d607-48a5-98f7-dd41758df7dc-tmp-dir\") pod \"e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq\" (UID: \"a6d05190-d607-48a5-98f7-dd41758df7dc\") " pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:18:22.428366 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.428347    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6d05190-d607-48a5-98f7-dd41758df7dc-model-cache\") pod \"e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq\" (UID: \"a6d05190-d607-48a5-98f7-dd41758df7dc\") " pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:18:22.430089 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.430040    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6d05190-d607-48a5-98f7-dd41758df7dc-dshm\") pod \"e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq\" (UID: \"a6d05190-d607-48a5-98f7-dd41758df7dc\") " pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:18:22.430319 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.430299    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6d05190-d607-48a5-98f7-dd41758df7dc-tls-certs\") pod \"e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq\" (UID: \"a6d05190-d607-48a5-98f7-dd41758df7dc\") " pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:18:22.436591 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.436546    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znhdb\" (UniqueName: \"kubernetes.io/projected/a6d05190-d607-48a5-98f7-dd41758df7dc-kube-api-access-znhdb\") pod \"e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq\" (UID: \"a6d05190-d607-48a5-98f7-dd41758df7dc\") " pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:18:22.498762 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.498724    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:18:22.783849 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:18:22.783813    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6d05190_d607_48a5_98f7_dd41758df7dc.slice/crio-0c6f07be5a23cea8da1c5d37f1e5d7102a878878b94bfe7ef8f4ed28770f76f7 WatchSource:0}: Error finding container 0c6f07be5a23cea8da1c5d37f1e5d7102a878878b94bfe7ef8f4ed28770f76f7: Status 404 returned error can't find the container with id 0c6f07be5a23cea8da1c5d37f1e5d7102a878878b94bfe7ef8f4ed28770f76f7
Sep 15 21:18:22.784215 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:22.783901    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"]
Sep 15 21:18:23.646323 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:23.646293    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq" event={"ID":"a6d05190-d607-48a5-98f7-dd41758df7dc","Type":"ContainerStarted","Data":"49e9d1bcb6e4754f4507c40874d7b5ed7cb692b55abb2cb33543570ea8668250"}
Sep 15 21:18:23.646517 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:23.646333    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq" event={"ID":"a6d05190-d607-48a5-98f7-dd41758df7dc","Type":"ContainerStarted","Data":"0c6f07be5a23cea8da1c5d37f1e5d7102a878878b94bfe7ef8f4ed28770f76f7"}
Sep 15 21:18:23.683430 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:23.683376    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq" podStartSLOduration=1.683360931 podStartE2EDuration="1.683360931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:23.680578238 +0000 UTC m=+1222.171223967" watchObservedRunningTime="2026-09-15 21:18:23.683360931 +0000 UTC m=+1222.174006663"
Sep 15 21:18:32.499467 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:32.499428    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:18:32.501891 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:32.499512    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:18:32.512348 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:32.512328    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:18:32.710591 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:32.710518    2558 generic.go:356] "Generic (PLEG): container finished" podID="a30f4dc5-6a5e-4474-be39-088181771d7c" containerID="0a40e8340d3e4f187668145c31919714194172ff8fac9df231af79ace4e73258" exitCode=0
Sep 15 21:18:32.710755 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:32.710590    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh" event={"ID":"a30f4dc5-6a5e-4474-be39-088181771d7c","Type":"ContainerDied","Data":"0a40e8340d3e4f187668145c31919714194172ff8fac9df231af79ace4e73258"}
Sep 15 21:18:32.711142 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:32.711123    2558 scope.go:117] "RemoveContainer" containerID="0a40e8340d3e4f187668145c31919714194172ff8fac9df231af79ace4e73258"
Sep 15 21:18:32.712228 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:32.712190    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:18:32.712307 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:32.712272    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:18:32.712366 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:32.712308    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:18:32.722672 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:32.722655    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:18:33.718848 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:33.718814    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh" event={"ID":"a30f4dc5-6a5e-4474-be39-088181771d7c","Type":"ContainerStarted","Data":"e45104aaddd06912411a3a272a325e56f0857da716a512019455a59e0df3580c"}
Sep 15 21:18:34.688912 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:34.688869    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:34.689153 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:34.689128    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 15 21:18:34.689204 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:34.689184    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh" podUID="a30f4dc5-6a5e-4474-be39-088181771d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 15 21:18:35.688316 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:35.688281    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 15 21:18:35.688701 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:35.688347    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh" podUID="a30f4dc5-6a5e-4474-be39-088181771d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 15 21:18:36.688411 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:36.688370    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 15 21:18:36.688803 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:36.688437    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh" podUID="a30f4dc5-6a5e-4474-be39-088181771d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 15 21:18:37.688345 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:37.688315    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 15 21:18:37.688518 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:37.688367    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh" podUID="a30f4dc5-6a5e-4474-be39-088181771d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 15 21:18:38.688574 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:38.688541    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 15 21:18:38.688932 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:38.688607    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh" podUID="a30f4dc5-6a5e-4474-be39-088181771d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 15 21:18:39.560322 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:39.560267    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:39.692530 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:39.692499    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:39.754514 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:39.754478    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:18:40.603117 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:40.603017    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:18:47.803986 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:47.803954    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9"]
Sep 15 21:18:47.808723 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:47.808699    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9"
Sep 15 21:18:47.812603 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:47.812585    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-41ffdf-serving-cert\""
Sep 15 21:18:47.829826 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:47.829803    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9"]
Sep 15 21:18:47.898031 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:47.897979    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23eece0-f61d-4cae-89ba-e0487a370327-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9\" (UID: \"e23eece0-f61d-4cae-89ba-e0487a370327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9"
Sep 15 21:18:47.898228 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:47.898063    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e23eece0-f61d-4cae-89ba-e0487a370327-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9\" (UID: \"e23eece0-f61d-4cae-89ba-e0487a370327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9"
Sep 15 21:18:47.898228 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:47.898111    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flj4x\" (UniqueName: \"kubernetes.io/projected/e23eece0-f61d-4cae-89ba-e0487a370327-kube-api-access-flj4x\") pod \"maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9\" (UID: \"e23eece0-f61d-4cae-89ba-e0487a370327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9"
Sep 15 21:18:47.898353 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:47.898240    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e23eece0-f61d-4cae-89ba-e0487a370327-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9\" (UID: \"e23eece0-f61d-4cae-89ba-e0487a370327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9"
Sep 15 21:18:47.999215 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:47.999186    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e23eece0-f61d-4cae-89ba-e0487a370327-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9\" (UID: \"e23eece0-f61d-4cae-89ba-e0487a370327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9"
Sep 15 21:18:47.999408 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:47.999228    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23eece0-f61d-4cae-89ba-e0487a370327-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9\" (UID: \"e23eece0-f61d-4cae-89ba-e0487a370327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9"
Sep 15 21:18:47.999408 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:47.999268    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e23eece0-f61d-4cae-89ba-e0487a370327-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9\" (UID: \"e23eece0-f61d-4cae-89ba-e0487a370327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9"
Sep 15 21:18:47.999408 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:47.999293    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flj4x\" (UniqueName: \"kubernetes.io/projected/e23eece0-f61d-4cae-89ba-e0487a370327-kube-api-access-flj4x\") pod \"maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9\" (UID: \"e23eece0-f61d-4cae-89ba-e0487a370327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9"
Sep 15 21:18:48.000075 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:48.000031    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23eece0-f61d-4cae-89ba-e0487a370327-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9\" (UID: \"e23eece0-f61d-4cae-89ba-e0487a370327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9"
Sep 15 21:18:48.001777 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:48.001754    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e23eece0-f61d-4cae-89ba-e0487a370327-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9\" (UID: \"e23eece0-f61d-4cae-89ba-e0487a370327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9"
Sep 15 21:18:48.001892 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:48.001841    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e23eece0-f61d-4cae-89ba-e0487a370327-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9\" (UID: \"e23eece0-f61d-4cae-89ba-e0487a370327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9"
Sep 15 21:18:48.018164 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:48.018144    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flj4x\" (UniqueName: \"kubernetes.io/projected/e23eece0-f61d-4cae-89ba-e0487a370327-kube-api-access-flj4x\") pod \"maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9\" (UID: \"e23eece0-f61d-4cae-89ba-e0487a370327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9"
Sep 15 21:18:48.119388 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:48.119308    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9"
Sep 15 21:18:48.415975 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:48.415950    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9"]
Sep 15 21:18:48.417714 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:18:48.417684    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23eece0_f61d_4cae_89ba_e0487a370327.slice/crio-9d8783351a9a108112e43b84b541673e1b2585256870f6b4dbd4060300b4c00f WatchSource:0}: Error finding container 9d8783351a9a108112e43b84b541673e1b2585256870f6b4dbd4060300b4c00f: Status 404 returned error can't find the container with id 9d8783351a9a108112e43b84b541673e1b2585256870f6b4dbd4060300b4c00f
Sep 15 21:18:48.804020 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:48.803990    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9" event={"ID":"e23eece0-f61d-4cae-89ba-e0487a370327","Type":"ContainerStarted","Data":"9d8783351a9a108112e43b84b541673e1b2585256870f6b4dbd4060300b4c00f"}
Sep 15 21:18:49.811986 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:49.811953    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9" event={"ID":"e23eece0-f61d-4cae-89ba-e0487a370327","Type":"ContainerStarted","Data":"7a4d8643e16b1d080457001d290239f1e399d377a9a2d4221c695d587db2ce0d"}
Sep 15 21:18:49.812367 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:49.812095    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9"
Sep 15 21:18:49.831132 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:49.831088    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9" podStartSLOduration=2.4748804079999998 podStartE2EDuration="2.831073676s" totalImagesPullingTime="356.193268ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:49.829224559 +0000 UTC m=+1248.319870291" watchObservedRunningTime="2026-09-15 21:18:49.831073676 +0000 UTC m=+1248.321719409"
Sep 15 21:18:55.823688 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:18:55.823611    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9"
Sep 15 21:19:12.614623 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.614584    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"]
Sep 15 21:19:12.619930 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.619907    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:19:12.624312 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.624257    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2f34ef\"/\"auth-test-model-2f34ef-kserve-self-signed-certs\""
Sep 15 21:19:12.624312 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.624272    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2f34ef\"/\"openshift-service-ca.crt\""
Sep 15 21:19:12.624514 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.624393    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2f34ef\"/\"default-dockercfg-fxdnv\""
Sep 15 21:19:12.624514 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.624495    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2f34ef\"/\"kube-root-ca.crt\""
Sep 15 21:19:12.628382 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.628223    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"]
Sep 15 21:19:12.641937 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.641913    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63e7736a-4199-4a75-8b09-b6cfd063c1e6-tmp-dir\") pod \"auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64\" (UID: \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:19:12.642042 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.641954    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63e7736a-4199-4a75-8b09-b6cfd063c1e6-dshm\") pod \"auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64\" (UID: \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:19:12.642106 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.642076    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48xd6\" (UniqueName: \"kubernetes.io/projected/63e7736a-4199-4a75-8b09-b6cfd063c1e6-kube-api-access-48xd6\") pod \"auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64\" (UID: \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:19:12.642144 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.642127    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63e7736a-4199-4a75-8b09-b6cfd063c1e6-model-cache\") pod \"auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64\" (UID: \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:19:12.642183 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.642158    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63e7736a-4199-4a75-8b09-b6cfd063c1e6-home\") pod \"auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64\" (UID: \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:19:12.642218 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.642198    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63e7736a-4199-4a75-8b09-b6cfd063c1e6-tls-certs\") pod \"auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64\" (UID: \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:19:12.743414 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.743380    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63e7736a-4199-4a75-8b09-b6cfd063c1e6-dshm\") pod \"auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64\" (UID: \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:19:12.743572 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.743469    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48xd6\" (UniqueName: \"kubernetes.io/projected/63e7736a-4199-4a75-8b09-b6cfd063c1e6-kube-api-access-48xd6\") pod \"auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64\" (UID: \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:19:12.743572 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.743528    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63e7736a-4199-4a75-8b09-b6cfd063c1e6-model-cache\") pod \"auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64\" (UID: \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:19:12.743572 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.743560    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63e7736a-4199-4a75-8b09-b6cfd063c1e6-home\") pod \"auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64\" (UID: \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:19:12.743734 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.743607    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63e7736a-4199-4a75-8b09-b6cfd063c1e6-tls-certs\") pod \"auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64\" (UID: \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:19:12.743734 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.743649    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63e7736a-4199-4a75-8b09-b6cfd063c1e6-tmp-dir\") pod \"auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64\" (UID: \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:19:12.743965 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.743932    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63e7736a-4199-4a75-8b09-b6cfd063c1e6-model-cache\") pod \"auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64\" (UID: \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:19:12.743965 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.743947    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63e7736a-4199-4a75-8b09-b6cfd063c1e6-home\") pod \"auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64\" (UID: \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:19:12.744187 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.744040    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63e7736a-4199-4a75-8b09-b6cfd063c1e6-tmp-dir\") pod \"auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64\" (UID: \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:19:12.745682 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.745660    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63e7736a-4199-4a75-8b09-b6cfd063c1e6-dshm\") pod \"auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64\" (UID: \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:19:12.746014 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.745995    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63e7736a-4199-4a75-8b09-b6cfd063c1e6-tls-certs\") pod \"auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64\" (UID: \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:19:12.752005 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.751980    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48xd6\" (UniqueName: \"kubernetes.io/projected/63e7736a-4199-4a75-8b09-b6cfd063c1e6-kube-api-access-48xd6\") pod \"auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64\" (UID: \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:19:12.930769 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:12.930738    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:19:13.000469 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:13.000442    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:19:13.202194 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:13.202165    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"]
Sep 15 21:19:13.203205 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:19:13.203169    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63e7736a_4199_4a75_8b09_b6cfd063c1e6.slice/crio-cdcddd5284667517df1dd61675b8e9873d9248023ee8b61aecbd33031c505e8a WatchSource:0}: Error finding container cdcddd5284667517df1dd61675b8e9873d9248023ee8b61aecbd33031c505e8a: Status 404 returned error can't find the container with id cdcddd5284667517df1dd61675b8e9873d9248023ee8b61aecbd33031c505e8a
Sep 15 21:19:13.951410 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:13.951379    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64" event={"ID":"63e7736a-4199-4a75-8b09-b6cfd063c1e6","Type":"ContainerStarted","Data":"e80aa38f2715816f4e2699482520fe1652d81045bef55e981277ab3490c998d6"}
Sep 15 21:19:13.951410 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:13.951416    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64" event={"ID":"63e7736a-4199-4a75-8b09-b6cfd063c1e6","Type":"ContainerStarted","Data":"cdcddd5284667517df1dd61675b8e9873d9248023ee8b61aecbd33031c505e8a"}
Sep 15 21:19:13.969011 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:13.968962    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64" podStartSLOduration=1.9689464829999999 podStartE2EDuration="1.968946483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:19:13.967293058 +0000 UTC m=+1272.457938789" watchObservedRunningTime="2026-09-15 21:19:13.968946483 +0000 UTC m=+1272.459592248"
Sep 15 21:19:22.931249 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:22.931210    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:19:22.931651 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:22.931264    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:19:22.943734 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:22.943708    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:19:23.011698 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:23.011673    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:19:25.005429 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:25.005393    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"]
Sep 15 21:19:40.025879 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:40.025811    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq" podUID="a6d05190-d607-48a5-98f7-dd41758df7dc" containerName="main" containerID="cri-o://49e9d1bcb6e4754f4507c40874d7b5ed7cb692b55abb2cb33543570ea8668250" gracePeriod=45
Sep 15 21:19:40.275700 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:40.275675    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:19:40.405797 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:40.405722    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6d05190-d607-48a5-98f7-dd41758df7dc-dshm\") pod \"a6d05190-d607-48a5-98f7-dd41758df7dc\" (UID: \"a6d05190-d607-48a5-98f7-dd41758df7dc\") "
Sep 15 21:19:40.405797 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:40.405760    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6d05190-d607-48a5-98f7-dd41758df7dc-tmp-dir\") pod \"a6d05190-d607-48a5-98f7-dd41758df7dc\" (UID: \"a6d05190-d607-48a5-98f7-dd41758df7dc\") "
Sep 15 21:19:40.405993 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:40.405808    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znhdb\" (UniqueName: \"kubernetes.io/projected/a6d05190-d607-48a5-98f7-dd41758df7dc-kube-api-access-znhdb\") pod \"a6d05190-d607-48a5-98f7-dd41758df7dc\" (UID: \"a6d05190-d607-48a5-98f7-dd41758df7dc\") "
Sep 15 21:19:40.405993 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:40.405844    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6d05190-d607-48a5-98f7-dd41758df7dc-tls-certs\") pod \"a6d05190-d607-48a5-98f7-dd41758df7dc\" (UID: \"a6d05190-d607-48a5-98f7-dd41758df7dc\") "
Sep 15 21:19:40.405993 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:40.405891    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6d05190-d607-48a5-98f7-dd41758df7dc-home\") pod \"a6d05190-d607-48a5-98f7-dd41758df7dc\" (UID: \"a6d05190-d607-48a5-98f7-dd41758df7dc\") "
Sep 15 21:19:40.405993 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:40.405920    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6d05190-d607-48a5-98f7-dd41758df7dc-model-cache\") pod \"a6d05190-d607-48a5-98f7-dd41758df7dc\" (UID: \"a6d05190-d607-48a5-98f7-dd41758df7dc\") "
Sep 15 21:19:40.406269 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:40.406140    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d05190-d607-48a5-98f7-dd41758df7dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a6d05190-d607-48a5-98f7-dd41758df7dc" (UID: "a6d05190-d607-48a5-98f7-dd41758df7dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:19:40.406269 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:40.406160    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d05190-d607-48a5-98f7-dd41758df7dc-home" (OuterVolumeSpecName: "home") pod "a6d05190-d607-48a5-98f7-dd41758df7dc" (UID: "a6d05190-d607-48a5-98f7-dd41758df7dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:19:40.406397 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:40.406274    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d05190-d607-48a5-98f7-dd41758df7dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "a6d05190-d607-48a5-98f7-dd41758df7dc" (UID: "a6d05190-d607-48a5-98f7-dd41758df7dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:19:40.406455 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:40.406408    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6d05190-d607-48a5-98f7-dd41758df7dc-tmp-dir\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:19:40.406455 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:40.406427    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6d05190-d607-48a5-98f7-dd41758df7dc-home\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:19:40.406455 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:40.406439    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6d05190-d607-48a5-98f7-dd41758df7dc-model-cache\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:19:40.408031 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:40.408007    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d05190-d607-48a5-98f7-dd41758df7dc-dshm" (OuterVolumeSpecName: "dshm") pod "a6d05190-d607-48a5-98f7-dd41758df7dc" (UID: "a6d05190-d607-48a5-98f7-dd41758df7dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:19:40.408031 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:40.408018    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d05190-d607-48a5-98f7-dd41758df7dc-kube-api-access-znhdb" (OuterVolumeSpecName: "kube-api-access-znhdb") pod "a6d05190-d607-48a5-98f7-dd41758df7dc" (UID: "a6d05190-d607-48a5-98f7-dd41758df7dc"). InnerVolumeSpecName "kube-api-access-znhdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:19:40.408197 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:40.408036    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6d05190-d607-48a5-98f7-dd41758df7dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6d05190-d607-48a5-98f7-dd41758df7dc" (UID: "a6d05190-d607-48a5-98f7-dd41758df7dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:19:40.507750 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:40.507722    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6d05190-d607-48a5-98f7-dd41758df7dc-dshm\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:19:40.507750 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:40.507746    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znhdb\" (UniqueName: \"kubernetes.io/projected/a6d05190-d607-48a5-98f7-dd41758df7dc-kube-api-access-znhdb\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:19:40.507900 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:40.507757    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6d05190-d607-48a5-98f7-dd41758df7dc-tls-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:19:41.110426 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:41.110393    2558 generic.go:356] "Generic (PLEG): container finished" podID="a6d05190-d607-48a5-98f7-dd41758df7dc" containerID="49e9d1bcb6e4754f4507c40874d7b5ed7cb692b55abb2cb33543570ea8668250" exitCode=0
Sep 15 21:19:41.110822 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:41.110431    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"
Sep 15 21:19:41.110822 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:41.110458    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq" event={"ID":"a6d05190-d607-48a5-98f7-dd41758df7dc","Type":"ContainerDied","Data":"49e9d1bcb6e4754f4507c40874d7b5ed7cb692b55abb2cb33543570ea8668250"}
Sep 15 21:19:41.110822 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:41.110499    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq" event={"ID":"a6d05190-d607-48a5-98f7-dd41758df7dc","Type":"ContainerDied","Data":"0c6f07be5a23cea8da1c5d37f1e5d7102a878878b94bfe7ef8f4ed28770f76f7"}
Sep 15 21:19:41.110822 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:41.110516    2558 scope.go:117] "RemoveContainer" containerID="49e9d1bcb6e4754f4507c40874d7b5ed7cb692b55abb2cb33543570ea8668250"
Sep 15 21:19:41.120292 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:41.120275    2558 scope.go:117] "RemoveContainer" containerID="49e9d1bcb6e4754f4507c40874d7b5ed7cb692b55abb2cb33543570ea8668250"
Sep 15 21:19:41.120546 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:19:41.120524    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49e9d1bcb6e4754f4507c40874d7b5ed7cb692b55abb2cb33543570ea8668250\": container with ID starting with 49e9d1bcb6e4754f4507c40874d7b5ed7cb692b55abb2cb33543570ea8668250 not found: ID does not exist" containerID="49e9d1bcb6e4754f4507c40874d7b5ed7cb692b55abb2cb33543570ea8668250"
Sep 15 21:19:41.120618 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:41.120552    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49e9d1bcb6e4754f4507c40874d7b5ed7cb692b55abb2cb33543570ea8668250"} err="failed to get container status \"49e9d1bcb6e4754f4507c40874d7b5ed7cb692b55abb2cb33543570ea8668250\": rpc error: code = NotFound desc = could not find container \"49e9d1bcb6e4754f4507c40874d7b5ed7cb692b55abb2cb33543570ea8668250\": container with ID starting with 49e9d1bcb6e4754f4507c40874d7b5ed7cb692b55abb2cb33543570ea8668250 not found: ID does not exist"
Sep 15 21:19:41.133719 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:41.133686    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"]
Sep 15 21:19:41.137039 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:41.137016    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-56c6b012/e2e-lifecycle-model-56c6b012-kserve-dc494df79-xrlnq"]
Sep 15 21:19:41.942911 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:41.942876    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6d05190-d607-48a5-98f7-dd41758df7dc" path="/var/lib/kubelet/pods/a6d05190-d607-48a5-98f7-dd41758df7dc/volumes"
Sep 15 21:19:42.085621 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:19:42.085582    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:20:03.988112 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:03.988071    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:20:14.026811 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:14.026775    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"]
Sep 15 21:20:14.027415 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:14.026985    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c" podUID="9d40317b-6afb-425c-a2d1-0afcbd5d8020" containerName="istio-proxy" containerID="cri-o://ecaf6e00447d74fa16c101479e86459827dcea56dbfab5e501e65159c548208f" gracePeriod=30
Sep 15 21:20:14.847206 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:14.847173    2558 patch_prober.go:28] interesting pod/e2e-mt-56c6b012-openshift-default-798965d66d-m265c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 21:20:14.847379 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:14.847238    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c" podUID="9d40317b-6afb-425c-a2d1-0afcbd5d8020" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 21:20:19.274615 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.274588    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:20:19.330506 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.330427    2558 generic.go:356] "Generic (PLEG): container finished" podID="9d40317b-6afb-425c-a2d1-0afcbd5d8020" containerID="ecaf6e00447d74fa16c101479e86459827dcea56dbfab5e501e65159c548208f" exitCode=0
Sep 15 21:20:19.330506 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.330475    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"
Sep 15 21:20:19.330671 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.330514    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c" event={"ID":"9d40317b-6afb-425c-a2d1-0afcbd5d8020","Type":"ContainerDied","Data":"ecaf6e00447d74fa16c101479e86459827dcea56dbfab5e501e65159c548208f"}
Sep 15 21:20:19.330671 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.330606    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c" event={"ID":"9d40317b-6afb-425c-a2d1-0afcbd5d8020","Type":"ContainerDied","Data":"dbd9d461b30123fa57877ba88ce1a0c5c96d720f1cbb8a32513894cfa07496cc"}
Sep 15 21:20:19.330671 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.330635    2558 scope.go:117] "RemoveContainer" containerID="ecaf6e00447d74fa16c101479e86459827dcea56dbfab5e501e65159c548208f"
Sep 15 21:20:19.339781 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.339765    2558 scope.go:117] "RemoveContainer" containerID="c47bcaea236f1e9465bed3a75f5742baad0e4e9fa54e54f20fb6f849841d164d"
Sep 15 21:20:19.347827 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.347810    2558 scope.go:117] "RemoveContainer" containerID="ecaf6e00447d74fa16c101479e86459827dcea56dbfab5e501e65159c548208f"
Sep 15 21:20:19.348166 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:20:19.348147    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecaf6e00447d74fa16c101479e86459827dcea56dbfab5e501e65159c548208f\": container with ID starting with ecaf6e00447d74fa16c101479e86459827dcea56dbfab5e501e65159c548208f not found: ID does not exist" containerID="ecaf6e00447d74fa16c101479e86459827dcea56dbfab5e501e65159c548208f"
Sep 15 21:20:19.348242 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.348175    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecaf6e00447d74fa16c101479e86459827dcea56dbfab5e501e65159c548208f"} err="failed to get container status \"ecaf6e00447d74fa16c101479e86459827dcea56dbfab5e501e65159c548208f\": rpc error: code = NotFound desc = could not find container \"ecaf6e00447d74fa16c101479e86459827dcea56dbfab5e501e65159c548208f\": container with ID starting with ecaf6e00447d74fa16c101479e86459827dcea56dbfab5e501e65159c548208f not found: ID does not exist"
Sep 15 21:20:19.348242 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.348192    2558 scope.go:117] "RemoveContainer" containerID="c47bcaea236f1e9465bed3a75f5742baad0e4e9fa54e54f20fb6f849841d164d"
Sep 15 21:20:19.348491 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:20:19.348473    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c47bcaea236f1e9465bed3a75f5742baad0e4e9fa54e54f20fb6f849841d164d\": container with ID starting with c47bcaea236f1e9465bed3a75f5742baad0e4e9fa54e54f20fb6f849841d164d not found: ID does not exist" containerID="c47bcaea236f1e9465bed3a75f5742baad0e4e9fa54e54f20fb6f849841d164d"
Sep 15 21:20:19.348541 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.348496    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c47bcaea236f1e9465bed3a75f5742baad0e4e9fa54e54f20fb6f849841d164d"} err="failed to get container status \"c47bcaea236f1e9465bed3a75f5742baad0e4e9fa54e54f20fb6f849841d164d\": rpc error: code = NotFound desc = could not find container \"c47bcaea236f1e9465bed3a75f5742baad0e4e9fa54e54f20fb6f849841d164d\": container with ID starting with c47bcaea236f1e9465bed3a75f5742baad0e4e9fa54e54f20fb6f849841d164d not found: ID does not exist"
Sep 15 21:20:19.371848 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.371827    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-workload-socket\") pod \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") "
Sep 15 21:20:19.371937 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.371859    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istio-podinfo\") pod \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") "
Sep 15 21:20:19.371937 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.371878    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istio-data\") pod \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") "
Sep 15 21:20:19.372082 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.371953    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngldk\" (UniqueName: \"kubernetes.io/projected/9d40317b-6afb-425c-a2d1-0afcbd5d8020-kube-api-access-ngldk\") pod \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") "
Sep 15 21:20:19.372082 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.371973    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-workload-certs\") pod \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") "
Sep 15 21:20:19.372082 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.372015    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istiod-ca-cert\") pod \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") "
Sep 15 21:20:19.372082 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.372042    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-credential-socket\") pod \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") "
Sep 15 21:20:19.372082 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.372044    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9d40317b-6afb-425c-a2d1-0afcbd5d8020" (UID: "9d40317b-6afb-425c-a2d1-0afcbd5d8020"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:20:19.372358 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.372111    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istio-envoy\") pod \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") "
Sep 15 21:20:19.372358 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.372141    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istio-token\") pod \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\" (UID: \"9d40317b-6afb-425c-a2d1-0afcbd5d8020\") "
Sep 15 21:20:19.372358 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.372291    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9d40317b-6afb-425c-a2d1-0afcbd5d8020" (UID: "9d40317b-6afb-425c-a2d1-0afcbd5d8020"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:20:19.372517 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.372299    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istio-data" (OuterVolumeSpecName: "istio-data") pod "9d40317b-6afb-425c-a2d1-0afcbd5d8020" (UID: "9d40317b-6afb-425c-a2d1-0afcbd5d8020"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:20:19.372517 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.372384    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9d40317b-6afb-425c-a2d1-0afcbd5d8020" (UID: "9d40317b-6afb-425c-a2d1-0afcbd5d8020"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:20:19.372635 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.372521    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9d40317b-6afb-425c-a2d1-0afcbd5d8020" (UID: "9d40317b-6afb-425c-a2d1-0afcbd5d8020"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:20:19.372635 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.372549    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-workload-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:20:19.372635 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.372566    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istiod-ca-cert\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:20:19.372635 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.372577    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-workload-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:20:19.372635 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.372586    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istio-data\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:20:19.374099 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.374075    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9d40317b-6afb-425c-a2d1-0afcbd5d8020" (UID: "9d40317b-6afb-425c-a2d1-0afcbd5d8020"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:20:19.374099 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.374081    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d40317b-6afb-425c-a2d1-0afcbd5d8020-kube-api-access-ngldk" (OuterVolumeSpecName: "kube-api-access-ngldk") pod "9d40317b-6afb-425c-a2d1-0afcbd5d8020" (UID: "9d40317b-6afb-425c-a2d1-0afcbd5d8020"). InnerVolumeSpecName "kube-api-access-ngldk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:20:19.374517 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.374495    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istio-token" (OuterVolumeSpecName: "istio-token") pod "9d40317b-6afb-425c-a2d1-0afcbd5d8020" (UID: "9d40317b-6afb-425c-a2d1-0afcbd5d8020"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:20:19.374570 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.374496    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9d40317b-6afb-425c-a2d1-0afcbd5d8020" (UID: "9d40317b-6afb-425c-a2d1-0afcbd5d8020"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:20:19.473027 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.473001    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istio-podinfo\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:20:19.473027 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.473025    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngldk\" (UniqueName: \"kubernetes.io/projected/9d40317b-6afb-425c-a2d1-0afcbd5d8020-kube-api-access-ngldk\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:20:19.473199 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.473036    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-credential-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:20:19.473199 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.473045    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istio-envoy\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:20:19.473199 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.473075    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d40317b-6afb-425c-a2d1-0afcbd5d8020-istio-token\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:20:19.654387 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.654359    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"]
Sep 15 21:20:19.660535 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.660511    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-56c6b012-openshift-default-798965d66d-m265c"]
Sep 15 21:20:19.789454 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.789423    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"]
Sep 15 21:20:19.789796 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.789784    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d40317b-6afb-425c-a2d1-0afcbd5d8020" containerName="istio-proxy"
Sep 15 21:20:19.789852 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.789797    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d40317b-6afb-425c-a2d1-0afcbd5d8020" containerName="istio-proxy"
Sep 15 21:20:19.789852 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.789820    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6d05190-d607-48a5-98f7-dd41758df7dc" containerName="main"
Sep 15 21:20:19.789852 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.789826    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6d05190-d607-48a5-98f7-dd41758df7dc" containerName="main"
Sep 15 21:20:19.789956 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.789886    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d40317b-6afb-425c-a2d1-0afcbd5d8020" containerName="istio-proxy"
Sep 15 21:20:19.789956 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.789896    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6d05190-d607-48a5-98f7-dd41758df7dc" containerName="main"
Sep 15 21:20:19.794348 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.794329    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:20:19.797676 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.797650    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-41ffdf\"/\"openshift-service-ca.crt\""
Sep 15 21:20:19.797676 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.797663    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-41ffdf\"/\"auth-test-model-41ffdf-kserve-self-signed-certs\""
Sep 15 21:20:19.797839 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.797823    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-41ffdf\"/\"kube-root-ca.crt\""
Sep 15 21:20:19.797878 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.797825    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-41ffdf\"/\"default-dockercfg-2p5wv\""
Sep 15 21:20:19.803584 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.803561    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"]
Sep 15 21:20:19.943068 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.943024    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d40317b-6afb-425c-a2d1-0afcbd5d8020" path="/var/lib/kubelet/pods/9d40317b-6afb-425c-a2d1-0afcbd5d8020/volumes"
Sep 15 21:20:19.977111 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.977086    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cngx\" (UniqueName: \"kubernetes.io/projected/33bfb8a8-523a-44a7-bcd3-093863d0abbb-kube-api-access-8cngx\") pod \"auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq\" (UID: \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\") " pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:20:19.977254 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.977135    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33bfb8a8-523a-44a7-bcd3-093863d0abbb-model-cache\") pod \"auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq\" (UID: \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\") " pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:20:19.977254 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.977232    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33bfb8a8-523a-44a7-bcd3-093863d0abbb-home\") pod \"auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq\" (UID: \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\") " pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:20:19.977335 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.977262    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33bfb8a8-523a-44a7-bcd3-093863d0abbb-tmp-dir\") pod \"auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq\" (UID: \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\") " pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:20:19.977335 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.977306    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33bfb8a8-523a-44a7-bcd3-093863d0abbb-tls-certs\") pod \"auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq\" (UID: \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\") " pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:20:19.977335 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:19.977322    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33bfb8a8-523a-44a7-bcd3-093863d0abbb-dshm\") pod \"auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq\" (UID: \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\") " pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:20:20.078588 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:20.078554    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33bfb8a8-523a-44a7-bcd3-093863d0abbb-tls-certs\") pod \"auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq\" (UID: \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\") " pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:20:20.078588 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:20.078590    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33bfb8a8-523a-44a7-bcd3-093863d0abbb-dshm\") pod \"auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq\" (UID: \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\") " pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:20:20.078866 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:20.078850    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cngx\" (UniqueName: \"kubernetes.io/projected/33bfb8a8-523a-44a7-bcd3-093863d0abbb-kube-api-access-8cngx\") pod \"auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq\" (UID: \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\") " pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:20:20.078953 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:20.078930    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33bfb8a8-523a-44a7-bcd3-093863d0abbb-model-cache\") pod \"auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq\" (UID: \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\") " pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:20:20.079105 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:20.079079    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33bfb8a8-523a-44a7-bcd3-093863d0abbb-home\") pod \"auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq\" (UID: \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\") " pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:20:20.079266 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:20.079117    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33bfb8a8-523a-44a7-bcd3-093863d0abbb-tmp-dir\") pod \"auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq\" (UID: \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\") " pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:20:20.079482 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:20.079449    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33bfb8a8-523a-44a7-bcd3-093863d0abbb-home\") pod \"auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq\" (UID: \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\") " pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:20:20.079608 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:20.079459    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33bfb8a8-523a-44a7-bcd3-093863d0abbb-tmp-dir\") pod \"auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq\" (UID: \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\") " pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:20:20.079608 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:20.079493    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33bfb8a8-523a-44a7-bcd3-093863d0abbb-model-cache\") pod \"auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq\" (UID: \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\") " pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:20:20.081105 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:20.081081    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33bfb8a8-523a-44a7-bcd3-093863d0abbb-dshm\") pod \"auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq\" (UID: \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\") " pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:20:20.081219 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:20.081203    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33bfb8a8-523a-44a7-bcd3-093863d0abbb-tls-certs\") pod \"auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq\" (UID: \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\") " pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:20:20.087061 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:20.087028    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cngx\" (UniqueName: \"kubernetes.io/projected/33bfb8a8-523a-44a7-bcd3-093863d0abbb-kube-api-access-8cngx\") pod \"auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq\" (UID: \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\") " pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:20:20.104637 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:20.104612    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:20:20.376880 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:20.376852    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"]
Sep 15 21:20:20.377780 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:20:20.377756    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33bfb8a8_523a_44a7_bcd3_093863d0abbb.slice/crio-d29e74d1d751a5fa8a6a5888b0f7669182485523d0576fa4cae05a6fcb1d779f WatchSource:0}: Error finding container d29e74d1d751a5fa8a6a5888b0f7669182485523d0576fa4cae05a6fcb1d779f: Status 404 returned error can't find the container with id d29e74d1d751a5fa8a6a5888b0f7669182485523d0576fa4cae05a6fcb1d779f
Sep 15 21:20:21.343592 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:21.343556    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq" event={"ID":"33bfb8a8-523a-44a7-bcd3-093863d0abbb","Type":"ContainerStarted","Data":"ac598b1455eaa8caefb5113252cb92ed61b8e591a319bebca7b0ca29da9a2fcc"}
Sep 15 21:20:21.343592 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:21.343597    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq" event={"ID":"33bfb8a8-523a-44a7-bcd3-093863d0abbb","Type":"ContainerStarted","Data":"d29e74d1d751a5fa8a6a5888b0f7669182485523d0576fa4cae05a6fcb1d779f"}
Sep 15 21:20:21.366967 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:21.366922    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq" podStartSLOduration=2.366906009 podStartE2EDuration="2.366906009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:20:21.366420054 +0000 UTC m=+1339.857065786" watchObservedRunningTime="2026-09-15 21:20:21.366906009 +0000 UTC m=+1339.857551756"
Sep 15 21:20:27.588262 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:27.588224    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:20:30.105212 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:30.105173    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:20:30.105690 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:30.105223    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:20:30.117925 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:30.117896    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:20:30.405345 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:30.405263    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:20:59.586967 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:20:59.586932    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:21:15.482418 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:15.482384    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"]
Sep 15 21:21:15.482969 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:15.482950    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d40317b-6afb-425c-a2d1-0afcbd5d8020" containerName="istio-proxy"
Sep 15 21:21:15.483070 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:15.482973    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d40317b-6afb-425c-a2d1-0afcbd5d8020" containerName="istio-proxy"
Sep 15 21:21:15.483141 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:15.483089    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d40317b-6afb-425c-a2d1-0afcbd5d8020" containerName="istio-proxy"
Sep 15 21:21:15.486807 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:15.486787    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"
Sep 15 21:21:15.489407 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:15.489386    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8f89be-serving-cert\""
Sep 15 21:21:15.499695 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:15.499674    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"]
Sep 15 21:21:15.595840 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:15.595808    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-openshift-service-ca\") pod \"maas-api-e2e-api-a-8f89be-6d44988647-gh759\" (UID: \"ebe933e1-0a0f-4e54-baff-2599a8f4df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"
Sep 15 21:21:15.595991 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:15.595863    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-metrics-certs\") pod \"maas-api-e2e-api-a-8f89be-6d44988647-gh759\" (UID: \"ebe933e1-0a0f-4e54-baff-2599a8f4df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"
Sep 15 21:21:15.595991 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:15.595940    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-maas-api-tls\") pod \"maas-api-e2e-api-a-8f89be-6d44988647-gh759\" (UID: \"ebe933e1-0a0f-4e54-baff-2599a8f4df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"
Sep 15 21:21:15.596093 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:15.595995    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5h92\" (UniqueName: \"kubernetes.io/projected/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-kube-api-access-n5h92\") pod \"maas-api-e2e-api-a-8f89be-6d44988647-gh759\" (UID: \"ebe933e1-0a0f-4e54-baff-2599a8f4df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"
Sep 15 21:21:15.696840 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:15.696804    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-openshift-service-ca\") pod \"maas-api-e2e-api-a-8f89be-6d44988647-gh759\" (UID: \"ebe933e1-0a0f-4e54-baff-2599a8f4df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"
Sep 15 21:21:15.697120 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:15.696867    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-metrics-certs\") pod \"maas-api-e2e-api-a-8f89be-6d44988647-gh759\" (UID: \"ebe933e1-0a0f-4e54-baff-2599a8f4df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"
Sep 15 21:21:15.697120 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:15.696900    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-maas-api-tls\") pod \"maas-api-e2e-api-a-8f89be-6d44988647-gh759\" (UID: \"ebe933e1-0a0f-4e54-baff-2599a8f4df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"
Sep 15 21:21:15.697120 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:15.696936    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5h92\" (UniqueName: \"kubernetes.io/projected/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-kube-api-access-n5h92\") pod \"maas-api-e2e-api-a-8f89be-6d44988647-gh759\" (UID: \"ebe933e1-0a0f-4e54-baff-2599a8f4df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"
Sep 15 21:21:15.697120 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:21:15.697033    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-serving-cert: secret "maas-api-e2e-api-a-8f89be-serving-cert" not found
Sep 15 21:21:15.697387 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:21:15.697152    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-maas-api-tls podName:ebe933e1-0a0f-4e54-baff-2599a8f4df5a nodeName:}" failed. No retries permitted until 2026-09-15 21:21:16.197129202 +0000 UTC m=+1394.687774930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-maas-api-tls") pod "maas-api-e2e-api-a-8f89be-6d44988647-gh759" (UID: "ebe933e1-0a0f-4e54-baff-2599a8f4df5a") : secret "maas-api-e2e-api-a-8f89be-serving-cert" not found
Sep 15 21:21:15.697605 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:15.697582    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-openshift-service-ca\") pod \"maas-api-e2e-api-a-8f89be-6d44988647-gh759\" (UID: \"ebe933e1-0a0f-4e54-baff-2599a8f4df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"
Sep 15 21:21:15.699341 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:15.699323    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-metrics-certs\") pod \"maas-api-e2e-api-a-8f89be-6d44988647-gh759\" (UID: \"ebe933e1-0a0f-4e54-baff-2599a8f4df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"
Sep 15 21:21:15.705407 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:15.705386    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5h92\" (UniqueName: \"kubernetes.io/projected/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-kube-api-access-n5h92\") pod \"maas-api-e2e-api-a-8f89be-6d44988647-gh759\" (UID: \"ebe933e1-0a0f-4e54-baff-2599a8f4df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"
Sep 15 21:21:16.201440 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:16.201408    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-maas-api-tls\") pod \"maas-api-e2e-api-a-8f89be-6d44988647-gh759\" (UID: \"ebe933e1-0a0f-4e54-baff-2599a8f4df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"
Sep 15 21:21:16.203851 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:16.203818    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-maas-api-tls\") pod \"maas-api-e2e-api-a-8f89be-6d44988647-gh759\" (UID: \"ebe933e1-0a0f-4e54-baff-2599a8f4df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"
Sep 15 21:21:16.398011 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:16.397973    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"
Sep 15 21:21:16.678133 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:16.678109    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"]
Sep 15 21:21:16.679731 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:21:16.679704    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe933e1_0a0f_4e54_baff_2599a8f4df5a.slice/crio-7b0a783948868051497f751eb0caebab967f07dd27e4842ee1852d2f94d945c4 WatchSource:0}: Error finding container 7b0a783948868051497f751eb0caebab967f07dd27e4842ee1852d2f94d945c4: Status 404 returned error can't find the container with id 7b0a783948868051497f751eb0caebab967f07dd27e4842ee1852d2f94d945c4
Sep 15 21:21:16.681042 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:16.681024    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:21:17.675979 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:17.675948    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759" event={"ID":"ebe933e1-0a0f-4e54-baff-2599a8f4df5a","Type":"ContainerStarted","Data":"b688286fdb947a44a2ffd48d9ec184631ff349ed91b22a0dedc62b85ed2b6c80"}
Sep 15 21:21:17.676190 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:17.676024    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759" event={"ID":"ebe933e1-0a0f-4e54-baff-2599a8f4df5a","Type":"ContainerStarted","Data":"7b0a783948868051497f751eb0caebab967f07dd27e4842ee1852d2f94d945c4"}
Sep 15 21:21:17.676190 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:17.676092    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"
Sep 15 21:21:17.694819 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:17.694771    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759" podStartSLOduration=2.194680924 podStartE2EDuration="2.694757172s" totalImagesPullingTime="500.076248ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:21:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:21:17.693101633 +0000 UTC m=+1396.183747364" watchObservedRunningTime="2026-09-15 21:21:17.694757172 +0000 UTC m=+1396.185402905"
Sep 15 21:21:23.687132 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:23.687104    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"
Sep 15 21:21:27.893861 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:27.893830    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:21:34.971498 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:34.971463    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-544890-29825115-bzz5t"]
Sep 15 21:21:34.976562 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:34.976535    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-544890-29825115-bzz5t"]
Sep 15 21:21:35.942878 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:35.942841    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cd559e9-c604-4256-97e1-805f47eac94f" path="/var/lib/kubelet/pods/1cd559e9-c604-4256-97e1-805f47eac94f/volumes"
Sep 15 21:21:41.911663 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:41.911630    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"]
Sep 15 21:21:41.912082 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:41.911839    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw" podUID="2ed5bb3c-b04d-4d34-a04f-f02c037fe755" containerName="istio-proxy" containerID="cri-o://096fe319cde25ac53ab6203b827fb00f01127a47eec857288d3d9ffab9d637c0" gracePeriod=30
Sep 15 21:21:43.778188 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:43.778036    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 21:21:43.778605 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:43.778218    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw" podUID="2ed5bb3c-b04d-4d34-a04f-f02c037fe755" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 21:21:47.164921 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.164895    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:21:47.191809 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.191756    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dcqk\" (UniqueName: \"kubernetes.io/projected/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-kube-api-access-8dcqk\") pod \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") "
Sep 15 21:21:47.191809 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.191800    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-credential-socket\") pod \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") "
Sep 15 21:21:47.191974 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.191852    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istio-token\") pod \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") "
Sep 15 21:21:47.191974 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.191879    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-workload-socket\") pod \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") "
Sep 15 21:21:47.191974 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.191918    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istio-envoy\") pod \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") "
Sep 15 21:21:47.192166 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.191981    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istiod-ca-cert\") pod \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") "
Sep 15 21:21:47.192166 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.192016    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-workload-certs\") pod \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") "
Sep 15 21:21:47.192166 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.192102    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istio-podinfo\") pod \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") "
Sep 15 21:21:47.192166 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.192130    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2ed5bb3c-b04d-4d34-a04f-f02c037fe755" (UID: "2ed5bb3c-b04d-4d34-a04f-f02c037fe755"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:21:47.192365 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.192199    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istio-data\") pod \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\" (UID: \"2ed5bb3c-b04d-4d34-a04f-f02c037fe755\") "
Sep 15 21:21:47.192365 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.192181    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2ed5bb3c-b04d-4d34-a04f-f02c037fe755" (UID: "2ed5bb3c-b04d-4d34-a04f-f02c037fe755"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:21:47.192365 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.192323    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2ed5bb3c-b04d-4d34-a04f-f02c037fe755" (UID: "2ed5bb3c-b04d-4d34-a04f-f02c037fe755"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:21:47.192525 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.192380    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2ed5bb3c-b04d-4d34-a04f-f02c037fe755" (UID: "2ed5bb3c-b04d-4d34-a04f-f02c037fe755"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:21:47.193140 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.192652    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istio-data" (OuterVolumeSpecName: "istio-data") pod "2ed5bb3c-b04d-4d34-a04f-f02c037fe755" (UID: "2ed5bb3c-b04d-4d34-a04f-f02c037fe755"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:21:47.193140 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.193004    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istio-data\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:21:47.193140 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.193030    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-credential-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:21:47.193140 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.193046    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-workload-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:21:47.193459 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.193185    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istiod-ca-cert\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:21:47.193459 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.193202    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-workload-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:21:47.194544 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.194517    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istio-token" (OuterVolumeSpecName: "istio-token") pod "2ed5bb3c-b04d-4d34-a04f-f02c037fe755" (UID: "2ed5bb3c-b04d-4d34-a04f-f02c037fe755"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:21:47.194786 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.194760    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-kube-api-access-8dcqk" (OuterVolumeSpecName: "kube-api-access-8dcqk") pod "2ed5bb3c-b04d-4d34-a04f-f02c037fe755" (UID: "2ed5bb3c-b04d-4d34-a04f-f02c037fe755"). InnerVolumeSpecName "kube-api-access-8dcqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:21:47.195253 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.195228    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2ed5bb3c-b04d-4d34-a04f-f02c037fe755" (UID: "2ed5bb3c-b04d-4d34-a04f-f02c037fe755"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:21:47.195253 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.195243    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2ed5bb3c-b04d-4d34-a04f-f02c037fe755" (UID: "2ed5bb3c-b04d-4d34-a04f-f02c037fe755"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:21:47.294470 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.294440    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istio-podinfo\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:21:47.294470 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.294468    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dcqk\" (UniqueName: \"kubernetes.io/projected/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-kube-api-access-8dcqk\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:21:47.294662 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.294479    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istio-token\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:21:47.294662 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.294487    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed5bb3c-b04d-4d34-a04f-f02c037fe755-istio-envoy\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:21:47.848604 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.848573    2558 generic.go:356] "Generic (PLEG): container finished" podID="2ed5bb3c-b04d-4d34-a04f-f02c037fe755" containerID="096fe319cde25ac53ab6203b827fb00f01127a47eec857288d3d9ffab9d637c0" exitCode=0
Sep 15 21:21:47.848769 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.848621    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"
Sep 15 21:21:47.848769 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.848653    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw" event={"ID":"2ed5bb3c-b04d-4d34-a04f-f02c037fe755","Type":"ContainerDied","Data":"096fe319cde25ac53ab6203b827fb00f01127a47eec857288d3d9ffab9d637c0"}
Sep 15 21:21:47.848769 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.848703    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw" event={"ID":"2ed5bb3c-b04d-4d34-a04f-f02c037fe755","Type":"ContainerDied","Data":"c8c6dd8b56fbf3bcb0bf1b547f03d131eca5416a423395d8cfc511915703f2c1"}
Sep 15 21:21:47.848769 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.848727    2558 scope.go:117] "RemoveContainer" containerID="096fe319cde25ac53ab6203b827fb00f01127a47eec857288d3d9ffab9d637c0"
Sep 15 21:21:47.858520 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.858501    2558 scope.go:117] "RemoveContainer" containerID="92a2c5c3762fad6e01220efe922a07deb0a29179c0c209be51be669c63c16ec5"
Sep 15 21:21:47.866331 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.866315    2558 scope.go:117] "RemoveContainer" containerID="096fe319cde25ac53ab6203b827fb00f01127a47eec857288d3d9ffab9d637c0"
Sep 15 21:21:47.866579 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:21:47.866561    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"096fe319cde25ac53ab6203b827fb00f01127a47eec857288d3d9ffab9d637c0\": container with ID starting with 096fe319cde25ac53ab6203b827fb00f01127a47eec857288d3d9ffab9d637c0 not found: ID does not exist" containerID="096fe319cde25ac53ab6203b827fb00f01127a47eec857288d3d9ffab9d637c0"
Sep 15 21:21:47.866621 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.866587    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"096fe319cde25ac53ab6203b827fb00f01127a47eec857288d3d9ffab9d637c0"} err="failed to get container status \"096fe319cde25ac53ab6203b827fb00f01127a47eec857288d3d9ffab9d637c0\": rpc error: code = NotFound desc = could not find container \"096fe319cde25ac53ab6203b827fb00f01127a47eec857288d3d9ffab9d637c0\": container with ID starting with 096fe319cde25ac53ab6203b827fb00f01127a47eec857288d3d9ffab9d637c0 not found: ID does not exist"
Sep 15 21:21:47.866621 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.866603    2558 scope.go:117] "RemoveContainer" containerID="92a2c5c3762fad6e01220efe922a07deb0a29179c0c209be51be669c63c16ec5"
Sep 15 21:21:47.866853 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:21:47.866836    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92a2c5c3762fad6e01220efe922a07deb0a29179c0c209be51be669c63c16ec5\": container with ID starting with 92a2c5c3762fad6e01220efe922a07deb0a29179c0c209be51be669c63c16ec5 not found: ID does not exist" containerID="92a2c5c3762fad6e01220efe922a07deb0a29179c0c209be51be669c63c16ec5"
Sep 15 21:21:47.866894 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.866860    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a2c5c3762fad6e01220efe922a07deb0a29179c0c209be51be669c63c16ec5"} err="failed to get container status \"92a2c5c3762fad6e01220efe922a07deb0a29179c0c209be51be669c63c16ec5\": rpc error: code = NotFound desc = could not find container \"92a2c5c3762fad6e01220efe922a07deb0a29179c0c209be51be669c63c16ec5\": container with ID starting with 92a2c5c3762fad6e01220efe922a07deb0a29179c0c209be51be669c63c16ec5 not found: ID does not exist"
Sep 15 21:21:47.877811 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.877793    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"]
Sep 15 21:21:47.881596 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.881579    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-544890-openshift-default-6c79fd49d4-k6qxw"]
Sep 15 21:21:47.942642 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:47.942614    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ed5bb3c-b04d-4d34-a04f-f02c037fe755" path="/var/lib/kubelet/pods/2ed5bb3c-b04d-4d34-a04f-f02c037fe755/volumes"
Sep 15 21:21:59.587880 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:21:59.587801    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:22:02.199303 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:02.199276    2558 scope.go:117] "RemoveContainer" containerID="2dd6743e99786b4b83cb4c01d23e7e113f3f73e4809cdc20d717c71cfed9731d"
Sep 15 21:22:02.207743 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:02.207721    2558 scope.go:117] "RemoveContainer" containerID="6816a9e0df09a1bcf4f9d768548715777370132f4c347f51ce462484db005b27"
Sep 15 21:22:02.215259 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:02.215235    2558 scope.go:117] "RemoveContainer" containerID="fd4b2b87034e27e877478315a4ae9e541b205ec4c79eb52b98cdca61facf1da5"
Sep 15 21:22:17.286434 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:17.286393    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:22:34.535400 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:34.535356    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"]
Sep 15 21:22:34.535965 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:34.535662    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759" podUID="ebe933e1-0a0f-4e54-baff-2599a8f4df5a" containerName="maas-api" containerID="cri-o://b688286fdb947a44a2ffd48d9ec184631ff349ed91b22a0dedc62b85ed2b6c80" gracePeriod=30
Sep 15 21:22:34.700234 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:34.700199    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:22:34.790574 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:34.790505    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"
Sep 15 21:22:34.845285 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:34.845255    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-openshift-service-ca\") pod \"ebe933e1-0a0f-4e54-baff-2599a8f4df5a\" (UID: \"ebe933e1-0a0f-4e54-baff-2599a8f4df5a\") "
Sep 15 21:22:34.845478 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:34.845314    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5h92\" (UniqueName: \"kubernetes.io/projected/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-kube-api-access-n5h92\") pod \"ebe933e1-0a0f-4e54-baff-2599a8f4df5a\" (UID: \"ebe933e1-0a0f-4e54-baff-2599a8f4df5a\") "
Sep 15 21:22:34.845478 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:34.845336    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-maas-api-tls\") pod \"ebe933e1-0a0f-4e54-baff-2599a8f4df5a\" (UID: \"ebe933e1-0a0f-4e54-baff-2599a8f4df5a\") "
Sep 15 21:22:34.845478 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:34.845367    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-metrics-certs\") pod \"ebe933e1-0a0f-4e54-baff-2599a8f4df5a\" (UID: \"ebe933e1-0a0f-4e54-baff-2599a8f4df5a\") "
Sep 15 21:22:34.845674 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:34.845614    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ebe933e1-0a0f-4e54-baff-2599a8f4df5a" (UID: "ebe933e1-0a0f-4e54-baff-2599a8f4df5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:22:34.847489 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:34.847458    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ebe933e1-0a0f-4e54-baff-2599a8f4df5a" (UID: "ebe933e1-0a0f-4e54-baff-2599a8f4df5a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:22:34.847606 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:34.847504    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ebe933e1-0a0f-4e54-baff-2599a8f4df5a" (UID: "ebe933e1-0a0f-4e54-baff-2599a8f4df5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:22:34.847845 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:34.847826    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-kube-api-access-n5h92" (OuterVolumeSpecName: "kube-api-access-n5h92") pod "ebe933e1-0a0f-4e54-baff-2599a8f4df5a" (UID: "ebe933e1-0a0f-4e54-baff-2599a8f4df5a"). InnerVolumeSpecName "kube-api-access-n5h92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:22:34.946535 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:34.946496    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5h92\" (UniqueName: \"kubernetes.io/projected/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-kube-api-access-n5h92\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:22:34.946535 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:34.946527    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-maas-api-tls\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:22:34.946535 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:34.946536    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-metrics-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:22:34.946535 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:34.946545    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe933e1-0a0f-4e54-baff-2599a8f4df5a-openshift-service-ca\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:22:35.114674 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:35.114593    2558 generic.go:356] "Generic (PLEG): container finished" podID="ebe933e1-0a0f-4e54-baff-2599a8f4df5a" containerID="b688286fdb947a44a2ffd48d9ec184631ff349ed91b22a0dedc62b85ed2b6c80" exitCode=0
Sep 15 21:22:35.114674 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:35.114639    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"
Sep 15 21:22:35.114870 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:35.114669    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759" event={"ID":"ebe933e1-0a0f-4e54-baff-2599a8f4df5a","Type":"ContainerDied","Data":"b688286fdb947a44a2ffd48d9ec184631ff349ed91b22a0dedc62b85ed2b6c80"}
Sep 15 21:22:35.114870 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:35.114717    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759" event={"ID":"ebe933e1-0a0f-4e54-baff-2599a8f4df5a","Type":"ContainerDied","Data":"7b0a783948868051497f751eb0caebab967f07dd27e4842ee1852d2f94d945c4"}
Sep 15 21:22:35.114870 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:35.114739    2558 scope.go:117] "RemoveContainer" containerID="b688286fdb947a44a2ffd48d9ec184631ff349ed91b22a0dedc62b85ed2b6c80"
Sep 15 21:22:35.124278 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:35.124260    2558 scope.go:117] "RemoveContainer" containerID="b688286fdb947a44a2ffd48d9ec184631ff349ed91b22a0dedc62b85ed2b6c80"
Sep 15 21:22:35.124539 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:22:35.124519    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b688286fdb947a44a2ffd48d9ec184631ff349ed91b22a0dedc62b85ed2b6c80\": container with ID starting with b688286fdb947a44a2ffd48d9ec184631ff349ed91b22a0dedc62b85ed2b6c80 not found: ID does not exist" containerID="b688286fdb947a44a2ffd48d9ec184631ff349ed91b22a0dedc62b85ed2b6c80"
Sep 15 21:22:35.124596 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:35.124552    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b688286fdb947a44a2ffd48d9ec184631ff349ed91b22a0dedc62b85ed2b6c80"} err="failed to get container status \"b688286fdb947a44a2ffd48d9ec184631ff349ed91b22a0dedc62b85ed2b6c80\": rpc error: code = NotFound desc = could not find container \"b688286fdb947a44a2ffd48d9ec184631ff349ed91b22a0dedc62b85ed2b6c80\": container with ID starting with b688286fdb947a44a2ffd48d9ec184631ff349ed91b22a0dedc62b85ed2b6c80 not found: ID does not exist"
Sep 15 21:22:35.144979 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:35.144955    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"]
Sep 15 21:22:35.148754 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:35.148731    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f89be-6d44988647-gh759"]
Sep 15 21:22:35.942333 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:35.942301    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebe933e1-0a0f-4e54-baff-2599a8f4df5a" path="/var/lib/kubelet/pods/ebe933e1-0a0f-4e54-baff-2599a8f4df5a/volumes"
Sep 15 21:22:52.985775 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:22:52.985738    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:23:07.599495 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:07.599452    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:23:09.557474 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:09.557435    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7928-456501"]
Sep 15 21:23:09.558002 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:09.557983    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ed5bb3c-b04d-4d34-a04f-f02c037fe755" containerName="istio-proxy"
Sep 15 21:23:09.558077 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:09.558007    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ed5bb3c-b04d-4d34-a04f-f02c037fe755" containerName="istio-proxy"
Sep 15 21:23:09.558077 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:09.558065    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ed5bb3c-b04d-4d34-a04f-f02c037fe755" containerName="istio-proxy"
Sep 15 21:23:09.558077 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:09.558077    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ed5bb3c-b04d-4d34-a04f-f02c037fe755" containerName="istio-proxy"
Sep 15 21:23:09.558424 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:09.558096    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebe933e1-0a0f-4e54-baff-2599a8f4df5a" containerName="maas-api"
Sep 15 21:23:09.558424 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:09.558105    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebe933e1-0a0f-4e54-baff-2599a8f4df5a" containerName="maas-api"
Sep 15 21:23:09.558424 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:09.558201    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebe933e1-0a0f-4e54-baff-2599a8f4df5a" containerName="maas-api"
Sep 15 21:23:09.558424 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:09.558212    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ed5bb3c-b04d-4d34-a04f-f02c037fe755" containerName="istio-proxy"
Sep 15 21:23:09.558424 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:09.558224    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ed5bb3c-b04d-4d34-a04f-f02c037fe755" containerName="istio-proxy"
Sep 15 21:23:09.562616 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:09.562597    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7928-456501"
Sep 15 21:23:09.577082 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:09.577032    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7928-456501"]
Sep 15 21:23:09.577553 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:09.577534    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-r6zqd\""
Sep 15 21:23:09.671609 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:09.671583    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqjjh\" (UniqueName: \"kubernetes.io/projected/4d274417-b615-40e1-9702-2d16bd00b33c-kube-api-access-dqjjh\") pod \"test-curl-7928-456501\" (UID: \"4d274417-b615-40e1-9702-2d16bd00b33c\") " pod="opendatahub/test-curl-7928-456501"
Sep 15 21:23:09.773006 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:09.772973    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqjjh\" (UniqueName: \"kubernetes.io/projected/4d274417-b615-40e1-9702-2d16bd00b33c-kube-api-access-dqjjh\") pod \"test-curl-7928-456501\" (UID: \"4d274417-b615-40e1-9702-2d16bd00b33c\") " pod="opendatahub/test-curl-7928-456501"
Sep 15 21:23:09.781755 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:09.781717    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqjjh\" (UniqueName: \"kubernetes.io/projected/4d274417-b615-40e1-9702-2d16bd00b33c-kube-api-access-dqjjh\") pod \"test-curl-7928-456501\" (UID: \"4d274417-b615-40e1-9702-2d16bd00b33c\") " pod="opendatahub/test-curl-7928-456501"
Sep 15 21:23:09.871952 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:09.871883    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7928-456501"
Sep 15 21:23:10.143007 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:10.142974    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7928-456501"]
Sep 15 21:23:10.145004 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:23:10.144981    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d274417_b615_40e1_9702_2d16bd00b33c.slice/crio-aa3cbcb33e54342566edc03067a8a016109a4e60278e6913dd9a2681b5990148 WatchSource:0}: Error finding container aa3cbcb33e54342566edc03067a8a016109a4e60278e6913dd9a2681b5990148: Status 404 returned error can't find the container with id aa3cbcb33e54342566edc03067a8a016109a4e60278e6913dd9a2681b5990148
Sep 15 21:23:10.309726 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:10.309694    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7928-456501" event={"ID":"4d274417-b615-40e1-9702-2d16bd00b33c","Type":"ContainerStarted","Data":"62e84de4365eb4d8e28bac37ac0d987a64f83bbac128e47b8d5500570e0f76a1"}
Sep 15 21:23:10.309896 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:10.309736    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7928-456501" event={"ID":"4d274417-b615-40e1-9702-2d16bd00b33c","Type":"ContainerStarted","Data":"aa3cbcb33e54342566edc03067a8a016109a4e60278e6913dd9a2681b5990148"}
Sep 15 21:23:10.328609 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:10.328564    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7928-456501" podStartSLOduration=1.328549287 podStartE2EDuration="1.328549287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:23:10.326661545 +0000 UTC m=+1508.817307278" watchObservedRunningTime="2026-09-15 21:23:10.328549287 +0000 UTC m=+1508.819195080"
Sep 15 21:23:10.913315 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:10.913227    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7928-456501"]
Sep 15 21:23:10.917090 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:10.917065    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7928-456501"]
Sep 15 21:23:11.315379 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:23:11.315347    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7928-456501" container="test-curl-7928-456501"
Sep 15 21:23:12.267285 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:12.267253    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"]
Sep 15 21:23:12.320731 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:12.320689    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7928-456501" podUID="4d274417-b615-40e1-9702-2d16bd00b33c" containerName="test-curl-7928-456501" containerID="cri-o://62e84de4365eb4d8e28bac37ac0d987a64f83bbac128e47b8d5500570e0f76a1" gracePeriod=2
Sep 15 21:23:13.377903 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:13.377870    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"]
Sep 15 21:23:14.506783 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:14.506755    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7928-456501"
Sep 15 21:23:14.617135 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:14.617027    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqjjh\" (UniqueName: \"kubernetes.io/projected/4d274417-b615-40e1-9702-2d16bd00b33c-kube-api-access-dqjjh\") pod \"4d274417-b615-40e1-9702-2d16bd00b33c\" (UID: \"4d274417-b615-40e1-9702-2d16bd00b33c\") "
Sep 15 21:23:14.619144 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:14.619122    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d274417-b615-40e1-9702-2d16bd00b33c-kube-api-access-dqjjh" (OuterVolumeSpecName: "kube-api-access-dqjjh") pod "4d274417-b615-40e1-9702-2d16bd00b33c" (UID: "4d274417-b615-40e1-9702-2d16bd00b33c"). InnerVolumeSpecName "kube-api-access-dqjjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:23:14.718401 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:14.718366    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqjjh\" (UniqueName: \"kubernetes.io/projected/4d274417-b615-40e1-9702-2d16bd00b33c-kube-api-access-dqjjh\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:15.341931 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:15.341892    2558 generic.go:356] "Generic (PLEG): container finished" podID="4d274417-b615-40e1-9702-2d16bd00b33c" containerID="62e84de4365eb4d8e28bac37ac0d987a64f83bbac128e47b8d5500570e0f76a1" exitCode=137
Sep 15 21:23:15.342126 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:15.341941    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7928-456501"
Sep 15 21:23:15.342126 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:15.341998    2558 scope.go:117] "RemoveContainer" containerID="62e84de4365eb4d8e28bac37ac0d987a64f83bbac128e47b8d5500570e0f76a1"
Sep 15 21:23:15.351190 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:15.351172    2558 scope.go:117] "RemoveContainer" containerID="62e84de4365eb4d8e28bac37ac0d987a64f83bbac128e47b8d5500570e0f76a1"
Sep 15 21:23:15.351482 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:23:15.351462    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62e84de4365eb4d8e28bac37ac0d987a64f83bbac128e47b8d5500570e0f76a1\": container with ID starting with 62e84de4365eb4d8e28bac37ac0d987a64f83bbac128e47b8d5500570e0f76a1 not found: ID does not exist" containerID="62e84de4365eb4d8e28bac37ac0d987a64f83bbac128e47b8d5500570e0f76a1"
Sep 15 21:23:15.351560 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:15.351494    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e84de4365eb4d8e28bac37ac0d987a64f83bbac128e47b8d5500570e0f76a1"} err="failed to get container status \"62e84de4365eb4d8e28bac37ac0d987a64f83bbac128e47b8d5500570e0f76a1\": rpc error: code = NotFound desc = could not find container \"62e84de4365eb4d8e28bac37ac0d987a64f83bbac128e47b8d5500570e0f76a1\": container with ID starting with 62e84de4365eb4d8e28bac37ac0d987a64f83bbac128e47b8d5500570e0f76a1 not found: ID does not exist"
Sep 15 21:23:15.352361 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:23:15.352340    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7928-456501\" is forbidden: User \"system:node:ip-10-0-142-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-207.ec2.internal' and this object" podUID="4d274417-b615-40e1-9702-2d16bd00b33c" pod="opendatahub/test-curl-7928-456501"
Sep 15 21:23:15.944006 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:15.943975    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d274417-b615-40e1-9702-2d16bd00b33c" path="/var/lib/kubelet/pods/4d274417-b615-40e1-9702-2d16bd00b33c/volumes"
Sep 15 21:23:16.575704 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:16.575671    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75"]
Sep 15 21:23:16.575978 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:16.575948    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75" podUID="20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b" containerName="maas-api" containerID="cri-o://43262ec2e8ed54c3adbb9523704f399476c6e4ba0da3c156859e1df1c7bd3142" gracePeriod=30
Sep 15 21:23:16.829905 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:16.829837    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75"
Sep 15 21:23:16.939838 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:16.939809    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b-openshift-service-ca\") pod \"20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b\" (UID: \"20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b\") "
Sep 15 21:23:16.940036 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:16.939874    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwm9t\" (UniqueName: \"kubernetes.io/projected/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b-kube-api-access-lwm9t\") pod \"20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b\" (UID: \"20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b\") "
Sep 15 21:23:16.940036 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:16.939895    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b-metrics-certs\") pod \"20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b\" (UID: \"20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b\") "
Sep 15 21:23:16.940036 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:16.939937    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b-maas-api-tls\") pod \"20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b\" (UID: \"20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b\") "
Sep 15 21:23:16.940304 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:16.940209    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b" (UID: "20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:23:16.942147 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:16.942114    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b" (UID: "20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:23:16.942249 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:16.942155    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b-kube-api-access-lwm9t" (OuterVolumeSpecName: "kube-api-access-lwm9t") pod "20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b" (UID: "20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b"). InnerVolumeSpecName "kube-api-access-lwm9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:23:16.942249 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:16.942176    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b" (UID: "20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:23:17.040762 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:17.040730    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b-openshift-service-ca\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:17.040762 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:17.040757    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwm9t\" (UniqueName: \"kubernetes.io/projected/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b-kube-api-access-lwm9t\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:17.040762 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:17.040767    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b-metrics-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:17.040762 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:17.040776    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b-maas-api-tls\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:17.357318 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:17.357285    2558 generic.go:356] "Generic (PLEG): container finished" podID="20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b" containerID="43262ec2e8ed54c3adbb9523704f399476c6e4ba0da3c156859e1df1c7bd3142" exitCode=0
Sep 15 21:23:17.357522 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:17.357328    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75"
Sep 15 21:23:17.357522 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:17.357372    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75" event={"ID":"20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b","Type":"ContainerDied","Data":"43262ec2e8ed54c3adbb9523704f399476c6e4ba0da3c156859e1df1c7bd3142"}
Sep 15 21:23:17.357522 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:17.357423    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75" event={"ID":"20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b","Type":"ContainerDied","Data":"4f20979867f16ef7ebb618d01aa206d30a3b306751bc034e1256adad05800e70"}
Sep 15 21:23:17.357522 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:17.357450    2558 scope.go:117] "RemoveContainer" containerID="43262ec2e8ed54c3adbb9523704f399476c6e4ba0da3c156859e1df1c7bd3142"
Sep 15 21:23:17.366974 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:17.366955    2558 scope.go:117] "RemoveContainer" containerID="43262ec2e8ed54c3adbb9523704f399476c6e4ba0da3c156859e1df1c7bd3142"
Sep 15 21:23:17.367294 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:23:17.367277    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43262ec2e8ed54c3adbb9523704f399476c6e4ba0da3c156859e1df1c7bd3142\": container with ID starting with 43262ec2e8ed54c3adbb9523704f399476c6e4ba0da3c156859e1df1c7bd3142 not found: ID does not exist" containerID="43262ec2e8ed54c3adbb9523704f399476c6e4ba0da3c156859e1df1c7bd3142"
Sep 15 21:23:17.367354 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:17.367304    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43262ec2e8ed54c3adbb9523704f399476c6e4ba0da3c156859e1df1c7bd3142"} err="failed to get container status \"43262ec2e8ed54c3adbb9523704f399476c6e4ba0da3c156859e1df1c7bd3142\": rpc error: code = NotFound desc = could not find container \"43262ec2e8ed54c3adbb9523704f399476c6e4ba0da3c156859e1df1c7bd3142\": container with ID starting with 43262ec2e8ed54c3adbb9523704f399476c6e4ba0da3c156859e1df1c7bd3142 not found: ID does not exist"
Sep 15 21:23:17.381920 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:17.381892    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75"]
Sep 15 21:23:17.388731 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:17.388703    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7c59a9-647f55ff5b-jnw75"]
Sep 15 21:23:17.942656 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:17.942623    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b" path="/var/lib/kubelet/pods/20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b/volumes"
Sep 15 21:23:22.993105 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:22.993070    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"]
Sep 15 21:23:22.993528 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:22.993280    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp" podUID="23930493-5ede-47bc-a2b8-437761d1f50f" containerName="istio-proxy" containerID="cri-o://99e6351ee15eb1b7f49dedfc181b1241bc5ddcd520e435250abea6a1555c1c2f" gracePeriod=30
Sep 15 21:23:23.983546 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:23.983515    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:23:27.286064 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:27.286024    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64" podUID="63e7736a-4199-4a75-8b09-b6cfd063c1e6" containerName="main" containerID="cri-o://e80aa38f2715816f4e2699482520fe1652d81045bef55e981277ab3490c998d6" gracePeriod=45
Sep 15 21:23:27.415862 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:27.415833    2558 generic.go:356] "Generic (PLEG): container finished" podID="63e7736a-4199-4a75-8b09-b6cfd063c1e6" containerID="e80aa38f2715816f4e2699482520fe1652d81045bef55e981277ab3490c998d6" exitCode=0
Sep 15 21:23:27.415989 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:27.415914    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64" event={"ID":"63e7736a-4199-4a75-8b09-b6cfd063c1e6","Type":"ContainerDied","Data":"e80aa38f2715816f4e2699482520fe1652d81045bef55e981277ab3490c998d6"}
Sep 15 21:23:27.541806 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:27.541734    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:23:27.641330 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:27.641293    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63e7736a-4199-4a75-8b09-b6cfd063c1e6-home\") pod \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\" (UID: \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\") "
Sep 15 21:23:27.641533 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:27.641364    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63e7736a-4199-4a75-8b09-b6cfd063c1e6-dshm\") pod \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\" (UID: \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\") "
Sep 15 21:23:27.641533 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:27.641420    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63e7736a-4199-4a75-8b09-b6cfd063c1e6-tmp-dir\") pod \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\" (UID: \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\") "
Sep 15 21:23:27.641533 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:27.641449    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63e7736a-4199-4a75-8b09-b6cfd063c1e6-tls-certs\") pod \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\" (UID: \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\") "
Sep 15 21:23:27.641533 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:27.641481    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63e7736a-4199-4a75-8b09-b6cfd063c1e6-model-cache\") pod \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\" (UID: \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\") "
Sep 15 21:23:27.641533 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:27.641507    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48xd6\" (UniqueName: \"kubernetes.io/projected/63e7736a-4199-4a75-8b09-b6cfd063c1e6-kube-api-access-48xd6\") pod \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\" (UID: \"63e7736a-4199-4a75-8b09-b6cfd063c1e6\") "
Sep 15 21:23:27.641804 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:27.641634    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e7736a-4199-4a75-8b09-b6cfd063c1e6-home" (OuterVolumeSpecName: "home") pod "63e7736a-4199-4a75-8b09-b6cfd063c1e6" (UID: "63e7736a-4199-4a75-8b09-b6cfd063c1e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:23:27.641804 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:27.641647    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e7736a-4199-4a75-8b09-b6cfd063c1e6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "63e7736a-4199-4a75-8b09-b6cfd063c1e6" (UID: "63e7736a-4199-4a75-8b09-b6cfd063c1e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:23:27.641804 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:27.641747    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e7736a-4199-4a75-8b09-b6cfd063c1e6-model-cache" (OuterVolumeSpecName: "model-cache") pod "63e7736a-4199-4a75-8b09-b6cfd063c1e6" (UID: "63e7736a-4199-4a75-8b09-b6cfd063c1e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:23:27.641912 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:27.641847    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63e7736a-4199-4a75-8b09-b6cfd063c1e6-tmp-dir\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:27.641912 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:27.641860    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63e7736a-4199-4a75-8b09-b6cfd063c1e6-model-cache\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:27.641912 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:27.641868    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63e7736a-4199-4a75-8b09-b6cfd063c1e6-home\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:27.643599 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:27.643570    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e7736a-4199-4a75-8b09-b6cfd063c1e6-dshm" (OuterVolumeSpecName: "dshm") pod "63e7736a-4199-4a75-8b09-b6cfd063c1e6" (UID: "63e7736a-4199-4a75-8b09-b6cfd063c1e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:23:27.643721 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:27.643615    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63e7736a-4199-4a75-8b09-b6cfd063c1e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "63e7736a-4199-4a75-8b09-b6cfd063c1e6" (UID: "63e7736a-4199-4a75-8b09-b6cfd063c1e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:23:27.643721 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:27.643656    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e7736a-4199-4a75-8b09-b6cfd063c1e6-kube-api-access-48xd6" (OuterVolumeSpecName: "kube-api-access-48xd6") pod "63e7736a-4199-4a75-8b09-b6cfd063c1e6" (UID: "63e7736a-4199-4a75-8b09-b6cfd063c1e6"). InnerVolumeSpecName "kube-api-access-48xd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:23:27.743250 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:27.743215    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48xd6\" (UniqueName: \"kubernetes.io/projected/63e7736a-4199-4a75-8b09-b6cfd063c1e6-kube-api-access-48xd6\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:27.743250 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:27.743245    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63e7736a-4199-4a75-8b09-b6cfd063c1e6-dshm\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:27.743250 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:27.743257    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63e7736a-4199-4a75-8b09-b6cfd063c1e6-tls-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:28.230980 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.230955    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:23:28.349293 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.349265    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-workload-certs\") pod \"23930493-5ede-47bc-a2b8-437761d1f50f\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") "
Sep 15 21:23:28.349641 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.349302    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23930493-5ede-47bc-a2b8-437761d1f50f-istio-token\") pod \"23930493-5ede-47bc-a2b8-437761d1f50f\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") "
Sep 15 21:23:28.349641 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.349331    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-istio-envoy\") pod \"23930493-5ede-47bc-a2b8-437761d1f50f\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") "
Sep 15 21:23:28.349641 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.349356    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-workload-socket\") pod \"23930493-5ede-47bc-a2b8-437761d1f50f\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") "
Sep 15 21:23:28.349641 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.349393    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23930493-5ede-47bc-a2b8-437761d1f50f-istiod-ca-cert\") pod \"23930493-5ede-47bc-a2b8-437761d1f50f\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") "
Sep 15 21:23:28.349641 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.349445    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-istio-data\") pod \"23930493-5ede-47bc-a2b8-437761d1f50f\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") "
Sep 15 21:23:28.349641 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.349483    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtmm9\" (UniqueName: \"kubernetes.io/projected/23930493-5ede-47bc-a2b8-437761d1f50f-kube-api-access-dtmm9\") pod \"23930493-5ede-47bc-a2b8-437761d1f50f\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") "
Sep 15 21:23:28.349641 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.349518    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-credential-socket\") pod \"23930493-5ede-47bc-a2b8-437761d1f50f\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") "
Sep 15 21:23:28.349641 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.349531    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "23930493-5ede-47bc-a2b8-437761d1f50f" (UID: "23930493-5ede-47bc-a2b8-437761d1f50f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:23:28.349641 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.349545    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23930493-5ede-47bc-a2b8-437761d1f50f-istio-podinfo\") pod \"23930493-5ede-47bc-a2b8-437761d1f50f\" (UID: \"23930493-5ede-47bc-a2b8-437761d1f50f\") "
Sep 15 21:23:28.350064 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.349735    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "23930493-5ede-47bc-a2b8-437761d1f50f" (UID: "23930493-5ede-47bc-a2b8-437761d1f50f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:23:28.350064 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.349753    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-istio-data" (OuterVolumeSpecName: "istio-data") pod "23930493-5ede-47bc-a2b8-437761d1f50f" (UID: "23930493-5ede-47bc-a2b8-437761d1f50f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:23:28.350064 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.349928    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "23930493-5ede-47bc-a2b8-437761d1f50f" (UID: "23930493-5ede-47bc-a2b8-437761d1f50f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:23:28.350064 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.350012    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23930493-5ede-47bc-a2b8-437761d1f50f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "23930493-5ede-47bc-a2b8-437761d1f50f" (UID: "23930493-5ede-47bc-a2b8-437761d1f50f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:23:28.350064 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.350034    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-workload-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:28.350262 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.350068    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-workload-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:28.350262 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.350083    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-istio-data\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:28.350262 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.350096    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-credential-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:28.351662 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.351639    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23930493-5ede-47bc-a2b8-437761d1f50f-kube-api-access-dtmm9" (OuterVolumeSpecName: "kube-api-access-dtmm9") pod "23930493-5ede-47bc-a2b8-437761d1f50f" (UID: "23930493-5ede-47bc-a2b8-437761d1f50f"). InnerVolumeSpecName "kube-api-access-dtmm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:23:28.351758 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.351688    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "23930493-5ede-47bc-a2b8-437761d1f50f" (UID: "23930493-5ede-47bc-a2b8-437761d1f50f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:23:28.351933 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.351912    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23930493-5ede-47bc-a2b8-437761d1f50f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "23930493-5ede-47bc-a2b8-437761d1f50f" (UID: "23930493-5ede-47bc-a2b8-437761d1f50f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:23:28.352030 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.351947    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23930493-5ede-47bc-a2b8-437761d1f50f-istio-token" (OuterVolumeSpecName: "istio-token") pod "23930493-5ede-47bc-a2b8-437761d1f50f" (UID: "23930493-5ede-47bc-a2b8-437761d1f50f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:23:28.396868 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.396840    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq" podUID="33bfb8a8-523a-44a7-bcd3-093863d0abbb" containerName="main" containerID="cri-o://ac598b1455eaa8caefb5113252cb92ed61b8e591a319bebca7b0ca29da9a2fcc" gracePeriod=45
Sep 15 21:23:28.422619 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.422596    2558 generic.go:356] "Generic (PLEG): container finished" podID="23930493-5ede-47bc-a2b8-437761d1f50f" containerID="99e6351ee15eb1b7f49dedfc181b1241bc5ddcd520e435250abea6a1555c1c2f" exitCode=0
Sep 15 21:23:28.422719 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.422642    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"
Sep 15 21:23:28.422719 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.422681    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp" event={"ID":"23930493-5ede-47bc-a2b8-437761d1f50f","Type":"ContainerDied","Data":"99e6351ee15eb1b7f49dedfc181b1241bc5ddcd520e435250abea6a1555c1c2f"}
Sep 15 21:23:28.422843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.422739    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp" event={"ID":"23930493-5ede-47bc-a2b8-437761d1f50f","Type":"ContainerDied","Data":"2c3dc35e914a9846f461bd086cd516fab5981a2ac71a8233dc7d528bc91d2201"}
Sep 15 21:23:28.422843 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.422772    2558 scope.go:117] "RemoveContainer" containerID="99e6351ee15eb1b7f49dedfc181b1241bc5ddcd520e435250abea6a1555c1c2f"
Sep 15 21:23:28.424161 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.424145    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"
Sep 15 21:23:28.424230 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.424164    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64" event={"ID":"63e7736a-4199-4a75-8b09-b6cfd063c1e6","Type":"ContainerDied","Data":"cdcddd5284667517df1dd61675b8e9873d9248023ee8b61aecbd33031c505e8a"}
Sep 15 21:23:28.450937 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.450914    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23930493-5ede-47bc-a2b8-437761d1f50f-istio-token\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:28.450937 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.450934    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23930493-5ede-47bc-a2b8-437761d1f50f-istio-envoy\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:28.451081 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.450948    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23930493-5ede-47bc-a2b8-437761d1f50f-istiod-ca-cert\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:28.451081 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.450958    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtmm9\" (UniqueName: \"kubernetes.io/projected/23930493-5ede-47bc-a2b8-437761d1f50f-kube-api-access-dtmm9\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:28.451081 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.450966    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23930493-5ede-47bc-a2b8-437761d1f50f-istio-podinfo\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:28.480878 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.480826    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"]
Sep 15 21:23:28.486743 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.486719    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2f34ef/auth-test-model-2f34ef-kserve-78ff5f89c9-5sx64"]
Sep 15 21:23:28.499633 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.499612    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"]
Sep 15 21:23:28.502512 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.502493    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7c59a9-openshift-default-688c67bffd-fxscp"]
Sep 15 21:23:28.508486 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.508468    2558 scope.go:117] "RemoveContainer" containerID="bd9f5f07fd72c58e5e7461548376b0c5dd75d078423cc4521277837767173380"
Sep 15 21:23:28.516795 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.516781    2558 scope.go:117] "RemoveContainer" containerID="99e6351ee15eb1b7f49dedfc181b1241bc5ddcd520e435250abea6a1555c1c2f"
Sep 15 21:23:28.517081 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:23:28.517062    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99e6351ee15eb1b7f49dedfc181b1241bc5ddcd520e435250abea6a1555c1c2f\": container with ID starting with 99e6351ee15eb1b7f49dedfc181b1241bc5ddcd520e435250abea6a1555c1c2f not found: ID does not exist" containerID="99e6351ee15eb1b7f49dedfc181b1241bc5ddcd520e435250abea6a1555c1c2f"
Sep 15 21:23:28.517138 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.517088    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99e6351ee15eb1b7f49dedfc181b1241bc5ddcd520e435250abea6a1555c1c2f"} err="failed to get container status \"99e6351ee15eb1b7f49dedfc181b1241bc5ddcd520e435250abea6a1555c1c2f\": rpc error: code = NotFound desc = could not find container \"99e6351ee15eb1b7f49dedfc181b1241bc5ddcd520e435250abea6a1555c1c2f\": container with ID starting with 99e6351ee15eb1b7f49dedfc181b1241bc5ddcd520e435250abea6a1555c1c2f not found: ID does not exist"
Sep 15 21:23:28.517138 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.517104    2558 scope.go:117] "RemoveContainer" containerID="bd9f5f07fd72c58e5e7461548376b0c5dd75d078423cc4521277837767173380"
Sep 15 21:23:28.517393 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:23:28.517370    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd9f5f07fd72c58e5e7461548376b0c5dd75d078423cc4521277837767173380\": container with ID starting with bd9f5f07fd72c58e5e7461548376b0c5dd75d078423cc4521277837767173380 not found: ID does not exist" containerID="bd9f5f07fd72c58e5e7461548376b0c5dd75d078423cc4521277837767173380"
Sep 15 21:23:28.517463 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.517397    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9f5f07fd72c58e5e7461548376b0c5dd75d078423cc4521277837767173380"} err="failed to get container status \"bd9f5f07fd72c58e5e7461548376b0c5dd75d078423cc4521277837767173380\": rpc error: code = NotFound desc = could not find container \"bd9f5f07fd72c58e5e7461548376b0c5dd75d078423cc4521277837767173380\": container with ID starting with bd9f5f07fd72c58e5e7461548376b0c5dd75d078423cc4521277837767173380 not found: ID does not exist"
Sep 15 21:23:28.517463 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.517413    2558 scope.go:117] "RemoveContainer" containerID="e80aa38f2715816f4e2699482520fe1652d81045bef55e981277ab3490c998d6"
Sep 15 21:23:28.651162 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.651132    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:23:28.753698 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.753613    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33bfb8a8-523a-44a7-bcd3-093863d0abbb-model-cache\") pod \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\" (UID: \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\") "
Sep 15 21:23:28.753698 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.753672    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33bfb8a8-523a-44a7-bcd3-093863d0abbb-home\") pod \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\" (UID: \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\") "
Sep 15 21:23:28.753911 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.753711    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33bfb8a8-523a-44a7-bcd3-093863d0abbb-tls-certs\") pod \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\" (UID: \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\") "
Sep 15 21:23:28.753911 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.753760    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cngx\" (UniqueName: \"kubernetes.io/projected/33bfb8a8-523a-44a7-bcd3-093863d0abbb-kube-api-access-8cngx\") pod \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\" (UID: \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\") "
Sep 15 21:23:28.753911 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.753788    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33bfb8a8-523a-44a7-bcd3-093863d0abbb-tmp-dir\") pod \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\" (UID: \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\") "
Sep 15 21:23:28.753911 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.753838    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33bfb8a8-523a-44a7-bcd3-093863d0abbb-dshm\") pod \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\" (UID: \"33bfb8a8-523a-44a7-bcd3-093863d0abbb\") "
Sep 15 21:23:28.754173 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.753906    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bfb8a8-523a-44a7-bcd3-093863d0abbb-model-cache" (OuterVolumeSpecName: "model-cache") pod "33bfb8a8-523a-44a7-bcd3-093863d0abbb" (UID: "33bfb8a8-523a-44a7-bcd3-093863d0abbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:23:28.754173 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.753962    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bfb8a8-523a-44a7-bcd3-093863d0abbb-home" (OuterVolumeSpecName: "home") pod "33bfb8a8-523a-44a7-bcd3-093863d0abbb" (UID: "33bfb8a8-523a-44a7-bcd3-093863d0abbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:23:28.754173 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.754119    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bfb8a8-523a-44a7-bcd3-093863d0abbb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33bfb8a8-523a-44a7-bcd3-093863d0abbb" (UID: "33bfb8a8-523a-44a7-bcd3-093863d0abbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:23:28.754173 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.754143    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33bfb8a8-523a-44a7-bcd3-093863d0abbb-home\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:28.754173 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.754157    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33bfb8a8-523a-44a7-bcd3-093863d0abbb-model-cache\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:28.756008 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.755986    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33bfb8a8-523a-44a7-bcd3-093863d0abbb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33bfb8a8-523a-44a7-bcd3-093863d0abbb" (UID: "33bfb8a8-523a-44a7-bcd3-093863d0abbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:23:28.756095 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.756036    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bfb8a8-523a-44a7-bcd3-093863d0abbb-dshm" (OuterVolumeSpecName: "dshm") pod "33bfb8a8-523a-44a7-bcd3-093863d0abbb" (UID: "33bfb8a8-523a-44a7-bcd3-093863d0abbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:23:28.756095 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.756075    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33bfb8a8-523a-44a7-bcd3-093863d0abbb-kube-api-access-8cngx" (OuterVolumeSpecName: "kube-api-access-8cngx") pod "33bfb8a8-523a-44a7-bcd3-093863d0abbb" (UID: "33bfb8a8-523a-44a7-bcd3-093863d0abbb"). InnerVolumeSpecName "kube-api-access-8cngx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:23:28.855118 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.855084    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33bfb8a8-523a-44a7-bcd3-093863d0abbb-tls-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:28.855118 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.855112    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cngx\" (UniqueName: \"kubernetes.io/projected/33bfb8a8-523a-44a7-bcd3-093863d0abbb-kube-api-access-8cngx\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:28.855118 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.855124    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33bfb8a8-523a-44a7-bcd3-093863d0abbb-tmp-dir\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:28.855366 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:28.855133    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33bfb8a8-523a-44a7-bcd3-093863d0abbb-dshm\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:29.432777 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:29.432743    2558 generic.go:356] "Generic (PLEG): container finished" podID="33bfb8a8-523a-44a7-bcd3-093863d0abbb" containerID="ac598b1455eaa8caefb5113252cb92ed61b8e591a319bebca7b0ca29da9a2fcc" exitCode=0
Sep 15 21:23:29.433340 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:29.432789    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"
Sep 15 21:23:29.433340 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:29.432815    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq" event={"ID":"33bfb8a8-523a-44a7-bcd3-093863d0abbb","Type":"ContainerDied","Data":"ac598b1455eaa8caefb5113252cb92ed61b8e591a319bebca7b0ca29da9a2fcc"}
Sep 15 21:23:29.433340 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:29.432862    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq" event={"ID":"33bfb8a8-523a-44a7-bcd3-093863d0abbb","Type":"ContainerDied","Data":"d29e74d1d751a5fa8a6a5888b0f7669182485523d0576fa4cae05a6fcb1d779f"}
Sep 15 21:23:29.433340 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:29.432888    2558 scope.go:117] "RemoveContainer" containerID="ac598b1455eaa8caefb5113252cb92ed61b8e591a319bebca7b0ca29da9a2fcc"
Sep 15 21:23:29.443640 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:29.443624    2558 scope.go:117] "RemoveContainer" containerID="ac598b1455eaa8caefb5113252cb92ed61b8e591a319bebca7b0ca29da9a2fcc"
Sep 15 21:23:29.443945 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:23:29.443924    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac598b1455eaa8caefb5113252cb92ed61b8e591a319bebca7b0ca29da9a2fcc\": container with ID starting with ac598b1455eaa8caefb5113252cb92ed61b8e591a319bebca7b0ca29da9a2fcc not found: ID does not exist" containerID="ac598b1455eaa8caefb5113252cb92ed61b8e591a319bebca7b0ca29da9a2fcc"
Sep 15 21:23:29.443990 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:29.443954    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac598b1455eaa8caefb5113252cb92ed61b8e591a319bebca7b0ca29da9a2fcc"} err="failed to get container status \"ac598b1455eaa8caefb5113252cb92ed61b8e591a319bebca7b0ca29da9a2fcc\": rpc error: code = NotFound desc = could not find container \"ac598b1455eaa8caefb5113252cb92ed61b8e591a319bebca7b0ca29da9a2fcc\": container with ID starting with ac598b1455eaa8caefb5113252cb92ed61b8e591a319bebca7b0ca29da9a2fcc not found: ID does not exist"
Sep 15 21:23:29.459412 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:29.459390    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"]
Sep 15 21:23:29.465806 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:29.465785    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-41ffdf/auth-test-model-41ffdf-kserve-65cd9b8d55-qpjsq"]
Sep 15 21:23:29.947617 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:29.947581    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23930493-5ede-47bc-a2b8-437761d1f50f" path="/var/lib/kubelet/pods/23930493-5ede-47bc-a2b8-437761d1f50f/volumes"
Sep 15 21:23:29.947997 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:29.947984    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33bfb8a8-523a-44a7-bcd3-093863d0abbb" path="/var/lib/kubelet/pods/33bfb8a8-523a-44a7-bcd3-093863d0abbb/volumes"
Sep 15 21:23:29.948357 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:29.948345    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63e7736a-4199-4a75-8b09-b6cfd063c1e6" path="/var/lib/kubelet/pods/63e7736a-4199-4a75-8b09-b6cfd063c1e6/volumes"
Sep 15 21:23:33.838319 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:33.838284    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf"]
Sep 15 21:23:33.840810 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:33.838541    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf" podUID="113066b8-2eb4-4d91-9c53-9451dd6067c9" containerName="maas-api" containerID="cri-o://e9065a7a4814b9b0a827ce906fd233e9d3a917cd675a13ddcab35177777c21a2" gracePeriod=30
Sep 15 21:23:33.886367 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:33.886332    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf" podUID="113066b8-2eb4-4d91-9c53-9451dd6067c9" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.64:8443/health\": dial tcp 10.133.0.64:8443: connect: connection refused"
Sep 15 21:23:34.079513 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:34.079486    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf"
Sep 15 21:23:34.203985 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:34.203954    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/113066b8-2eb4-4d91-9c53-9451dd6067c9-openshift-service-ca\") pod \"113066b8-2eb4-4d91-9c53-9451dd6067c9\" (UID: \"113066b8-2eb4-4d91-9c53-9451dd6067c9\") "
Sep 15 21:23:34.204178 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:34.203996    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/113066b8-2eb4-4d91-9c53-9451dd6067c9-metrics-certs\") pod \"113066b8-2eb4-4d91-9c53-9451dd6067c9\" (UID: \"113066b8-2eb4-4d91-9c53-9451dd6067c9\") "
Sep 15 21:23:34.204178 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:34.204027    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzmdk\" (UniqueName: \"kubernetes.io/projected/113066b8-2eb4-4d91-9c53-9451dd6067c9-kube-api-access-bzmdk\") pod \"113066b8-2eb4-4d91-9c53-9451dd6067c9\" (UID: \"113066b8-2eb4-4d91-9c53-9451dd6067c9\") "
Sep 15 21:23:34.204178 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:34.204107    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/113066b8-2eb4-4d91-9c53-9451dd6067c9-maas-api-tls\") pod \"113066b8-2eb4-4d91-9c53-9451dd6067c9\" (UID: \"113066b8-2eb4-4d91-9c53-9451dd6067c9\") "
Sep 15 21:23:34.204552 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:34.204526    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/113066b8-2eb4-4d91-9c53-9451dd6067c9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "113066b8-2eb4-4d91-9c53-9451dd6067c9" (UID: "113066b8-2eb4-4d91-9c53-9451dd6067c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:23:34.206113 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:34.206090    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113066b8-2eb4-4d91-9c53-9451dd6067c9-kube-api-access-bzmdk" (OuterVolumeSpecName: "kube-api-access-bzmdk") pod "113066b8-2eb4-4d91-9c53-9451dd6067c9" (UID: "113066b8-2eb4-4d91-9c53-9451dd6067c9"). InnerVolumeSpecName "kube-api-access-bzmdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:23:34.206197 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:34.206117    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/113066b8-2eb4-4d91-9c53-9451dd6067c9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "113066b8-2eb4-4d91-9c53-9451dd6067c9" (UID: "113066b8-2eb4-4d91-9c53-9451dd6067c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:23:34.206248 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:34.206199    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/113066b8-2eb4-4d91-9c53-9451dd6067c9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "113066b8-2eb4-4d91-9c53-9451dd6067c9" (UID: "113066b8-2eb4-4d91-9c53-9451dd6067c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:23:34.305155 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:34.305120    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/113066b8-2eb4-4d91-9c53-9451dd6067c9-openshift-service-ca\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:34.305155 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:34.305152    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/113066b8-2eb4-4d91-9c53-9451dd6067c9-metrics-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:34.305155 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:34.305160    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzmdk\" (UniqueName: \"kubernetes.io/projected/113066b8-2eb4-4d91-9c53-9451dd6067c9-kube-api-access-bzmdk\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:34.305373 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:34.305168    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/113066b8-2eb4-4d91-9c53-9451dd6067c9-maas-api-tls\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:23:34.464190 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:34.464115    2558 generic.go:356] "Generic (PLEG): container finished" podID="113066b8-2eb4-4d91-9c53-9451dd6067c9" containerID="e9065a7a4814b9b0a827ce906fd233e9d3a917cd675a13ddcab35177777c21a2" exitCode=0
Sep 15 21:23:34.464190 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:34.464159    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf"
Sep 15 21:23:34.464376 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:34.464201    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf" event={"ID":"113066b8-2eb4-4d91-9c53-9451dd6067c9","Type":"ContainerDied","Data":"e9065a7a4814b9b0a827ce906fd233e9d3a917cd675a13ddcab35177777c21a2"}
Sep 15 21:23:34.464376 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:34.464257    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf" event={"ID":"113066b8-2eb4-4d91-9c53-9451dd6067c9","Type":"ContainerDied","Data":"071808cf8aecc343d478b83ee7953fc537b6578cbd64908103e0edf5c4ee135a"}
Sep 15 21:23:34.464376 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:34.464284    2558 scope.go:117] "RemoveContainer" containerID="e9065a7a4814b9b0a827ce906fd233e9d3a917cd675a13ddcab35177777c21a2"
Sep 15 21:23:34.473797 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:34.473780    2558 scope.go:117] "RemoveContainer" containerID="e9065a7a4814b9b0a827ce906fd233e9d3a917cd675a13ddcab35177777c21a2"
Sep 15 21:23:34.474039 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:23:34.474019    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9065a7a4814b9b0a827ce906fd233e9d3a917cd675a13ddcab35177777c21a2\": container with ID starting with e9065a7a4814b9b0a827ce906fd233e9d3a917cd675a13ddcab35177777c21a2 not found: ID does not exist" containerID="e9065a7a4814b9b0a827ce906fd233e9d3a917cd675a13ddcab35177777c21a2"
Sep 15 21:23:34.474162 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:34.474070    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9065a7a4814b9b0a827ce906fd233e9d3a917cd675a13ddcab35177777c21a2"} err="failed to get container status \"e9065a7a4814b9b0a827ce906fd233e9d3a917cd675a13ddcab35177777c21a2\": rpc error: code = NotFound desc = could not find container \"e9065a7a4814b9b0a827ce906fd233e9d3a917cd675a13ddcab35177777c21a2\": container with ID starting with e9065a7a4814b9b0a827ce906fd233e9d3a917cd675a13ddcab35177777c21a2 not found: ID does not exist"
Sep 15 21:23:34.491264 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:34.491244    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf"]
Sep 15 21:23:34.497593 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:34.497572    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2c7774-9f885ddf5-x96qf"]
Sep 15 21:23:35.942240 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:35.942207    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="113066b8-2eb4-4d91-9c53-9451dd6067c9" path="/var/lib/kubelet/pods/113066b8-2eb4-4d91-9c53-9451dd6067c9/volumes"
Sep 15 21:23:37.885320 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:37.885285    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:23:55.217003 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:23:55.216961    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:24:26.899916 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:26.899882    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:24:40.219090 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:40.219060    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:24:43.585262 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.585229    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7928-d62926"]
Sep 15 21:24:43.585644 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.585562    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d274417-b615-40e1-9702-2d16bd00b33c" containerName="test-curl-7928-456501"
Sep 15 21:24:43.585644 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.585575    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d274417-b615-40e1-9702-2d16bd00b33c" containerName="test-curl-7928-456501"
Sep 15 21:24:43.585644 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.585584    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="113066b8-2eb4-4d91-9c53-9451dd6067c9" containerName="maas-api"
Sep 15 21:24:43.585644 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.585589    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="113066b8-2eb4-4d91-9c53-9451dd6067c9" containerName="maas-api"
Sep 15 21:24:43.585644 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.585599    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23930493-5ede-47bc-a2b8-437761d1f50f" containerName="istio-proxy"
Sep 15 21:24:43.585644 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.585605    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="23930493-5ede-47bc-a2b8-437761d1f50f" containerName="istio-proxy"
Sep 15 21:24:43.585644 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.585614    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33bfb8a8-523a-44a7-bcd3-093863d0abbb" containerName="main"
Sep 15 21:24:43.585644 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.585619    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="33bfb8a8-523a-44a7-bcd3-093863d0abbb" containerName="main"
Sep 15 21:24:43.585644 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.585627    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b" containerName="maas-api"
Sep 15 21:24:43.585644 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.585632    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b" containerName="maas-api"
Sep 15 21:24:43.585644 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.585639    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23930493-5ede-47bc-a2b8-437761d1f50f" containerName="istio-proxy"
Sep 15 21:24:43.585644 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.585644    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="23930493-5ede-47bc-a2b8-437761d1f50f" containerName="istio-proxy"
Sep 15 21:24:43.585644 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.585650    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63e7736a-4199-4a75-8b09-b6cfd063c1e6" containerName="main"
Sep 15 21:24:43.586133 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.585654    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="63e7736a-4199-4a75-8b09-b6cfd063c1e6" containerName="main"
Sep 15 21:24:43.586133 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.585721    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="63e7736a-4199-4a75-8b09-b6cfd063c1e6" containerName="main"
Sep 15 21:24:43.586133 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.585730    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="20ecac1d-85e4-45b3-844c-2ccb5cc5ed4b" containerName="maas-api"
Sep 15 21:24:43.586133 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.585735    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="113066b8-2eb4-4d91-9c53-9451dd6067c9" containerName="maas-api"
Sep 15 21:24:43.586133 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.585742    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="23930493-5ede-47bc-a2b8-437761d1f50f" containerName="istio-proxy"
Sep 15 21:24:43.586133 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.585747    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="33bfb8a8-523a-44a7-bcd3-093863d0abbb" containerName="main"
Sep 15 21:24:43.586133 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.585753    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="23930493-5ede-47bc-a2b8-437761d1f50f" containerName="istio-proxy"
Sep 15 21:24:43.586133 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.585759    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d274417-b615-40e1-9702-2d16bd00b33c" containerName="test-curl-7928-456501"
Sep 15 21:24:43.588728 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.588712    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7928-d62926"
Sep 15 21:24:43.591951 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.591921    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-r6zqd\""
Sep 15 21:24:43.595799 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.595775    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7928-d62926"]
Sep 15 21:24:43.607590 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.607565    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw9kl\" (UniqueName: \"kubernetes.io/projected/7f639aa5-16a3-4436-94ad-093c3c9f6854-kube-api-access-dw9kl\") pod \"test-curl-7928-d62926\" (UID: \"7f639aa5-16a3-4436-94ad-093c3c9f6854\") " pod="opendatahub/test-curl-7928-d62926"
Sep 15 21:24:43.707954 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.707932    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw9kl\" (UniqueName: \"kubernetes.io/projected/7f639aa5-16a3-4436-94ad-093c3c9f6854-kube-api-access-dw9kl\") pod \"test-curl-7928-d62926\" (UID: \"7f639aa5-16a3-4436-94ad-093c3c9f6854\") " pod="opendatahub/test-curl-7928-d62926"
Sep 15 21:24:43.717897 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.717869    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw9kl\" (UniqueName: \"kubernetes.io/projected/7f639aa5-16a3-4436-94ad-093c3c9f6854-kube-api-access-dw9kl\") pod \"test-curl-7928-d62926\" (UID: \"7f639aa5-16a3-4436-94ad-093c3c9f6854\") " pod="opendatahub/test-curl-7928-d62926"
Sep 15 21:24:43.899253 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:43.899171    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7928-d62926"
Sep 15 21:24:44.171556 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:44.171528    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7928-d62926"]
Sep 15 21:24:44.172247 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:24:44.172218    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f639aa5_16a3_4436_94ad_093c3c9f6854.slice/crio-d0be2a7ad3a924bd76cbcc589442f6e53ba44930ba85c222501b9478886c2c05 WatchSource:0}: Error finding container d0be2a7ad3a924bd76cbcc589442f6e53ba44930ba85c222501b9478886c2c05: Status 404 returned error can't find the container with id d0be2a7ad3a924bd76cbcc589442f6e53ba44930ba85c222501b9478886c2c05
Sep 15 21:24:44.832721 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:44.832688    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7928-d62926" event={"ID":"7f639aa5-16a3-4436-94ad-093c3c9f6854","Type":"ContainerStarted","Data":"bd2d018ced3688fcf996c78fee73266ca2da7dd642c24b248d2b6cc7e9d1f9f3"}
Sep 15 21:24:44.832721 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:44.832725    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7928-d62926" event={"ID":"7f639aa5-16a3-4436-94ad-093c3c9f6854","Type":"ContainerStarted","Data":"d0be2a7ad3a924bd76cbcc589442f6e53ba44930ba85c222501b9478886c2c05"}
Sep 15 21:24:44.853514 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:44.853466    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7928-d62926" podStartSLOduration=1.853452013 podStartE2EDuration="1.853452013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:44.850991053 +0000 UTC m=+1603.341636781" watchObservedRunningTime="2026-09-15 21:24:44.853452013 +0000 UTC m=+1603.344097744"
Sep 15 21:24:45.511510 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:45.511481    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7928-d62926"]
Sep 15 21:24:45.516427 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:45.516398    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7928-d62926"]
Sep 15 21:24:45.632763 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:45.632693    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7928-850ac3"]
Sep 15 21:24:45.633036 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:45.633024    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f639aa5-16a3-4436-94ad-093c3c9f6854" containerName="test-curl-7928-d62926"
Sep 15 21:24:45.633117 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:45.633037    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f639aa5-16a3-4436-94ad-093c3c9f6854" containerName="test-curl-7928-d62926"
Sep 15 21:24:45.633117 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:45.633107    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f639aa5-16a3-4436-94ad-093c3c9f6854" containerName="test-curl-7928-d62926"
Sep 15 21:24:45.636000 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:45.635986    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7928-850ac3"
Sep 15 21:24:45.646028 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:45.646002    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7928-850ac3"]
Sep 15 21:24:45.727542 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:45.727515    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nxtr\" (UniqueName: \"kubernetes.io/projected/723f3936-3753-4c84-b03f-28b41c6fc87a-kube-api-access-4nxtr\") pod \"test-curl-7928-850ac3\" (UID: \"723f3936-3753-4c84-b03f-28b41c6fc87a\") " pod="opendatahub/test-curl-7928-850ac3"
Sep 15 21:24:45.828836 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:45.828805    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nxtr\" (UniqueName: \"kubernetes.io/projected/723f3936-3753-4c84-b03f-28b41c6fc87a-kube-api-access-4nxtr\") pod \"test-curl-7928-850ac3\" (UID: \"723f3936-3753-4c84-b03f-28b41c6fc87a\") " pod="opendatahub/test-curl-7928-850ac3"
Sep 15 21:24:45.838153 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:24:45.838131    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7928-d62926" container="test-curl-7928-d62926"
Sep 15 21:24:45.846521 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:45.846498    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nxtr\" (UniqueName: \"kubernetes.io/projected/723f3936-3753-4c84-b03f-28b41c6fc87a-kube-api-access-4nxtr\") pod \"test-curl-7928-850ac3\" (UID: \"723f3936-3753-4c84-b03f-28b41c6fc87a\") " pod="opendatahub/test-curl-7928-850ac3"
Sep 15 21:24:45.944693 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:45.944674    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7928-850ac3"
Sep 15 21:24:46.215483 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:46.215460    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7928-850ac3"]
Sep 15 21:24:46.217181 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:24:46.217157    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod723f3936_3753_4c84_b03f_28b41c6fc87a.slice/crio-a49d222b7988fe3cdb5566997573a5ce004dd58520a8c6f30db4621cfea3d339 WatchSource:0}: Error finding container a49d222b7988fe3cdb5566997573a5ce004dd58520a8c6f30db4621cfea3d339: Status 404 returned error can't find the container with id a49d222b7988fe3cdb5566997573a5ce004dd58520a8c6f30db4621cfea3d339
Sep 15 21:24:46.850355 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:46.850316    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7928-850ac3" event={"ID":"723f3936-3753-4c84-b03f-28b41c6fc87a","Type":"ContainerStarted","Data":"9515e70552523651aef1f0b1e86e62571b87d1da4ce0701bb94efcd0216208ec"}
Sep 15 21:24:46.850797 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:46.850365    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7928-850ac3" event={"ID":"723f3936-3753-4c84-b03f-28b41c6fc87a","Type":"ContainerStarted","Data":"a49d222b7988fe3cdb5566997573a5ce004dd58520a8c6f30db4621cfea3d339"}
Sep 15 21:24:46.850797 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:46.850366    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7928-d62926" podUID="7f639aa5-16a3-4436-94ad-093c3c9f6854" containerName="test-curl-7928-d62926" containerID="cri-o://bd2d018ced3688fcf996c78fee73266ca2da7dd642c24b248d2b6cc7e9d1f9f3" gracePeriod=2
Sep 15 21:24:46.868199 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:46.868154    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7928-850ac3" podStartSLOduration=1.868142598 podStartE2EDuration="1.868142598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:46.866578525 +0000 UTC m=+1605.357224254" watchObservedRunningTime="2026-09-15 21:24:46.868142598 +0000 UTC m=+1605.358788375"
Sep 15 21:24:47.483642 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:47.483606    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7928-850ac3"]
Sep 15 21:24:47.487575 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:47.487548    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7928-850ac3"]
Sep 15 21:24:47.856425 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:24:47.856396    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7928-850ac3" container="test-curl-7928-850ac3"
Sep 15 21:24:48.861063 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:48.861017    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7928-850ac3" podUID="723f3936-3753-4c84-b03f-28b41c6fc87a" containerName="test-curl-7928-850ac3" containerID="cri-o://9515e70552523651aef1f0b1e86e62571b87d1da4ce0701bb94efcd0216208ec" gracePeriod=2
Sep 15 21:24:49.023011 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:49.022985    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7928-d62926"
Sep 15 21:24:49.058317 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:49.058293    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dw9kl\" (UniqueName: \"kubernetes.io/projected/7f639aa5-16a3-4436-94ad-093c3c9f6854-kube-api-access-dw9kl\") pod \"7f639aa5-16a3-4436-94ad-093c3c9f6854\" (UID: \"7f639aa5-16a3-4436-94ad-093c3c9f6854\") "
Sep 15 21:24:49.060170 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:49.060149    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f639aa5-16a3-4436-94ad-093c3c9f6854-kube-api-access-dw9kl" (OuterVolumeSpecName: "kube-api-access-dw9kl") pod "7f639aa5-16a3-4436-94ad-093c3c9f6854" (UID: "7f639aa5-16a3-4436-94ad-093c3c9f6854"). InnerVolumeSpecName "kube-api-access-dw9kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:24:49.159854 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:49.159801    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw9kl\" (UniqueName: \"kubernetes.io/projected/7f639aa5-16a3-4436-94ad-093c3c9f6854-kube-api-access-dw9kl\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:24:49.867284 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:49.867251    2558 generic.go:356] "Generic (PLEG): container finished" podID="7f639aa5-16a3-4436-94ad-093c3c9f6854" containerID="bd2d018ced3688fcf996c78fee73266ca2da7dd642c24b248d2b6cc7e9d1f9f3" exitCode=137
Sep 15 21:24:49.867762 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:49.867295    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7928-d62926"
Sep 15 21:24:49.867762 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:49.867314    2558 scope.go:117] "RemoveContainer" containerID="bd2d018ced3688fcf996c78fee73266ca2da7dd642c24b248d2b6cc7e9d1f9f3"
Sep 15 21:24:49.875971 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:49.875955    2558 scope.go:117] "RemoveContainer" containerID="bd2d018ced3688fcf996c78fee73266ca2da7dd642c24b248d2b6cc7e9d1f9f3"
Sep 15 21:24:49.876248 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:24:49.876228    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd2d018ced3688fcf996c78fee73266ca2da7dd642c24b248d2b6cc7e9d1f9f3\": container with ID starting with bd2d018ced3688fcf996c78fee73266ca2da7dd642c24b248d2b6cc7e9d1f9f3 not found: ID does not exist" containerID="bd2d018ced3688fcf996c78fee73266ca2da7dd642c24b248d2b6cc7e9d1f9f3"
Sep 15 21:24:49.876325 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:49.876255    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd2d018ced3688fcf996c78fee73266ca2da7dd642c24b248d2b6cc7e9d1f9f3"} err="failed to get container status \"bd2d018ced3688fcf996c78fee73266ca2da7dd642c24b248d2b6cc7e9d1f9f3\": rpc error: code = NotFound desc = could not find container \"bd2d018ced3688fcf996c78fee73266ca2da7dd642c24b248d2b6cc7e9d1f9f3\": container with ID starting with bd2d018ced3688fcf996c78fee73266ca2da7dd642c24b248d2b6cc7e9d1f9f3 not found: ID does not exist"
Sep 15 21:24:49.877774 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:24:49.877757    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7928-d62926\" is forbidden: User \"system:node:ip-10-0-142-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-207.ec2.internal' and this object" podUID="7f639aa5-16a3-4436-94ad-093c3c9f6854" pod="opendatahub/test-curl-7928-d62926"
Sep 15 21:24:49.942290 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:49.942266    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f639aa5-16a3-4436-94ad-093c3c9f6854" path="/var/lib/kubelet/pods/7f639aa5-16a3-4436-94ad-093c3c9f6854/volumes"
Sep 15 21:24:51.050974 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:51.050949    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7928-850ac3"
Sep 15 21:24:51.177189 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:51.177168    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nxtr\" (UniqueName: \"kubernetes.io/projected/723f3936-3753-4c84-b03f-28b41c6fc87a-kube-api-access-4nxtr\") pod \"723f3936-3753-4c84-b03f-28b41c6fc87a\" (UID: \"723f3936-3753-4c84-b03f-28b41c6fc87a\") "
Sep 15 21:24:51.179172 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:51.179141    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/723f3936-3753-4c84-b03f-28b41c6fc87a-kube-api-access-4nxtr" (OuterVolumeSpecName: "kube-api-access-4nxtr") pod "723f3936-3753-4c84-b03f-28b41c6fc87a" (UID: "723f3936-3753-4c84-b03f-28b41c6fc87a"). InnerVolumeSpecName "kube-api-access-4nxtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:24:51.278560 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:51.278536    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nxtr\" (UniqueName: \"kubernetes.io/projected/723f3936-3753-4c84-b03f-28b41c6fc87a-kube-api-access-4nxtr\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:24:51.886114 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:51.886077    2558 generic.go:356] "Generic (PLEG): container finished" podID="723f3936-3753-4c84-b03f-28b41c6fc87a" containerID="9515e70552523651aef1f0b1e86e62571b87d1da4ce0701bb94efcd0216208ec" exitCode=137
Sep 15 21:24:51.886316 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:51.886124    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7928-850ac3"
Sep 15 21:24:51.886316 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:51.886138    2558 scope.go:117] "RemoveContainer" containerID="9515e70552523651aef1f0b1e86e62571b87d1da4ce0701bb94efcd0216208ec"
Sep 15 21:24:51.895609 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:51.895593    2558 scope.go:117] "RemoveContainer" containerID="9515e70552523651aef1f0b1e86e62571b87d1da4ce0701bb94efcd0216208ec"
Sep 15 21:24:51.895872 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:24:51.895853    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9515e70552523651aef1f0b1e86e62571b87d1da4ce0701bb94efcd0216208ec\": container with ID starting with 9515e70552523651aef1f0b1e86e62571b87d1da4ce0701bb94efcd0216208ec not found: ID does not exist" containerID="9515e70552523651aef1f0b1e86e62571b87d1da4ce0701bb94efcd0216208ec"
Sep 15 21:24:51.895921 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:51.895880    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9515e70552523651aef1f0b1e86e62571b87d1da4ce0701bb94efcd0216208ec"} err="failed to get container status \"9515e70552523651aef1f0b1e86e62571b87d1da4ce0701bb94efcd0216208ec\": rpc error: code = NotFound desc = could not find container \"9515e70552523651aef1f0b1e86e62571b87d1da4ce0701bb94efcd0216208ec\": container with ID starting with 9515e70552523651aef1f0b1e86e62571b87d1da4ce0701bb94efcd0216208ec not found: ID does not exist"
Sep 15 21:24:51.898263 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:24:51.898241    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7928-850ac3\" is forbidden: User \"system:node:ip-10-0-142-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-207.ec2.internal' and this object" podUID="723f3936-3753-4c84-b03f-28b41c6fc87a" pod="opendatahub/test-curl-7928-850ac3"
Sep 15 21:24:51.943199 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:51.943169    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="723f3936-3753-4c84-b03f-28b41c6fc87a" path="/var/lib/kubelet/pods/723f3936-3753-4c84-b03f-28b41c6fc87a/volumes"
Sep 15 21:24:51.943394 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:24:51.943371    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7928-850ac3\" is forbidden: User \"system:node:ip-10-0-142-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-207.ec2.internal' and this object" podUID="723f3936-3753-4c84-b03f-28b41c6fc87a" pod="opendatahub/test-curl-7928-850ac3"
Sep 15 21:24:53.189530 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:24:53.189499    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:25:01.198249 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.198166    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"]
Sep 15 21:25:01.198732 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.198710    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="723f3936-3753-4c84-b03f-28b41c6fc87a" containerName="test-curl-7928-850ac3"
Sep 15 21:25:01.198780 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.198737    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="723f3936-3753-4c84-b03f-28b41c6fc87a" containerName="test-curl-7928-850ac3"
Sep 15 21:25:01.198864 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.198852    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="723f3936-3753-4c84-b03f-28b41c6fc87a" containerName="test-curl-7928-850ac3"
Sep 15 21:25:01.203286 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.203261    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:25:01.208644 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.207989    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2f34ef\"/\"kube-root-ca.crt\""
Sep 15 21:25:01.208644 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.208017    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2f34ef\"/\"openshift-service-ca.crt\""
Sep 15 21:25:01.208644 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.207994    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2f34ef\"/\"default-dockercfg-fxdnv\""
Sep 15 21:25:01.208644 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.208494    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2f34ef\"/\"rate-test-model-2f34ef-kserve-self-signed-certs\""
Sep 15 21:25:01.211374 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.211345    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"]
Sep 15 21:25:01.267328 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.267301    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-tls-certs\") pod \"rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn\" (UID: \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:25:01.267467 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.267347    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29p9q\" (UniqueName: \"kubernetes.io/projected/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-kube-api-access-29p9q\") pod \"rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn\" (UID: \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:25:01.267467 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.267402    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-model-cache\") pod \"rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn\" (UID: \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:25:01.267467 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.267452    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-dshm\") pod \"rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn\" (UID: \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:25:01.267622 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.267560    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-home\") pod \"rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn\" (UID: \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:25:01.267622 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.267598    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-tmp-dir\") pod \"rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn\" (UID: \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:25:01.368884 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.368839    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-dshm\") pod \"rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn\" (UID: \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:25:01.369126 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.368925    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-home\") pod \"rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn\" (UID: \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:25:01.369126 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.368948    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-tmp-dir\") pod \"rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn\" (UID: \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:25:01.369126 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.368981    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-tls-certs\") pod \"rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn\" (UID: \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:25:01.369126 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.369027    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29p9q\" (UniqueName: \"kubernetes.io/projected/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-kube-api-access-29p9q\") pod \"rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn\" (UID: \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:25:01.369126 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.369092    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-model-cache\") pod \"rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn\" (UID: \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:25:01.369425 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.369402    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-home\") pod \"rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn\" (UID: \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:25:01.369468 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.369412    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-tmp-dir\") pod \"rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn\" (UID: \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:25:01.369503 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.369472    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-model-cache\") pod \"rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn\" (UID: \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:25:01.371164 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.371142    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-dshm\") pod \"rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn\" (UID: \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:25:01.371465 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.371448    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-tls-certs\") pod \"rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn\" (UID: \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:25:01.379253 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.379234    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29p9q\" (UniqueName: \"kubernetes.io/projected/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-kube-api-access-29p9q\") pod \"rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn\" (UID: \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\") " pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:25:01.514340 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.514257    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:25:01.786949 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.786926    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"]
Sep 15 21:25:01.788779 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:25:01.788751    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea8469c5_7ea9_4f25_b412_fdd40eb0ba7d.slice/crio-6205c88637905169b59996645548b29c535a4d0ad450f84c975d8f55fc8d3ebc WatchSource:0}: Error finding container 6205c88637905169b59996645548b29c535a4d0ad450f84c975d8f55fc8d3ebc: Status 404 returned error can't find the container with id 6205c88637905169b59996645548b29c535a4d0ad450f84c975d8f55fc8d3ebc
Sep 15 21:25:01.941996 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.941959    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn" event={"ID":"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d","Type":"ContainerStarted","Data":"7b78ce08fdf8d0f623e2f06edd31490083f047931f18d293422a5a93d8e9a624"}
Sep 15 21:25:01.941996 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.942000    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn" event={"ID":"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d","Type":"ContainerStarted","Data":"6205c88637905169b59996645548b29c535a4d0ad450f84c975d8f55fc8d3ebc"}
Sep 15 21:25:01.962826 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:01.962773    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn" podStartSLOduration=0.962758596 podStartE2EDuration="962.758596ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:25:01.959915886 +0000 UTC m=+1620.450561619" watchObservedRunningTime="2026-09-15 21:25:01.962758596 +0000 UTC m=+1620.453404327"
Sep 15 21:25:06.039913 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:06.039876    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:25:11.515113 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:11.515076    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:25:11.515592 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:11.515126    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:25:11.527747 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:11.527716    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:25:12.004299 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:12.004269    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:25:34.723449 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:34.723416    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:25:50.383134 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:25:50.383098    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:26:00.295581 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:00.295546    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:26:06.585942 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:06.585905    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"]
Sep 15 21:26:11.290430 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:11.290395    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:26:21.604546 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:21.604506    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn" podUID="ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d" containerName="main" containerID="cri-o://7b78ce08fdf8d0f623e2f06edd31490083f047931f18d293422a5a93d8e9a624" gracePeriod=45
Sep 15 21:26:21.843814 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:21.843787    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:26:21.878609 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:21.878534    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29p9q\" (UniqueName: \"kubernetes.io/projected/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-kube-api-access-29p9q\") pod \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\" (UID: \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\") "
Sep 15 21:26:21.878754 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:21.878626    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-dshm\") pod \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\" (UID: \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\") "
Sep 15 21:26:21.878754 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:21.878688    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-tls-certs\") pod \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\" (UID: \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\") "
Sep 15 21:26:21.878754 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:21.878739    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-tmp-dir\") pod \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\" (UID: \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\") "
Sep 15 21:26:21.878924 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:21.878892    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-model-cache\") pod \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\" (UID: \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\") "
Sep 15 21:26:21.879018 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:21.878997    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d" (UID: "ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:26:21.879126 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:21.879014    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-home\") pod \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\" (UID: \"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d\") "
Sep 15 21:26:21.879271 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:21.879208    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d" (UID: "ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:26:21.879412 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:21.879344    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-tmp-dir\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:26:21.879412 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:21.879371    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-model-cache\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:26:21.879412 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:21.879386    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-home" (OuterVolumeSpecName: "home") pod "ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d" (UID: "ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:26:21.880928 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:21.880899    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d" (UID: "ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:26:21.881197 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:21.881179    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-dshm" (OuterVolumeSpecName: "dshm") pod "ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d" (UID: "ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:26:21.881252 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:21.881229    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-kube-api-access-29p9q" (OuterVolumeSpecName: "kube-api-access-29p9q") pod "ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d" (UID: "ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d"). InnerVolumeSpecName "kube-api-access-29p9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:26:21.980408 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:21.980377    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-home\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:26:21.980408 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:21.980407    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29p9q\" (UniqueName: \"kubernetes.io/projected/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-kube-api-access-29p9q\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:26:21.980590 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:21.980418    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-dshm\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:26:21.980590 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:21.980427    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d-tls-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:26:22.354445 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:22.354412    2558 generic.go:356] "Generic (PLEG): container finished" podID="ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d" containerID="7b78ce08fdf8d0f623e2f06edd31490083f047931f18d293422a5a93d8e9a624" exitCode=0
Sep 15 21:26:22.354644 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:22.354458    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"
Sep 15 21:26:22.354644 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:22.354469    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn" event={"ID":"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d","Type":"ContainerDied","Data":"7b78ce08fdf8d0f623e2f06edd31490083f047931f18d293422a5a93d8e9a624"}
Sep 15 21:26:22.354644 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:22.354548    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn" event={"ID":"ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d","Type":"ContainerDied","Data":"6205c88637905169b59996645548b29c535a4d0ad450f84c975d8f55fc8d3ebc"}
Sep 15 21:26:22.354644 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:22.354567    2558 scope.go:117] "RemoveContainer" containerID="7b78ce08fdf8d0f623e2f06edd31490083f047931f18d293422a5a93d8e9a624"
Sep 15 21:26:22.363936 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:22.363919    2558 scope.go:117] "RemoveContainer" containerID="7b78ce08fdf8d0f623e2f06edd31490083f047931f18d293422a5a93d8e9a624"
Sep 15 21:26:22.364230 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:26:22.364212    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b78ce08fdf8d0f623e2f06edd31490083f047931f18d293422a5a93d8e9a624\": container with ID starting with 7b78ce08fdf8d0f623e2f06edd31490083f047931f18d293422a5a93d8e9a624 not found: ID does not exist" containerID="7b78ce08fdf8d0f623e2f06edd31490083f047931f18d293422a5a93d8e9a624"
Sep 15 21:26:22.364292 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:22.364243    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b78ce08fdf8d0f623e2f06edd31490083f047931f18d293422a5a93d8e9a624"} err="failed to get container status \"7b78ce08fdf8d0f623e2f06edd31490083f047931f18d293422a5a93d8e9a624\": rpc error: code = NotFound desc = could not find container \"7b78ce08fdf8d0f623e2f06edd31490083f047931f18d293422a5a93d8e9a624\": container with ID starting with 7b78ce08fdf8d0f623e2f06edd31490083f047931f18d293422a5a93d8e9a624 not found: ID does not exist"
Sep 15 21:26:22.373543 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:22.373514    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"]
Sep 15 21:26:22.381528 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:22.381500    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2f34ef/rate-test-model-2f34ef-kserve-5fdbdb46f5-6zfdn"]
Sep 15 21:26:23.943105 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:23.943073    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d" path="/var/lib/kubelet/pods/ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d/volumes"
Sep 15 21:26:25.682025 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:25.681988    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:26:37.380043 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.380006    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"]
Sep 15 21:26:37.380447 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.380355    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d" containerName="main"
Sep 15 21:26:37.380447 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.380366    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d" containerName="main"
Sep 15 21:26:37.380447 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.380423    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea8469c5-7ea9-4f25-b412-fdd40eb0ba7d" containerName="main"
Sep 15 21:26:37.383260 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.383242    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.387273 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.387253    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ffb334-openshift-default-dockercfg-8t8lc\""
Sep 15 21:26:37.484657 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.484626    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"]
Sep 15 21:26:37.516200 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.516164    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b821b12-7463-448a-af49-e3a5ea9c4b95-istio-podinfo\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.516376 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.516209    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsg77\" (UniqueName: \"kubernetes.io/projected/4b821b12-7463-448a-af49-e3a5ea9c4b95-kube-api-access-bsg77\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.516376 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.516294    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b821b12-7463-448a-af49-e3a5ea9c4b95-istio-token\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.516376 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.516318    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b821b12-7463-448a-af49-e3a5ea9c4b95-istiod-ca-cert\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.516376 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.516347    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-istio-data\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.516376 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.516374    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-credential-socket\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.516628 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.516416    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-istio-envoy\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.516628 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.516446    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-workload-socket\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.516628 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.516470    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-workload-certs\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.617963 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.617928    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b821b12-7463-448a-af49-e3a5ea9c4b95-istio-token\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.617963 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.617961    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b821b12-7463-448a-af49-e3a5ea9c4b95-istiod-ca-cert\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.618354 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.617981    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-istio-data\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.618354 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.618000    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-credential-socket\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.618354 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.618019    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-istio-envoy\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.618354 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.618042    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-workload-socket\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.618354 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.618093    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-workload-certs\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.618354 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.618171    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b821b12-7463-448a-af49-e3a5ea9c4b95-istio-podinfo\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.618354 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.618201    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsg77\" (UniqueName: \"kubernetes.io/projected/4b821b12-7463-448a-af49-e3a5ea9c4b95-kube-api-access-bsg77\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.618737 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.618440    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-istio-data\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.618737 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.618544    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-credential-socket\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.618737 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.618616    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-workload-socket\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.618737 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.618721    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-workload-certs\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.618887 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.618824    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b821b12-7463-448a-af49-e3a5ea9c4b95-istiod-ca-cert\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.620418 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.620400    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-istio-envoy\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.620695 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.620678    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b821b12-7463-448a-af49-e3a5ea9c4b95-istio-podinfo\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.627163 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.627140    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b821b12-7463-448a-af49-e3a5ea9c4b95-istio-token\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.627763 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.627697    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsg77\" (UniqueName: \"kubernetes.io/projected/4b821b12-7463-448a-af49-e3a5ea9c4b95-kube-api-access-bsg77\") pod \"e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") " pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.693490 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.693463    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:37.976218 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.976191    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"]
Sep 15 21:26:37.977224 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:26:37.977201    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b821b12_7463_448a_af49_e3a5ea9c4b95.slice/crio-7ab540026e0189ee68e5c511988582628033a7068a9cd2cab48e5df6d33b906f WatchSource:0}: Error finding container 7ab540026e0189ee68e5c511988582628033a7068a9cd2cab48e5df6d33b906f: Status 404 returned error can't find the container with id 7ab540026e0189ee68e5c511988582628033a7068a9cd2cab48e5df6d33b906f
Sep 15 21:26:37.979269 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.979229    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:26:37.979363 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.979319    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:26:37.979363 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:37.979349    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:26:38.438707 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:38.438673    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865" event={"ID":"4b821b12-7463-448a-af49-e3a5ea9c4b95","Type":"ContainerStarted","Data":"d9e96a611b0a64546adf914b362015c674e54733aed43a7a6a66d2f069b9dda2"}
Sep 15 21:26:38.439155 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:38.438728    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865" event={"ID":"4b821b12-7463-448a-af49-e3a5ea9c4b95","Type":"ContainerStarted","Data":"7ab540026e0189ee68e5c511988582628033a7068a9cd2cab48e5df6d33b906f"}
Sep 15 21:26:38.472377 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:38.472330    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865" podStartSLOduration=1.472316814 podStartE2EDuration="1.472316814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:38.469238178 +0000 UTC m=+1716.959883912" watchObservedRunningTime="2026-09-15 21:26:38.472316814 +0000 UTC m=+1716.962962545"
Sep 15 21:26:38.694408 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:38.694333    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:39.698489 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:39.698461    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:40.450121 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:40.450025    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:40.451164 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:40.451142    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:43.781313 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:43.781275    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd"]
Sep 15 21:26:43.784898 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:43.784881    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd"
Sep 15 21:26:43.787620 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:43.787598    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-ffb334-serving-cert\""
Sep 15 21:26:43.796493 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:43.796472    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd"]
Sep 15 21:26:43.876876 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:43.876843    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e02ae5-4e2b-414a-ab5a-a71180e29de9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd\" (UID: \"d4e02ae5-4e2b-414a-ab5a-a71180e29de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd"
Sep 15 21:26:43.877036 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:43.876902    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6kcc\" (UniqueName: \"kubernetes.io/projected/d4e02ae5-4e2b-414a-ab5a-a71180e29de9-kube-api-access-b6kcc\") pod \"maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd\" (UID: \"d4e02ae5-4e2b-414a-ab5a-a71180e29de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd"
Sep 15 21:26:43.877036 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:43.877002    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4e02ae5-4e2b-414a-ab5a-a71180e29de9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd\" (UID: \"d4e02ae5-4e2b-414a-ab5a-a71180e29de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd"
Sep 15 21:26:43.877154 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:43.877102    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4e02ae5-4e2b-414a-ab5a-a71180e29de9-metrics-certs\") pod \"maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd\" (UID: \"d4e02ae5-4e2b-414a-ab5a-a71180e29de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd"
Sep 15 21:26:43.978412 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:43.978379    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e02ae5-4e2b-414a-ab5a-a71180e29de9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd\" (UID: \"d4e02ae5-4e2b-414a-ab5a-a71180e29de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd"
Sep 15 21:26:43.978605 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:43.978433    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6kcc\" (UniqueName: \"kubernetes.io/projected/d4e02ae5-4e2b-414a-ab5a-a71180e29de9-kube-api-access-b6kcc\") pod \"maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd\" (UID: \"d4e02ae5-4e2b-414a-ab5a-a71180e29de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd"
Sep 15 21:26:43.978605 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:43.978455    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4e02ae5-4e2b-414a-ab5a-a71180e29de9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd\" (UID: \"d4e02ae5-4e2b-414a-ab5a-a71180e29de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd"
Sep 15 21:26:43.978605 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:43.978482    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4e02ae5-4e2b-414a-ab5a-a71180e29de9-metrics-certs\") pod \"maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd\" (UID: \"d4e02ae5-4e2b-414a-ab5a-a71180e29de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd"
Sep 15 21:26:43.979031 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:43.979009    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e02ae5-4e2b-414a-ab5a-a71180e29de9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd\" (UID: \"d4e02ae5-4e2b-414a-ab5a-a71180e29de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd"
Sep 15 21:26:43.980824 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:43.980795    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4e02ae5-4e2b-414a-ab5a-a71180e29de9-metrics-certs\") pod \"maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd\" (UID: \"d4e02ae5-4e2b-414a-ab5a-a71180e29de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd"
Sep 15 21:26:43.981001 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:43.980977    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4e02ae5-4e2b-414a-ab5a-a71180e29de9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd\" (UID: \"d4e02ae5-4e2b-414a-ab5a-a71180e29de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd"
Sep 15 21:26:43.986631 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:43.986611    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6kcc\" (UniqueName: \"kubernetes.io/projected/d4e02ae5-4e2b-414a-ab5a-a71180e29de9-kube-api-access-b6kcc\") pod \"maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd\" (UID: \"d4e02ae5-4e2b-414a-ab5a-a71180e29de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd"
Sep 15 21:26:44.096924 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:44.096837    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd"
Sep 15 21:26:44.374705 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:44.374680    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd"]
Sep 15 21:26:44.376580 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:26:44.376548    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e02ae5_4e2b_414a_ab5a_a71180e29de9.slice/crio-f7a690887dad03937ef5b4bb188df89897421a5e5fc6ba9d7e74fb19a88ecf9a WatchSource:0}: Error finding container f7a690887dad03937ef5b4bb188df89897421a5e5fc6ba9d7e74fb19a88ecf9a: Status 404 returned error can't find the container with id f7a690887dad03937ef5b4bb188df89897421a5e5fc6ba9d7e74fb19a88ecf9a
Sep 15 21:26:44.377891 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:44.377872    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:26:44.473773 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:44.473738    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd" event={"ID":"d4e02ae5-4e2b-414a-ab5a-a71180e29de9","Type":"ContainerStarted","Data":"f7a690887dad03937ef5b4bb188df89897421a5e5fc6ba9d7e74fb19a88ecf9a"}
Sep 15 21:26:45.481566 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:45.481519    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd" event={"ID":"d4e02ae5-4e2b-414a-ab5a-a71180e29de9","Type":"ContainerStarted","Data":"720dab171bbc0c24806bfd643ddf1884bfd4aae4fcf95d82af2e640adc0ffbc9"}
Sep 15 21:26:45.482015 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:45.481588    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd"
Sep 15 21:26:45.500647 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:45.500592    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd" podStartSLOduration=1.7240370390000002 podStartE2EDuration="2.500576541s" totalImagesPullingTime="776.539502ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:26:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:45.499113277 +0000 UTC m=+1723.989759010" watchObservedRunningTime="2026-09-15 21:26:45.500576541 +0000 UTC m=+1723.991222272"
Sep 15 21:26:48.313216 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:48.313175    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:26:51.493272 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:51.493241    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd"
Sep 15 21:26:56.493187 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:56.493146    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 15 21:26:56.493566 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:56.493228    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865" podUID="4b821b12-7463-448a-af49-e3a5ea9c4b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 15 21:26:58.552892 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:58.552861    2558 generic.go:356] "Generic (PLEG): container finished" podID="4b821b12-7463-448a-af49-e3a5ea9c4b95" containerID="d9e96a611b0a64546adf914b362015c674e54733aed43a7a6a66d2f069b9dda2" exitCode=0
Sep 15 21:26:58.553316 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:58.552932    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865" event={"ID":"4b821b12-7463-448a-af49-e3a5ea9c4b95","Type":"ContainerDied","Data":"d9e96a611b0a64546adf914b362015c674e54733aed43a7a6a66d2f069b9dda2"}
Sep 15 21:26:58.553360 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:58.553334    2558 scope.go:117] "RemoveContainer" containerID="d9e96a611b0a64546adf914b362015c674e54733aed43a7a6a66d2f069b9dda2"
Sep 15 21:26:58.554455 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:58.554429    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:26:58.554531 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:58.554495    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:26:58.554569 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:58.554536    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:26:59.564835 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:59.564794    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865" event={"ID":"4b821b12-7463-448a-af49-e3a5ea9c4b95","Type":"ContainerStarted","Data":"d4d65b3dc96bec6adfc3ceea8a6733578b11fe1a7f77cddfe798a51bd251a8ae"}
Sep 15 21:26:59.694070 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:59.694024    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:26:59.694339 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:59.694317    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 15 21:26:59.694383 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:26:59.694366    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865" podUID="4b821b12-7463-448a-af49-e3a5ea9c4b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 15 21:27:00.694160 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:00.694121    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 15 21:27:00.694578 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:00.694184    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865" podUID="4b821b12-7463-448a-af49-e3a5ea9c4b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 15 21:27:01.694185 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:01.694144    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 15 21:27:01.694618 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:01.694227    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865" podUID="4b821b12-7463-448a-af49-e3a5ea9c4b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 15 21:27:02.694417 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:02.694380    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 15 21:27:02.694823 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:02.694436    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865" podUID="4b821b12-7463-448a-af49-e3a5ea9c4b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 15 21:27:03.693954 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:03.693911    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 15 21:27:03.694191 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:03.693971    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865" podUID="4b821b12-7463-448a-af49-e3a5ea9c4b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 15 21:27:04.697906 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:04.697875    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:27:04.698378 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:04.698158    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:27:04.698903 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:04.698887    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:27:24.094584 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:24.094553    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:27:39.803952 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:39.803909    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:27:46.964744 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:46.964708    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"]
Sep 15 21:27:46.969986 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:46.969957    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:46.972863 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:46.972838    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-9d9247-openshift-default-dockercfg-sbtwv\""
Sep 15 21:27:46.979898 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:46.979871    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"]
Sep 15 21:27:47.070951 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.070914    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-workload-socket\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.071177 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.071002    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56b137a3-3518-4882-84d8-aba07c7cd6c0-istiod-ca-cert\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.071177 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.071071    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-credential-socket\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.071177 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.071129    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-istio-envoy\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.071177 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.071169    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-workload-certs\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.071378 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.071235    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56b137a3-3518-4882-84d8-aba07c7cd6c0-istio-podinfo\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.071378 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.071262    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-istio-data\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.071378 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.071308    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxpdw\" (UniqueName: \"kubernetes.io/projected/56b137a3-3518-4882-84d8-aba07c7cd6c0-kube-api-access-jxpdw\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.071378 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.071356    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56b137a3-3518-4882-84d8-aba07c7cd6c0-istio-token\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.171773 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.171740    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-workload-socket\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.171773 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.171777    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56b137a3-3518-4882-84d8-aba07c7cd6c0-istiod-ca-cert\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.171984 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.171801    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-credential-socket\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.171984 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.171911    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-istio-envoy\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.171984 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.171954    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-workload-certs\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.172176 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.171999    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56b137a3-3518-4882-84d8-aba07c7cd6c0-istio-podinfo\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.172176 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.172018    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-istio-data\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.172176 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.172086    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxpdw\" (UniqueName: \"kubernetes.io/projected/56b137a3-3518-4882-84d8-aba07c7cd6c0-kube-api-access-jxpdw\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.172176 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.172143    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-credential-socket\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.172365 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.172247    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-workload-socket\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.172365 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.172296    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56b137a3-3518-4882-84d8-aba07c7cd6c0-istio-token\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.172503 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.172476    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-istio-data\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.172561 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.172496    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-workload-certs\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.172628 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.172606    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56b137a3-3518-4882-84d8-aba07c7cd6c0-istiod-ca-cert\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.174328 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.174301    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-istio-envoy\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.174538 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.174514    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56b137a3-3518-4882-84d8-aba07c7cd6c0-istio-podinfo\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.180892 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.180869    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56b137a3-3518-4882-84d8-aba07c7cd6c0-istio-token\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.180984 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.180916    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxpdw\" (UniqueName: \"kubernetes.io/projected/56b137a3-3518-4882-84d8-aba07c7cd6c0-kube-api-access-jxpdw\") pod \"e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") " pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.281375 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.281305    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:47.559909 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:27:47.559881    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b137a3_3518_4882_84d8_aba07c7cd6c0.slice/crio-62b9e0ab7e4e7659ebf89140b71c92d576de83c532704e046025a90a4b32a00e WatchSource:0}: Error finding container 62b9e0ab7e4e7659ebf89140b71c92d576de83c532704e046025a90a4b32a00e: Status 404 returned error can't find the container with id 62b9e0ab7e4e7659ebf89140b71c92d576de83c532704e046025a90a4b32a00e
Sep 15 21:27:47.560208 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.560184    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"]
Sep 15 21:27:47.562789 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.562756    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:27:47.562878 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.562819    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:27:47.562878 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.562847    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:27:47.844173 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.844081    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf" event={"ID":"56b137a3-3518-4882-84d8-aba07c7cd6c0","Type":"ContainerStarted","Data":"ae4f163a825da756a71fb767d791f73bc4652362cfda5dd7361fd01d7206f17d"}
Sep 15 21:27:47.844173 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.844139    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf" event={"ID":"56b137a3-3518-4882-84d8-aba07c7cd6c0","Type":"ContainerStarted","Data":"62b9e0ab7e4e7659ebf89140b71c92d576de83c532704e046025a90a4b32a00e"}
Sep 15 21:27:47.867444 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:47.867390    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf" podStartSLOduration=1.867372702 podStartE2EDuration="1.867372702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:47.865934432 +0000 UTC m=+1786.356580163" watchObservedRunningTime="2026-09-15 21:27:47.867372702 +0000 UTC m=+1786.358018437"
Sep 15 21:27:48.281785 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:48.281739    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:48.283583 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:48.283555    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Sep 15 21:27:48.283709 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:48.283611    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf" podUID="56b137a3-3518-4882-84d8-aba07c7cd6c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Sep 15 21:27:49.285680 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:49.285648    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:49.854980 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:49.854953    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:49.855961 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:49.855937    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:27:59.990985 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:27:59.990897    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:28:10.976774 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:10.976735    2558 generic.go:356] "Generic (PLEG): container finished" podID="56b137a3-3518-4882-84d8-aba07c7cd6c0" containerID="ae4f163a825da756a71fb767d791f73bc4652362cfda5dd7361fd01d7206f17d" exitCode=0
Sep 15 21:28:10.977325 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:10.976791    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf" event={"ID":"56b137a3-3518-4882-84d8-aba07c7cd6c0","Type":"ContainerDied","Data":"ae4f163a825da756a71fb767d791f73bc4652362cfda5dd7361fd01d7206f17d"}
Sep 15 21:28:10.977325 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:10.977287    2558 scope.go:117] "RemoveContainer" containerID="ae4f163a825da756a71fb767d791f73bc4652362cfda5dd7361fd01d7206f17d"
Sep 15 21:28:10.978451 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:10.978420    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:28:10.978515 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:10.978503    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:28:10.978560 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:10.978544    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:28:11.985302 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:11.985270    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf" event={"ID":"56b137a3-3518-4882-84d8-aba07c7cd6c0","Type":"ContainerStarted","Data":"db62419242f8caddfb71e90a8af3c94b47ca8a3a122b77692d4449c8d3c0b53c"}
Sep 15 21:28:12.282265 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:12.282188    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:28:12.282423 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:12.282405    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Sep 15 21:28:12.282471 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:12.282448    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf" podUID="56b137a3-3518-4882-84d8-aba07c7cd6c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Sep 15 21:28:13.281791 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:13.281754    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Sep 15 21:28:13.282216 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:13.281818    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf" podUID="56b137a3-3518-4882-84d8-aba07c7cd6c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Sep 15 21:28:14.282673 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:14.282635    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Sep 15 21:28:14.283096 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:14.282704    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf" podUID="56b137a3-3518-4882-84d8-aba07c7cd6c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Sep 15 21:28:15.281959 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:15.281921    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Sep 15 21:28:15.282307 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:15.282275    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf" podUID="56b137a3-3518-4882-84d8-aba07c7cd6c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Sep 15 21:28:16.281729 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:16.281697    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Sep 15 21:28:16.282095 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:16.281752    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf" podUID="56b137a3-3518-4882-84d8-aba07c7cd6c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Sep 15 21:28:17.349548 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:17.349502    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:28:17.350029 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:17.349913    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:28:17.350881 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:17.350854    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:28:36.191768 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:36.191731    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:28:38.238347 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:38.238315    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"]
Sep 15 21:28:38.238832 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:38.238590    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf" podUID="56b137a3-3518-4882-84d8-aba07c7cd6c0" containerName="istio-proxy" containerID="cri-o://db62419242f8caddfb71e90a8af3c94b47ca8a3a122b77692d4449c8d3c0b53c" gracePeriod=30
Sep 15 21:28:43.488261 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.488236    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:28:43.618872 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.618792    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56b137a3-3518-4882-84d8-aba07c7cd6c0-istio-podinfo\") pod \"56b137a3-3518-4882-84d8-aba07c7cd6c0\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") "
Sep 15 21:28:43.618872 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.618836    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56b137a3-3518-4882-84d8-aba07c7cd6c0-istio-token\") pod \"56b137a3-3518-4882-84d8-aba07c7cd6c0\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") "
Sep 15 21:28:43.619100 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.618886    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-workload-certs\") pod \"56b137a3-3518-4882-84d8-aba07c7cd6c0\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") "
Sep 15 21:28:43.619100 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.618902    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56b137a3-3518-4882-84d8-aba07c7cd6c0-istiod-ca-cert\") pod \"56b137a3-3518-4882-84d8-aba07c7cd6c0\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") "
Sep 15 21:28:43.619100 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.618918    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-credential-socket\") pod \"56b137a3-3518-4882-84d8-aba07c7cd6c0\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") "
Sep 15 21:28:43.619100 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.618988    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-workload-socket\") pod \"56b137a3-3518-4882-84d8-aba07c7cd6c0\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") "
Sep 15 21:28:43.619100 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.619018    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-istio-envoy\") pod \"56b137a3-3518-4882-84d8-aba07c7cd6c0\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") "
Sep 15 21:28:43.619100 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.619080    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-istio-data\") pod \"56b137a3-3518-4882-84d8-aba07c7cd6c0\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") "
Sep 15 21:28:43.619398 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.619111    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxpdw\" (UniqueName: \"kubernetes.io/projected/56b137a3-3518-4882-84d8-aba07c7cd6c0-kube-api-access-jxpdw\") pod \"56b137a3-3518-4882-84d8-aba07c7cd6c0\" (UID: \"56b137a3-3518-4882-84d8-aba07c7cd6c0\") "
Sep 15 21:28:43.619457 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.619412    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "56b137a3-3518-4882-84d8-aba07c7cd6c0" (UID: "56b137a3-3518-4882-84d8-aba07c7cd6c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:28:43.619457 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.619426    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "56b137a3-3518-4882-84d8-aba07c7cd6c0" (UID: "56b137a3-3518-4882-84d8-aba07c7cd6c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:28:43.619581 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.619459    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56b137a3-3518-4882-84d8-aba07c7cd6c0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "56b137a3-3518-4882-84d8-aba07c7cd6c0" (UID: "56b137a3-3518-4882-84d8-aba07c7cd6c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:28:43.619642 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.619574    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "56b137a3-3518-4882-84d8-aba07c7cd6c0" (UID: "56b137a3-3518-4882-84d8-aba07c7cd6c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:28:43.619642 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.619619    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-istio-data" (OuterVolumeSpecName: "istio-data") pod "56b137a3-3518-4882-84d8-aba07c7cd6c0" (UID: "56b137a3-3518-4882-84d8-aba07c7cd6c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:28:43.621150 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.621123    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b137a3-3518-4882-84d8-aba07c7cd6c0-istio-token" (OuterVolumeSpecName: "istio-token") pod "56b137a3-3518-4882-84d8-aba07c7cd6c0" (UID: "56b137a3-3518-4882-84d8-aba07c7cd6c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:28:43.621312 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.621286    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "56b137a3-3518-4882-84d8-aba07c7cd6c0" (UID: "56b137a3-3518-4882-84d8-aba07c7cd6c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:28:43.621382 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.621329    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b137a3-3518-4882-84d8-aba07c7cd6c0-kube-api-access-jxpdw" (OuterVolumeSpecName: "kube-api-access-jxpdw") pod "56b137a3-3518-4882-84d8-aba07c7cd6c0" (UID: "56b137a3-3518-4882-84d8-aba07c7cd6c0"). InnerVolumeSpecName "kube-api-access-jxpdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:28:43.621382 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.621343    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56b137a3-3518-4882-84d8-aba07c7cd6c0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "56b137a3-3518-4882-84d8-aba07c7cd6c0" (UID: "56b137a3-3518-4882-84d8-aba07c7cd6c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:28:43.719992 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.719963    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-workload-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:28:43.719992 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.719988    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-istio-envoy\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:28:43.720181 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.719999    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-istio-data\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:28:43.720181 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.720007    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxpdw\" (UniqueName: \"kubernetes.io/projected/56b137a3-3518-4882-84d8-aba07c7cd6c0-kube-api-access-jxpdw\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:28:43.720181 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.720016    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56b137a3-3518-4882-84d8-aba07c7cd6c0-istio-podinfo\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:28:43.720181 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.720024    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56b137a3-3518-4882-84d8-aba07c7cd6c0-istio-token\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:28:43.720181 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.720032    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-workload-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:28:43.720181 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.720039    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56b137a3-3518-4882-84d8-aba07c7cd6c0-istiod-ca-cert\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:28:43.720181 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:43.720046    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b137a3-3518-4882-84d8-aba07c7cd6c0-credential-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:28:44.159158 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:44.159123    2558 generic.go:356] "Generic (PLEG): container finished" podID="56b137a3-3518-4882-84d8-aba07c7cd6c0" containerID="db62419242f8caddfb71e90a8af3c94b47ca8a3a122b77692d4449c8d3c0b53c" exitCode=0
Sep 15 21:28:44.159312 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:44.159169    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"
Sep 15 21:28:44.159312 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:44.159207    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf" event={"ID":"56b137a3-3518-4882-84d8-aba07c7cd6c0","Type":"ContainerDied","Data":"db62419242f8caddfb71e90a8af3c94b47ca8a3a122b77692d4449c8d3c0b53c"}
Sep 15 21:28:44.159312 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:44.159262    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf" event={"ID":"56b137a3-3518-4882-84d8-aba07c7cd6c0","Type":"ContainerDied","Data":"62b9e0ab7e4e7659ebf89140b71c92d576de83c532704e046025a90a4b32a00e"}
Sep 15 21:28:44.159312 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:44.159285    2558 scope.go:117] "RemoveContainer" containerID="db62419242f8caddfb71e90a8af3c94b47ca8a3a122b77692d4449c8d3c0b53c"
Sep 15 21:28:44.168508 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:44.168480    2558 scope.go:117] "RemoveContainer" containerID="ae4f163a825da756a71fb767d791f73bc4652362cfda5dd7361fd01d7206f17d"
Sep 15 21:28:44.177195 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:44.177176    2558 scope.go:117] "RemoveContainer" containerID="db62419242f8caddfb71e90a8af3c94b47ca8a3a122b77692d4449c8d3c0b53c"
Sep 15 21:28:44.177466 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:28:44.177442    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db62419242f8caddfb71e90a8af3c94b47ca8a3a122b77692d4449c8d3c0b53c\": container with ID starting with db62419242f8caddfb71e90a8af3c94b47ca8a3a122b77692d4449c8d3c0b53c not found: ID does not exist" containerID="db62419242f8caddfb71e90a8af3c94b47ca8a3a122b77692d4449c8d3c0b53c"
Sep 15 21:28:44.177556 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:44.177477    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db62419242f8caddfb71e90a8af3c94b47ca8a3a122b77692d4449c8d3c0b53c"} err="failed to get container status \"db62419242f8caddfb71e90a8af3c94b47ca8a3a122b77692d4449c8d3c0b53c\": rpc error: code = NotFound desc = could not find container \"db62419242f8caddfb71e90a8af3c94b47ca8a3a122b77692d4449c8d3c0b53c\": container with ID starting with db62419242f8caddfb71e90a8af3c94b47ca8a3a122b77692d4449c8d3c0b53c not found: ID does not exist"
Sep 15 21:28:44.177556 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:44.177500    2558 scope.go:117] "RemoveContainer" containerID="ae4f163a825da756a71fb767d791f73bc4652362cfda5dd7361fd01d7206f17d"
Sep 15 21:28:44.177783 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:28:44.177762    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae4f163a825da756a71fb767d791f73bc4652362cfda5dd7361fd01d7206f17d\": container with ID starting with ae4f163a825da756a71fb767d791f73bc4652362cfda5dd7361fd01d7206f17d not found: ID does not exist" containerID="ae4f163a825da756a71fb767d791f73bc4652362cfda5dd7361fd01d7206f17d"
Sep 15 21:28:44.177824 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:44.177793    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae4f163a825da756a71fb767d791f73bc4652362cfda5dd7361fd01d7206f17d"} err="failed to get container status \"ae4f163a825da756a71fb767d791f73bc4652362cfda5dd7361fd01d7206f17d\": rpc error: code = NotFound desc = could not find container \"ae4f163a825da756a71fb767d791f73bc4652362cfda5dd7361fd01d7206f17d\": container with ID starting with ae4f163a825da756a71fb767d791f73bc4652362cfda5dd7361fd01d7206f17d not found: ID does not exist"
Sep 15 21:28:44.177862 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:44.177836    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"]
Sep 15 21:28:44.184542 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:44.184519    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-9d9247-openshift-default-bf4c7b9c4-4x2sf"]
Sep 15 21:28:45.942464 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:45.942432    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56b137a3-3518-4882-84d8-aba07c7cd6c0" path="/var/lib/kubelet/pods/56b137a3-3518-4882-84d8-aba07c7cd6c0/volumes"
Sep 15 21:28:55.389632 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:28:55.389595    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:29:07.594301 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:29:07.594265    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:30:04.569447 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:04.569413    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd"]
Sep 15 21:30:04.569955 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:04.569619    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd" podUID="d4e02ae5-4e2b-414a-ab5a-a71180e29de9" containerName="maas-api" containerID="cri-o://720dab171bbc0c24806bfd643ddf1884bfd4aae4fcf95d82af2e640adc0ffbc9" gracePeriod=30
Sep 15 21:30:04.815802 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:04.815768    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd"
Sep 15 21:30:04.868432 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:04.868353    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e02ae5-4e2b-414a-ab5a-a71180e29de9-openshift-service-ca\") pod \"d4e02ae5-4e2b-414a-ab5a-a71180e29de9\" (UID: \"d4e02ae5-4e2b-414a-ab5a-a71180e29de9\") "
Sep 15 21:30:04.868432 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:04.868388    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4e02ae5-4e2b-414a-ab5a-a71180e29de9-maas-api-tls\") pod \"d4e02ae5-4e2b-414a-ab5a-a71180e29de9\" (UID: \"d4e02ae5-4e2b-414a-ab5a-a71180e29de9\") "
Sep 15 21:30:04.868432 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:04.868428    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6kcc\" (UniqueName: \"kubernetes.io/projected/d4e02ae5-4e2b-414a-ab5a-a71180e29de9-kube-api-access-b6kcc\") pod \"d4e02ae5-4e2b-414a-ab5a-a71180e29de9\" (UID: \"d4e02ae5-4e2b-414a-ab5a-a71180e29de9\") "
Sep 15 21:30:04.868647 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:04.868522    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4e02ae5-4e2b-414a-ab5a-a71180e29de9-metrics-certs\") pod \"d4e02ae5-4e2b-414a-ab5a-a71180e29de9\" (UID: \"d4e02ae5-4e2b-414a-ab5a-a71180e29de9\") "
Sep 15 21:30:04.868901 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:04.868876    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e02ae5-4e2b-414a-ab5a-a71180e29de9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d4e02ae5-4e2b-414a-ab5a-a71180e29de9" (UID: "d4e02ae5-4e2b-414a-ab5a-a71180e29de9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:30:04.870440 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:04.870406    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4e02ae5-4e2b-414a-ab5a-a71180e29de9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d4e02ae5-4e2b-414a-ab5a-a71180e29de9" (UID: "d4e02ae5-4e2b-414a-ab5a-a71180e29de9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:30:04.870552 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:04.870526    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e02ae5-4e2b-414a-ab5a-a71180e29de9-kube-api-access-b6kcc" (OuterVolumeSpecName: "kube-api-access-b6kcc") pod "d4e02ae5-4e2b-414a-ab5a-a71180e29de9" (UID: "d4e02ae5-4e2b-414a-ab5a-a71180e29de9"). InnerVolumeSpecName "kube-api-access-b6kcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:30:04.870605 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:04.870595    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4e02ae5-4e2b-414a-ab5a-a71180e29de9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d4e02ae5-4e2b-414a-ab5a-a71180e29de9" (UID: "d4e02ae5-4e2b-414a-ab5a-a71180e29de9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:30:04.969586 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:04.969553    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e02ae5-4e2b-414a-ab5a-a71180e29de9-openshift-service-ca\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:04.969586 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:04.969584    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4e02ae5-4e2b-414a-ab5a-a71180e29de9-maas-api-tls\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:04.969791 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:04.969599    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6kcc\" (UniqueName: \"kubernetes.io/projected/d4e02ae5-4e2b-414a-ab5a-a71180e29de9-kube-api-access-b6kcc\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:04.969791 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:04.969610    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4e02ae5-4e2b-414a-ab5a-a71180e29de9-metrics-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:05.595037 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:05.595004    2558 generic.go:356] "Generic (PLEG): container finished" podID="d4e02ae5-4e2b-414a-ab5a-a71180e29de9" containerID="720dab171bbc0c24806bfd643ddf1884bfd4aae4fcf95d82af2e640adc0ffbc9" exitCode=0
Sep 15 21:30:05.595531 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:05.595045    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd"
Sep 15 21:30:05.595531 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:05.595093    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd" event={"ID":"d4e02ae5-4e2b-414a-ab5a-a71180e29de9","Type":"ContainerDied","Data":"720dab171bbc0c24806bfd643ddf1884bfd4aae4fcf95d82af2e640adc0ffbc9"}
Sep 15 21:30:05.595531 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:05.595161    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd" event={"ID":"d4e02ae5-4e2b-414a-ab5a-a71180e29de9","Type":"ContainerDied","Data":"f7a690887dad03937ef5b4bb188df89897421a5e5fc6ba9d7e74fb19a88ecf9a"}
Sep 15 21:30:05.595531 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:05.595198    2558 scope.go:117] "RemoveContainer" containerID="720dab171bbc0c24806bfd643ddf1884bfd4aae4fcf95d82af2e640adc0ffbc9"
Sep 15 21:30:05.606636 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:05.606618    2558 scope.go:117] "RemoveContainer" containerID="720dab171bbc0c24806bfd643ddf1884bfd4aae4fcf95d82af2e640adc0ffbc9"
Sep 15 21:30:05.606998 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:30:05.606977    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"720dab171bbc0c24806bfd643ddf1884bfd4aae4fcf95d82af2e640adc0ffbc9\": container with ID starting with 720dab171bbc0c24806bfd643ddf1884bfd4aae4fcf95d82af2e640adc0ffbc9 not found: ID does not exist" containerID="720dab171bbc0c24806bfd643ddf1884bfd4aae4fcf95d82af2e640adc0ffbc9"
Sep 15 21:30:05.607120 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:05.607008    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720dab171bbc0c24806bfd643ddf1884bfd4aae4fcf95d82af2e640adc0ffbc9"} err="failed to get container status \"720dab171bbc0c24806bfd643ddf1884bfd4aae4fcf95d82af2e640adc0ffbc9\": rpc error: code = NotFound desc = could not find container \"720dab171bbc0c24806bfd643ddf1884bfd4aae4fcf95d82af2e640adc0ffbc9\": container with ID starting with 720dab171bbc0c24806bfd643ddf1884bfd4aae4fcf95d82af2e640adc0ffbc9 not found: ID does not exist"
Sep 15 21:30:05.625954 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:05.625923    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd"]
Sep 15 21:30:05.630780 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:05.630754    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ffb334-7ffff47585-dtswd"]
Sep 15 21:30:05.942712 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:05.942677    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4e02ae5-4e2b-414a-ab5a-a71180e29de9" path="/var/lib/kubelet/pods/d4e02ae5-4e2b-414a-ab5a-a71180e29de9/volumes"
Sep 15 21:30:08.106881 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:08.106845    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:30:11.537859 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:11.537825    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"]
Sep 15 21:30:11.538414 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:11.538074    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865" podUID="4b821b12-7463-448a-af49-e3a5ea9c4b95" containerName="istio-proxy" containerID="cri-o://d4d65b3dc96bec6adfc3ceea8a6733578b11fe1a7f77cddfe798a51bd251a8ae" gracePeriod=30
Sep 15 21:30:16.657524 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.657489    2558 generic.go:356] "Generic (PLEG): container finished" podID="4b821b12-7463-448a-af49-e3a5ea9c4b95" containerID="d4d65b3dc96bec6adfc3ceea8a6733578b11fe1a7f77cddfe798a51bd251a8ae" exitCode=0
Sep 15 21:30:16.657944 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.657561    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865" event={"ID":"4b821b12-7463-448a-af49-e3a5ea9c4b95","Type":"ContainerDied","Data":"d4d65b3dc96bec6adfc3ceea8a6733578b11fe1a7f77cddfe798a51bd251a8ae"}
Sep 15 21:30:16.657944 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.657629    2558 scope.go:117] "RemoveContainer" containerID="d9e96a611b0a64546adf914b362015c674e54733aed43a7a6a66d2f069b9dda2"
Sep 15 21:30:16.785772 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.785746    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:30:16.885223 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.885129    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-workload-socket\") pod \"4b821b12-7463-448a-af49-e3a5ea9c4b95\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") "
Sep 15 21:30:16.885223 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.885176    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b821b12-7463-448a-af49-e3a5ea9c4b95-istiod-ca-cert\") pod \"4b821b12-7463-448a-af49-e3a5ea9c4b95\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") "
Sep 15 21:30:16.885223 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.885199    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-istio-envoy\") pod \"4b821b12-7463-448a-af49-e3a5ea9c4b95\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") "
Sep 15 21:30:16.885223 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.885220    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b821b12-7463-448a-af49-e3a5ea9c4b95-istio-podinfo\") pod \"4b821b12-7463-448a-af49-e3a5ea9c4b95\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") "
Sep 15 21:30:16.885572 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.885245    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-credential-socket\") pod \"4b821b12-7463-448a-af49-e3a5ea9c4b95\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") "
Sep 15 21:30:16.885572 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.885274    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-istio-data\") pod \"4b821b12-7463-448a-af49-e3a5ea9c4b95\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") "
Sep 15 21:30:16.885572 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.885292    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsg77\" (UniqueName: \"kubernetes.io/projected/4b821b12-7463-448a-af49-e3a5ea9c4b95-kube-api-access-bsg77\") pod \"4b821b12-7463-448a-af49-e3a5ea9c4b95\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") "
Sep 15 21:30:16.885572 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.885318    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b821b12-7463-448a-af49-e3a5ea9c4b95-istio-token\") pod \"4b821b12-7463-448a-af49-e3a5ea9c4b95\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") "
Sep 15 21:30:16.885572 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.885368    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-workload-certs\") pod \"4b821b12-7463-448a-af49-e3a5ea9c4b95\" (UID: \"4b821b12-7463-448a-af49-e3a5ea9c4b95\") "
Sep 15 21:30:16.885572 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.885457    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4b821b12-7463-448a-af49-e3a5ea9c4b95" (UID: "4b821b12-7463-448a-af49-e3a5ea9c4b95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:30:16.885878 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.885601    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b821b12-7463-448a-af49-e3a5ea9c4b95-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4b821b12-7463-448a-af49-e3a5ea9c4b95" (UID: "4b821b12-7463-448a-af49-e3a5ea9c4b95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:30:16.885878 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.885724    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4b821b12-7463-448a-af49-e3a5ea9c4b95" (UID: "4b821b12-7463-448a-af49-e3a5ea9c4b95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:30:16.885878 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.885742    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-istio-data" (OuterVolumeSpecName: "istio-data") pod "4b821b12-7463-448a-af49-e3a5ea9c4b95" (UID: "4b821b12-7463-448a-af49-e3a5ea9c4b95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:30:16.885878 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.885845    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4b821b12-7463-448a-af49-e3a5ea9c4b95" (UID: "4b821b12-7463-448a-af49-e3a5ea9c4b95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:30:16.885878 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.885859    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b821b12-7463-448a-af49-e3a5ea9c4b95-istiod-ca-cert\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:16.885878 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.885877    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-istio-data\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:16.886247 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.885890    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-workload-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:16.886247 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.885903    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-workload-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:16.887725 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.887688    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b821b12-7463-448a-af49-e3a5ea9c4b95-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4b821b12-7463-448a-af49-e3a5ea9c4b95" (UID: "4b821b12-7463-448a-af49-e3a5ea9c4b95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:30:16.887725 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.887698    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b821b12-7463-448a-af49-e3a5ea9c4b95-kube-api-access-bsg77" (OuterVolumeSpecName: "kube-api-access-bsg77") pod "4b821b12-7463-448a-af49-e3a5ea9c4b95" (UID: "4b821b12-7463-448a-af49-e3a5ea9c4b95"). InnerVolumeSpecName "kube-api-access-bsg77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:30:16.887863 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.887717    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b821b12-7463-448a-af49-e3a5ea9c4b95-istio-token" (OuterVolumeSpecName: "istio-token") pod "4b821b12-7463-448a-af49-e3a5ea9c4b95" (UID: "4b821b12-7463-448a-af49-e3a5ea9c4b95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:30:16.887863 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.887723    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4b821b12-7463-448a-af49-e3a5ea9c4b95" (UID: "4b821b12-7463-448a-af49-e3a5ea9c4b95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:30:16.986776 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.986739    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-istio-envoy\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:16.986776 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.986771    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b821b12-7463-448a-af49-e3a5ea9c4b95-istio-podinfo\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:16.986776 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.986779    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b821b12-7463-448a-af49-e3a5ea9c4b95-credential-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:16.987024 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.986789    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsg77\" (UniqueName: \"kubernetes.io/projected/4b821b12-7463-448a-af49-e3a5ea9c4b95-kube-api-access-bsg77\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:16.987024 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:16.986800    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b821b12-7463-448a-af49-e3a5ea9c4b95-istio-token\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:17.664249 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:17.664221    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"
Sep 15 21:30:17.664680 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:17.664221    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865" event={"ID":"4b821b12-7463-448a-af49-e3a5ea9c4b95","Type":"ContainerDied","Data":"7ab540026e0189ee68e5c511988582628033a7068a9cd2cab48e5df6d33b906f"}
Sep 15 21:30:17.664680 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:17.664352    2558 scope.go:117] "RemoveContainer" containerID="d4d65b3dc96bec6adfc3ceea8a6733578b11fe1a7f77cddfe798a51bd251a8ae"
Sep 15 21:30:17.688767 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:17.688743    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"]
Sep 15 21:30:17.694242 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:17.694219    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ffb334-openshift-default-64b5b89d8-7b865"]
Sep 15 21:30:17.942485 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:17.942398    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b821b12-7463-448a-af49-e3a5ea9c4b95" path="/var/lib/kubelet/pods/4b821b12-7463-448a-af49-e3a5ea9c4b95/volumes"
Sep 15 21:30:22.501459 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:22.501424    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"]
Sep 15 21:30:22.502031 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:22.501690    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg" podUID="ff08845a-385f-42be-8231-eb324d9faeb1" containerName="maas-api" containerID="cri-o://053f5333bc0133f5200c4c022d2181a019bfe9a033031d1d73760063a2130cbd" gracePeriod=30
Sep 15 21:30:22.694844 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:22.694813    2558 generic.go:356] "Generic (PLEG): container finished" podID="ff08845a-385f-42be-8231-eb324d9faeb1" containerID="053f5333bc0133f5200c4c022d2181a019bfe9a033031d1d73760063a2130cbd" exitCode=0
Sep 15 21:30:22.695028 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:22.694883    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg" event={"ID":"ff08845a-385f-42be-8231-eb324d9faeb1","Type":"ContainerDied","Data":"053f5333bc0133f5200c4c022d2181a019bfe9a033031d1d73760063a2130cbd"}
Sep 15 21:30:22.746920 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:22.746892    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"
Sep 15 21:30:22.938646 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:22.938611    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff08845a-385f-42be-8231-eb324d9faeb1-metrics-certs\") pod \"ff08845a-385f-42be-8231-eb324d9faeb1\" (UID: \"ff08845a-385f-42be-8231-eb324d9faeb1\") "
Sep 15 21:30:22.938849 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:22.938674    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h4wz\" (UniqueName: \"kubernetes.io/projected/ff08845a-385f-42be-8231-eb324d9faeb1-kube-api-access-9h4wz\") pod \"ff08845a-385f-42be-8231-eb324d9faeb1\" (UID: \"ff08845a-385f-42be-8231-eb324d9faeb1\") "
Sep 15 21:30:22.938849 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:22.938692    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff08845a-385f-42be-8231-eb324d9faeb1-maas-api-tls\") pod \"ff08845a-385f-42be-8231-eb324d9faeb1\" (UID: \"ff08845a-385f-42be-8231-eb324d9faeb1\") "
Sep 15 21:30:22.938849 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:22.938715    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff08845a-385f-42be-8231-eb324d9faeb1-openshift-service-ca\") pod \"ff08845a-385f-42be-8231-eb324d9faeb1\" (UID: \"ff08845a-385f-42be-8231-eb324d9faeb1\") "
Sep 15 21:30:22.939159 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:22.939132    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff08845a-385f-42be-8231-eb324d9faeb1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ff08845a-385f-42be-8231-eb324d9faeb1" (UID: "ff08845a-385f-42be-8231-eb324d9faeb1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:30:22.940829 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:22.940805    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff08845a-385f-42be-8231-eb324d9faeb1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ff08845a-385f-42be-8231-eb324d9faeb1" (UID: "ff08845a-385f-42be-8231-eb324d9faeb1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:30:22.940933 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:22.940845    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff08845a-385f-42be-8231-eb324d9faeb1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ff08845a-385f-42be-8231-eb324d9faeb1" (UID: "ff08845a-385f-42be-8231-eb324d9faeb1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:30:22.940933 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:22.940860    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff08845a-385f-42be-8231-eb324d9faeb1-kube-api-access-9h4wz" (OuterVolumeSpecName: "kube-api-access-9h4wz") pod "ff08845a-385f-42be-8231-eb324d9faeb1" (UID: "ff08845a-385f-42be-8231-eb324d9faeb1"). InnerVolumeSpecName "kube-api-access-9h4wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:30:23.039541 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:23.039501    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h4wz\" (UniqueName: \"kubernetes.io/projected/ff08845a-385f-42be-8231-eb324d9faeb1-kube-api-access-9h4wz\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:23.039541 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:23.039540    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff08845a-385f-42be-8231-eb324d9faeb1-maas-api-tls\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:23.039541 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:23.039553    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff08845a-385f-42be-8231-eb324d9faeb1-openshift-service-ca\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:23.039780 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:23.039561    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff08845a-385f-42be-8231-eb324d9faeb1-metrics-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:23.701935 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:23.701903    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg" event={"ID":"ff08845a-385f-42be-8231-eb324d9faeb1","Type":"ContainerDied","Data":"f37efd9559289631ee00cba9926dcdaa3984f5466943e76af1535c67c0132ca2"}
Sep 15 21:30:23.701935 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:23.701937    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"
Sep 15 21:30:23.702467 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:23.701966    2558 scope.go:117] "RemoveContainer" containerID="053f5333bc0133f5200c4c022d2181a019bfe9a033031d1d73760063a2130cbd"
Sep 15 21:30:23.726246 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:23.726213    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"]
Sep 15 21:30:23.735293 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:23.735262    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2f34ef-846bf97f9c-gdrwg"]
Sep 15 21:30:23.942892 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:23.942863    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff08845a-385f-42be-8231-eb324d9faeb1" path="/var/lib/kubelet/pods/ff08845a-385f-42be-8231-eb324d9faeb1/volumes"
Sep 15 21:30:25.581729 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:25.581691    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:30:29.225836 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:29.225794    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"]
Sep 15 21:30:29.226250 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:29.226067    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh" podUID="a30f4dc5-6a5e-4474-be39-088181771d7c" containerName="istio-proxy" containerID="cri-o://e45104aaddd06912411a3a272a325e56f0857da716a512019455a59e0df3580c" gracePeriod=30
Sep 15 21:30:34.474390 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.474362    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:30:34.538539 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.538462    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-istio-data\") pod \"a30f4dc5-6a5e-4474-be39-088181771d7c\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") "
Sep 15 21:30:34.538539 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.538496    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-istio-envoy\") pod \"a30f4dc5-6a5e-4474-be39-088181771d7c\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") "
Sep 15 21:30:34.538539 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.538538    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-workload-certs\") pod \"a30f4dc5-6a5e-4474-be39-088181771d7c\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") "
Sep 15 21:30:34.538768 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.538574    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9qrq\" (UniqueName: \"kubernetes.io/projected/a30f4dc5-6a5e-4474-be39-088181771d7c-kube-api-access-q9qrq\") pod \"a30f4dc5-6a5e-4474-be39-088181771d7c\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") "
Sep 15 21:30:34.538768 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.538599    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-credential-socket\") pod \"a30f4dc5-6a5e-4474-be39-088181771d7c\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") "
Sep 15 21:30:34.538768 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.538624    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30f4dc5-6a5e-4474-be39-088181771d7c-istio-podinfo\") pod \"a30f4dc5-6a5e-4474-be39-088181771d7c\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") "
Sep 15 21:30:34.538768 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.538674    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-workload-socket\") pod \"a30f4dc5-6a5e-4474-be39-088181771d7c\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") "
Sep 15 21:30:34.538768 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.538737    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a30f4dc5-6a5e-4474-be39-088181771d7c-istio-token\") pod \"a30f4dc5-6a5e-4474-be39-088181771d7c\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") "
Sep 15 21:30:34.539035 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.538772    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30f4dc5-6a5e-4474-be39-088181771d7c-istiod-ca-cert\") pod \"a30f4dc5-6a5e-4474-be39-088181771d7c\" (UID: \"a30f4dc5-6a5e-4474-be39-088181771d7c\") "
Sep 15 21:30:34.539035 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.538774    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-istio-data" (OuterVolumeSpecName: "istio-data") pod "a30f4dc5-6a5e-4474-be39-088181771d7c" (UID: "a30f4dc5-6a5e-4474-be39-088181771d7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:30:34.539035 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.538880    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a30f4dc5-6a5e-4474-be39-088181771d7c" (UID: "a30f4dc5-6a5e-4474-be39-088181771d7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:30:34.539035 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.538890    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a30f4dc5-6a5e-4474-be39-088181771d7c" (UID: "a30f4dc5-6a5e-4474-be39-088181771d7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:30:34.539276 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.539065    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a30f4dc5-6a5e-4474-be39-088181771d7c" (UID: "a30f4dc5-6a5e-4474-be39-088181771d7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:30:34.539276 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.539185    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-istio-data\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:34.539276 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.539227    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-workload-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:34.539276 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.539232    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a30f4dc5-6a5e-4474-be39-088181771d7c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a30f4dc5-6a5e-4474-be39-088181771d7c" (UID: "a30f4dc5-6a5e-4474-be39-088181771d7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:30:34.539276 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.539243    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-credential-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:34.539276 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.539258    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-workload-socket\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:34.540802 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.540765    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a30f4dc5-6a5e-4474-be39-088181771d7c-kube-api-access-q9qrq" (OuterVolumeSpecName: "kube-api-access-q9qrq") pod "a30f4dc5-6a5e-4474-be39-088181771d7c" (UID: "a30f4dc5-6a5e-4474-be39-088181771d7c"). InnerVolumeSpecName "kube-api-access-q9qrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:30:34.540802 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.540784    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a30f4dc5-6a5e-4474-be39-088181771d7c" (UID: "a30f4dc5-6a5e-4474-be39-088181771d7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:30:34.541247 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.541224    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a30f4dc5-6a5e-4474-be39-088181771d7c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a30f4dc5-6a5e-4474-be39-088181771d7c" (UID: "a30f4dc5-6a5e-4474-be39-088181771d7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:30:34.541295 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.541232    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a30f4dc5-6a5e-4474-be39-088181771d7c-istio-token" (OuterVolumeSpecName: "istio-token") pod "a30f4dc5-6a5e-4474-be39-088181771d7c" (UID: "a30f4dc5-6a5e-4474-be39-088181771d7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:30:34.639842 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.639798    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9qrq\" (UniqueName: \"kubernetes.io/projected/a30f4dc5-6a5e-4474-be39-088181771d7c-kube-api-access-q9qrq\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:34.639842 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.639837    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30f4dc5-6a5e-4474-be39-088181771d7c-istio-podinfo\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:34.639842 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.639847    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a30f4dc5-6a5e-4474-be39-088181771d7c-istio-token\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:34.639842 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.639857    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30f4dc5-6a5e-4474-be39-088181771d7c-istiod-ca-cert\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:34.640122 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.639865    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30f4dc5-6a5e-4474-be39-088181771d7c-istio-envoy\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:34.763233 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.763199    2558 generic.go:356] "Generic (PLEG): container finished" podID="a30f4dc5-6a5e-4474-be39-088181771d7c" containerID="e45104aaddd06912411a3a272a325e56f0857da716a512019455a59e0df3580c" exitCode=0
Sep 15 21:30:34.763414 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.763245    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"
Sep 15 21:30:34.763414 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.763282    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh" event={"ID":"a30f4dc5-6a5e-4474-be39-088181771d7c","Type":"ContainerDied","Data":"e45104aaddd06912411a3a272a325e56f0857da716a512019455a59e0df3580c"}
Sep 15 21:30:34.763414 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.763337    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh" event={"ID":"a30f4dc5-6a5e-4474-be39-088181771d7c","Type":"ContainerDied","Data":"39991ed6ef435582880b0f730286c9088702838089320d93a61fc864a10e8eaf"}
Sep 15 21:30:34.763414 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.763360    2558 scope.go:117] "RemoveContainer" containerID="e45104aaddd06912411a3a272a325e56f0857da716a512019455a59e0df3580c"
Sep 15 21:30:34.773134 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.773115    2558 scope.go:117] "RemoveContainer" containerID="0a40e8340d3e4f187668145c31919714194172ff8fac9df231af79ace4e73258"
Sep 15 21:30:34.781273 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.781255    2558 scope.go:117] "RemoveContainer" containerID="e45104aaddd06912411a3a272a325e56f0857da716a512019455a59e0df3580c"
Sep 15 21:30:34.781546 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:30:34.781528    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e45104aaddd06912411a3a272a325e56f0857da716a512019455a59e0df3580c\": container with ID starting with e45104aaddd06912411a3a272a325e56f0857da716a512019455a59e0df3580c not found: ID does not exist" containerID="e45104aaddd06912411a3a272a325e56f0857da716a512019455a59e0df3580c"
Sep 15 21:30:34.781602 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.781557    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e45104aaddd06912411a3a272a325e56f0857da716a512019455a59e0df3580c"} err="failed to get container status \"e45104aaddd06912411a3a272a325e56f0857da716a512019455a59e0df3580c\": rpc error: code = NotFound desc = could not find container \"e45104aaddd06912411a3a272a325e56f0857da716a512019455a59e0df3580c\": container with ID starting with e45104aaddd06912411a3a272a325e56f0857da716a512019455a59e0df3580c not found: ID does not exist"
Sep 15 21:30:34.781602 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.781575    2558 scope.go:117] "RemoveContainer" containerID="0a40e8340d3e4f187668145c31919714194172ff8fac9df231af79ace4e73258"
Sep 15 21:30:34.781801 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:30:34.781783    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a40e8340d3e4f187668145c31919714194172ff8fac9df231af79ace4e73258\": container with ID starting with 0a40e8340d3e4f187668145c31919714194172ff8fac9df231af79ace4e73258 not found: ID does not exist" containerID="0a40e8340d3e4f187668145c31919714194172ff8fac9df231af79ace4e73258"
Sep 15 21:30:34.781861 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.781811    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a40e8340d3e4f187668145c31919714194172ff8fac9df231af79ace4e73258"} err="failed to get container status \"0a40e8340d3e4f187668145c31919714194172ff8fac9df231af79ace4e73258\": rpc error: code = NotFound desc = could not find container \"0a40e8340d3e4f187668145c31919714194172ff8fac9df231af79ace4e73258\": container with ID starting with 0a40e8340d3e4f187668145c31919714194172ff8fac9df231af79ace4e73258 not found: ID does not exist"
Sep 15 21:30:34.802389 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.802325    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"]
Sep 15 21:30:34.820796 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:34.820765    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2f34ef-openshift-default-5957b9d6b8-zhwmh"]
Sep 15 21:30:35.941786 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:35.941748    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a30f4dc5-6a5e-4474-be39-088181771d7c" path="/var/lib/kubelet/pods/a30f4dc5-6a5e-4474-be39-088181771d7c/volumes"
Sep 15 21:30:40.126225 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:40.126192    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9"]
Sep 15 21:30:40.126633 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:40.126410    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9" podUID="e23eece0-f61d-4cae-89ba-e0487a370327" containerName="maas-api" containerID="cri-o://7a4d8643e16b1d080457001d290239f1e399d377a9a2d4221c695d587db2ce0d" gracePeriod=30
Sep 15 21:30:40.376261 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:40.376204    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9"
Sep 15 21:30:40.492502 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:40.492469    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23eece0-f61d-4cae-89ba-e0487a370327-openshift-service-ca\") pod \"e23eece0-f61d-4cae-89ba-e0487a370327\" (UID: \"e23eece0-f61d-4cae-89ba-e0487a370327\") "
Sep 15 21:30:40.492502 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:40.492506    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flj4x\" (UniqueName: \"kubernetes.io/projected/e23eece0-f61d-4cae-89ba-e0487a370327-kube-api-access-flj4x\") pod \"e23eece0-f61d-4cae-89ba-e0487a370327\" (UID: \"e23eece0-f61d-4cae-89ba-e0487a370327\") "
Sep 15 21:30:40.492692 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:40.492527    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e23eece0-f61d-4cae-89ba-e0487a370327-metrics-certs\") pod \"e23eece0-f61d-4cae-89ba-e0487a370327\" (UID: \"e23eece0-f61d-4cae-89ba-e0487a370327\") "
Sep 15 21:30:40.492692 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:40.492589    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e23eece0-f61d-4cae-89ba-e0487a370327-maas-api-tls\") pod \"e23eece0-f61d-4cae-89ba-e0487a370327\" (UID: \"e23eece0-f61d-4cae-89ba-e0487a370327\") "
Sep 15 21:30:40.492944 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:40.492912    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23eece0-f61d-4cae-89ba-e0487a370327-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e23eece0-f61d-4cae-89ba-e0487a370327" (UID: "e23eece0-f61d-4cae-89ba-e0487a370327"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:30:40.494767 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:40.494737    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e23eece0-f61d-4cae-89ba-e0487a370327-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e23eece0-f61d-4cae-89ba-e0487a370327" (UID: "e23eece0-f61d-4cae-89ba-e0487a370327"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:30:40.494767 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:40.494753    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e23eece0-f61d-4cae-89ba-e0487a370327-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e23eece0-f61d-4cae-89ba-e0487a370327" (UID: "e23eece0-f61d-4cae-89ba-e0487a370327"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:30:40.494929 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:40.494909    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23eece0-f61d-4cae-89ba-e0487a370327-kube-api-access-flj4x" (OuterVolumeSpecName: "kube-api-access-flj4x") pod "e23eece0-f61d-4cae-89ba-e0487a370327" (UID: "e23eece0-f61d-4cae-89ba-e0487a370327"). InnerVolumeSpecName "kube-api-access-flj4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:30:40.594140 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:40.594102    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23eece0-f61d-4cae-89ba-e0487a370327-openshift-service-ca\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:40.594140 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:40.594132    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flj4x\" (UniqueName: \"kubernetes.io/projected/e23eece0-f61d-4cae-89ba-e0487a370327-kube-api-access-flj4x\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:40.594140 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:40.594142    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e23eece0-f61d-4cae-89ba-e0487a370327-metrics-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:40.594140 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:40.594151    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e23eece0-f61d-4cae-89ba-e0487a370327-maas-api-tls\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:30:40.798328 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:40.798292    2558 generic.go:356] "Generic (PLEG): container finished" podID="e23eece0-f61d-4cae-89ba-e0487a370327" containerID="7a4d8643e16b1d080457001d290239f1e399d377a9a2d4221c695d587db2ce0d" exitCode=0
Sep 15 21:30:40.798328 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:40.798331    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9"
Sep 15 21:30:40.798547 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:40.798370    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9" event={"ID":"e23eece0-f61d-4cae-89ba-e0487a370327","Type":"ContainerDied","Data":"7a4d8643e16b1d080457001d290239f1e399d377a9a2d4221c695d587db2ce0d"}
Sep 15 21:30:40.798547 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:40.798425    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9" event={"ID":"e23eece0-f61d-4cae-89ba-e0487a370327","Type":"ContainerDied","Data":"9d8783351a9a108112e43b84b541673e1b2585256870f6b4dbd4060300b4c00f"}
Sep 15 21:30:40.798547 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:40.798448    2558 scope.go:117] "RemoveContainer" containerID="7a4d8643e16b1d080457001d290239f1e399d377a9a2d4221c695d587db2ce0d"
Sep 15 21:30:40.808042 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:40.808025    2558 scope.go:117] "RemoveContainer" containerID="7a4d8643e16b1d080457001d290239f1e399d377a9a2d4221c695d587db2ce0d"
Sep 15 21:30:40.808493 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:30:40.808472    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a4d8643e16b1d080457001d290239f1e399d377a9a2d4221c695d587db2ce0d\": container with ID starting with 7a4d8643e16b1d080457001d290239f1e399d377a9a2d4221c695d587db2ce0d not found: ID does not exist" containerID="7a4d8643e16b1d080457001d290239f1e399d377a9a2d4221c695d587db2ce0d"
Sep 15 21:30:40.808555 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:40.808503    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a4d8643e16b1d080457001d290239f1e399d377a9a2d4221c695d587db2ce0d"} err="failed to get container status \"7a4d8643e16b1d080457001d290239f1e399d377a9a2d4221c695d587db2ce0d\": rpc error: code = NotFound desc = could not find container \"7a4d8643e16b1d080457001d290239f1e399d377a9a2d4221c695d587db2ce0d\": container with ID starting with 7a4d8643e16b1d080457001d290239f1e399d377a9a2d4221c695d587db2ce0d not found: ID does not exist"
Sep 15 21:30:40.830804 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:40.830774    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9"]
Sep 15 21:30:40.835968 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:40.835945    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-41ffdf-5f7f6d6c8f-2fqs9"]
Sep 15 21:30:41.942203 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:41.942170    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e23eece0-f61d-4cae-89ba-e0487a370327" path="/var/lib/kubelet/pods/e23eece0-f61d-4cae-89ba-e0487a370327/volumes"
Sep 15 21:30:43.090465 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:43.090426    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:30:52.294271 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:30:52.294235    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:32:15.891670 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:32:15.891629    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:32:43.486476 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:32:43.486439    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:32:51.093285 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:32:51.093247    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:33:00.988481 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:33:00.988442    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:33:19.397338 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:33:19.397305    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:33:25.683904 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:33:25.683864    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:34:50.985499 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:34:50.985463    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:34:58.935745 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:34:58.935710    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:35:19.534444 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:35:19.534401    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:35:32.452465 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:35:32.452418    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:35:54.491619 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:35:54.491579    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:36:03.396373 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:03.396338    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:36:12.893119 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:12.893086    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:36:13.184750 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.184717    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9"]
Sep 15 21:36:13.185073 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.185060    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56b137a3-3518-4882-84d8-aba07c7cd6c0" containerName="istio-proxy"
Sep 15 21:36:13.185073 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.185074    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="56b137a3-3518-4882-84d8-aba07c7cd6c0" containerName="istio-proxy"
Sep 15 21:36:13.185166 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.185087    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56b137a3-3518-4882-84d8-aba07c7cd6c0" containerName="istio-proxy"
Sep 15 21:36:13.185166 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.185093    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="56b137a3-3518-4882-84d8-aba07c7cd6c0" containerName="istio-proxy"
Sep 15 21:36:13.185166 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.185102    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff08845a-385f-42be-8231-eb324d9faeb1" containerName="maas-api"
Sep 15 21:36:13.185166 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.185108    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff08845a-385f-42be-8231-eb324d9faeb1" containerName="maas-api"
Sep 15 21:36:13.185166 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.185119    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b821b12-7463-448a-af49-e3a5ea9c4b95" containerName="istio-proxy"
Sep 15 21:36:13.185166 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.185124    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b821b12-7463-448a-af49-e3a5ea9c4b95" containerName="istio-proxy"
Sep 15 21:36:13.185166 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.185131    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a30f4dc5-6a5e-4474-be39-088181771d7c" containerName="istio-proxy"
Sep 15 21:36:13.185166 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.185136    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a30f4dc5-6a5e-4474-be39-088181771d7c" containerName="istio-proxy"
Sep 15 21:36:13.185166 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.185145    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e23eece0-f61d-4cae-89ba-e0487a370327" containerName="maas-api"
Sep 15 21:36:13.185166 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.185150    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e23eece0-f61d-4cae-89ba-e0487a370327" containerName="maas-api"
Sep 15 21:36:13.185166 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.185161    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e02ae5-4e2b-414a-ab5a-a71180e29de9" containerName="maas-api"
Sep 15 21:36:13.185166 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.185166    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e02ae5-4e2b-414a-ab5a-a71180e29de9" containerName="maas-api"
Sep 15 21:36:13.185720 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.185215    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4e02ae5-4e2b-414a-ab5a-a71180e29de9" containerName="maas-api"
Sep 15 21:36:13.185720 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.185223    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a30f4dc5-6a5e-4474-be39-088181771d7c" containerName="istio-proxy"
Sep 15 21:36:13.185720 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.185228    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b821b12-7463-448a-af49-e3a5ea9c4b95" containerName="istio-proxy"
Sep 15 21:36:13.185720 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.185234    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a30f4dc5-6a5e-4474-be39-088181771d7c" containerName="istio-proxy"
Sep 15 21:36:13.185720 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.185239    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e23eece0-f61d-4cae-89ba-e0487a370327" containerName="maas-api"
Sep 15 21:36:13.185720 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.185245    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff08845a-385f-42be-8231-eb324d9faeb1" containerName="maas-api"
Sep 15 21:36:13.185720 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.185250    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="56b137a3-3518-4882-84d8-aba07c7cd6c0" containerName="istio-proxy"
Sep 15 21:36:13.185720 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.185255    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="56b137a3-3518-4882-84d8-aba07c7cd6c0" containerName="istio-proxy"
Sep 15 21:36:13.188179 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.188162    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9"
Sep 15 21:36:13.193327 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.193300    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8hfhw\""
Sep 15 21:36:13.193481 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.193348    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 15 21:36:13.193555 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.193496    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:36:13.193631 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.193613    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 21:36:13.193728 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.193712    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 21:36:13.205441 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.205414    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9"]
Sep 15 21:36:13.207481 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.207461    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5cf7bfc64b-bxpx4"]
Sep 15 21:36:13.207803 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.207777    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a30f4dc5-6a5e-4474-be39-088181771d7c" containerName="istio-proxy"
Sep 15 21:36:13.207803 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.207797    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a30f4dc5-6a5e-4474-be39-088181771d7c" containerName="istio-proxy"
Sep 15 21:36:13.207961 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.207824    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b821b12-7463-448a-af49-e3a5ea9c4b95" containerName="istio-proxy"
Sep 15 21:36:13.207961 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.207830    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b821b12-7463-448a-af49-e3a5ea9c4b95" containerName="istio-proxy"
Sep 15 21:36:13.207961 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.207886    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b821b12-7463-448a-af49-e3a5ea9c4b95" containerName="istio-proxy"
Sep 15 21:36:13.210951 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.210932    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cf7bfc64b-bxpx4"
Sep 15 21:36:13.225952 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.225928    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cf7bfc64b-bxpx4"]
Sep 15 21:36:13.361171 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.361135    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfadc584-e376-425f-b77f-004d6ee049fa-openshift-service-ca\") pod \"maas-api-5cf7bfc64b-bxpx4\" (UID: \"bfadc584-e376-425f-b77f-004d6ee049fa\") " pod="odh-ai-gateway-infra/maas-api-5cf7bfc64b-bxpx4"
Sep 15 21:36:13.361171 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.361169    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a357c326-f932-48cc-89bc-61cd7f19aa8a-metrics-certs\") pod \"maas-api-646ff496d6-ttjq9\" (UID: \"a357c326-f932-48cc-89bc-61cd7f19aa8a\") " pod="odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9"
Sep 15 21:36:13.361384 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.361191    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfadc584-e376-425f-b77f-004d6ee049fa-maas-api-tls\") pod \"maas-api-5cf7bfc64b-bxpx4\" (UID: \"bfadc584-e376-425f-b77f-004d6ee049fa\") " pod="odh-ai-gateway-infra/maas-api-5cf7bfc64b-bxpx4"
Sep 15 21:36:13.361384 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.361239    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vf44\" (UniqueName: \"kubernetes.io/projected/bfadc584-e376-425f-b77f-004d6ee049fa-kube-api-access-8vf44\") pod \"maas-api-5cf7bfc64b-bxpx4\" (UID: \"bfadc584-e376-425f-b77f-004d6ee049fa\") " pod="odh-ai-gateway-infra/maas-api-5cf7bfc64b-bxpx4"
Sep 15 21:36:13.361384 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.361284    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcdqd\" (UniqueName: \"kubernetes.io/projected/a357c326-f932-48cc-89bc-61cd7f19aa8a-kube-api-access-mcdqd\") pod \"maas-api-646ff496d6-ttjq9\" (UID: \"a357c326-f932-48cc-89bc-61cd7f19aa8a\") " pod="odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9"
Sep 15 21:36:13.361384 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.361328    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfadc584-e376-425f-b77f-004d6ee049fa-metrics-certs\") pod \"maas-api-5cf7bfc64b-bxpx4\" (UID: \"bfadc584-e376-425f-b77f-004d6ee049fa\") " pod="odh-ai-gateway-infra/maas-api-5cf7bfc64b-bxpx4"
Sep 15 21:36:13.361384 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.361352    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a357c326-f932-48cc-89bc-61cd7f19aa8a-maas-api-tls\") pod \"maas-api-646ff496d6-ttjq9\" (UID: \"a357c326-f932-48cc-89bc-61cd7f19aa8a\") " pod="odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9"
Sep 15 21:36:13.361554 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.361424    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a357c326-f932-48cc-89bc-61cd7f19aa8a-openshift-service-ca\") pod \"maas-api-646ff496d6-ttjq9\" (UID: \"a357c326-f932-48cc-89bc-61cd7f19aa8a\") " pod="odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9"
Sep 15 21:36:13.462077 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.461976    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfadc584-e376-425f-b77f-004d6ee049fa-maas-api-tls\") pod \"maas-api-5cf7bfc64b-bxpx4\" (UID: \"bfadc584-e376-425f-b77f-004d6ee049fa\") " pod="odh-ai-gateway-infra/maas-api-5cf7bfc64b-bxpx4"
Sep 15 21:36:13.462077 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.462012    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vf44\" (UniqueName: \"kubernetes.io/projected/bfadc584-e376-425f-b77f-004d6ee049fa-kube-api-access-8vf44\") pod \"maas-api-5cf7bfc64b-bxpx4\" (UID: \"bfadc584-e376-425f-b77f-004d6ee049fa\") " pod="odh-ai-gateway-infra/maas-api-5cf7bfc64b-bxpx4"
Sep 15 21:36:13.462077 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.462036    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcdqd\" (UniqueName: \"kubernetes.io/projected/a357c326-f932-48cc-89bc-61cd7f19aa8a-kube-api-access-mcdqd\") pod \"maas-api-646ff496d6-ttjq9\" (UID: \"a357c326-f932-48cc-89bc-61cd7f19aa8a\") " pod="odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9"
Sep 15 21:36:13.462289 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.462241    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfadc584-e376-425f-b77f-004d6ee049fa-metrics-certs\") pod \"maas-api-5cf7bfc64b-bxpx4\" (UID: \"bfadc584-e376-425f-b77f-004d6ee049fa\") " pod="odh-ai-gateway-infra/maas-api-5cf7bfc64b-bxpx4"
Sep 15 21:36:13.462289 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.462273    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a357c326-f932-48cc-89bc-61cd7f19aa8a-maas-api-tls\") pod \"maas-api-646ff496d6-ttjq9\" (UID: \"a357c326-f932-48cc-89bc-61cd7f19aa8a\") " pod="odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9"
Sep 15 21:36:13.462361 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.462325    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a357c326-f932-48cc-89bc-61cd7f19aa8a-openshift-service-ca\") pod \"maas-api-646ff496d6-ttjq9\" (UID: \"a357c326-f932-48cc-89bc-61cd7f19aa8a\") " pod="odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9"
Sep 15 21:36:13.462413 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.462370    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfadc584-e376-425f-b77f-004d6ee049fa-openshift-service-ca\") pod \"maas-api-5cf7bfc64b-bxpx4\" (UID: \"bfadc584-e376-425f-b77f-004d6ee049fa\") " pod="odh-ai-gateway-infra/maas-api-5cf7bfc64b-bxpx4"
Sep 15 21:36:13.462467 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.462429    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a357c326-f932-48cc-89bc-61cd7f19aa8a-metrics-certs\") pod \"maas-api-646ff496d6-ttjq9\" (UID: \"a357c326-f932-48cc-89bc-61cd7f19aa8a\") " pod="odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9"
Sep 15 21:36:13.463065 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.463025    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a357c326-f932-48cc-89bc-61cd7f19aa8a-openshift-service-ca\") pod \"maas-api-646ff496d6-ttjq9\" (UID: \"a357c326-f932-48cc-89bc-61cd7f19aa8a\") " pod="odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9"
Sep 15 21:36:13.463190 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.463089    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfadc584-e376-425f-b77f-004d6ee049fa-openshift-service-ca\") pod \"maas-api-5cf7bfc64b-bxpx4\" (UID: \"bfadc584-e376-425f-b77f-004d6ee049fa\") " pod="odh-ai-gateway-infra/maas-api-5cf7bfc64b-bxpx4"
Sep 15 21:36:13.465247 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.465219    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfadc584-e376-425f-b77f-004d6ee049fa-metrics-certs\") pod \"maas-api-5cf7bfc64b-bxpx4\" (UID: \"bfadc584-e376-425f-b77f-004d6ee049fa\") " pod="odh-ai-gateway-infra/maas-api-5cf7bfc64b-bxpx4"
Sep 15 21:36:13.465335 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.465219    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfadc584-e376-425f-b77f-004d6ee049fa-maas-api-tls\") pod \"maas-api-5cf7bfc64b-bxpx4\" (UID: \"bfadc584-e376-425f-b77f-004d6ee049fa\") " pod="odh-ai-gateway-infra/maas-api-5cf7bfc64b-bxpx4"
Sep 15 21:36:13.465335 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.465227    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a357c326-f932-48cc-89bc-61cd7f19aa8a-metrics-certs\") pod \"maas-api-646ff496d6-ttjq9\" (UID: \"a357c326-f932-48cc-89bc-61cd7f19aa8a\") " pod="odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9"
Sep 15 21:36:13.465408 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.465358    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a357c326-f932-48cc-89bc-61cd7f19aa8a-maas-api-tls\") pod \"maas-api-646ff496d6-ttjq9\" (UID: \"a357c326-f932-48cc-89bc-61cd7f19aa8a\") " pod="odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9"
Sep 15 21:36:13.473955 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.473932    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcdqd\" (UniqueName: \"kubernetes.io/projected/a357c326-f932-48cc-89bc-61cd7f19aa8a-kube-api-access-mcdqd\") pod \"maas-api-646ff496d6-ttjq9\" (UID: \"a357c326-f932-48cc-89bc-61cd7f19aa8a\") " pod="odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9"
Sep 15 21:36:13.475025 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.474997    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vf44\" (UniqueName: \"kubernetes.io/projected/bfadc584-e376-425f-b77f-004d6ee049fa-kube-api-access-8vf44\") pod \"maas-api-5cf7bfc64b-bxpx4\" (UID: \"bfadc584-e376-425f-b77f-004d6ee049fa\") " pod="odh-ai-gateway-infra/maas-api-5cf7bfc64b-bxpx4"
Sep 15 21:36:13.498740 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.498720    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9"
Sep 15 21:36:13.520733 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.520704    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cf7bfc64b-bxpx4"
Sep 15 21:36:13.776870 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.776846    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9"]
Sep 15 21:36:13.778246 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:36:13.778222    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda357c326_f932_48cc_89bc_61cd7f19aa8a.slice/crio-0bd0b28527f25c092fe04862c2d1b75ae1d2b8dda37c17c6faea1f98e0a3f5eb WatchSource:0}: Error finding container 0bd0b28527f25c092fe04862c2d1b75ae1d2b8dda37c17c6faea1f98e0a3f5eb: Status 404 returned error can't find the container with id 0bd0b28527f25c092fe04862c2d1b75ae1d2b8dda37c17c6faea1f98e0a3f5eb
Sep 15 21:36:13.779590 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.779572    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:36:13.806558 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:13.806533    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cf7bfc64b-bxpx4"]
Sep 15 21:36:13.807437 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:36:13.807412    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfadc584_e376_425f_b77f_004d6ee049fa.slice/crio-7e09a8d2d3c9253176fe0820ab4f9088137f32b90af1091cb6cce5ffa38d98d4 WatchSource:0}: Error finding container 7e09a8d2d3c9253176fe0820ab4f9088137f32b90af1091cb6cce5ffa38d98d4: Status 404 returned error can't find the container with id 7e09a8d2d3c9253176fe0820ab4f9088137f32b90af1091cb6cce5ffa38d98d4
Sep 15 21:36:14.497937 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:14.497896    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9" event={"ID":"a357c326-f932-48cc-89bc-61cd7f19aa8a","Type":"ContainerStarted","Data":"fa7daa02d89bfb920d3f5aa72532cda832436f8490f191663655d97f311003f5"}
Sep 15 21:36:14.498439 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:14.497968    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9" event={"ID":"a357c326-f932-48cc-89bc-61cd7f19aa8a","Type":"ContainerStarted","Data":"0bd0b28527f25c092fe04862c2d1b75ae1d2b8dda37c17c6faea1f98e0a3f5eb"}
Sep 15 21:36:14.498439 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:14.498000    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9"
Sep 15 21:36:14.500341 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:14.500310    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cf7bfc64b-bxpx4" event={"ID":"bfadc584-e376-425f-b77f-004d6ee049fa","Type":"ContainerStarted","Data":"ef27d4c7c51cfdf0e08ee76064b5dc2db0dd7688cd1d72b83da609218ae4359c"}
Sep 15 21:36:14.500464 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:14.500367    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cf7bfc64b-bxpx4" event={"ID":"bfadc584-e376-425f-b77f-004d6ee049fa","Type":"ContainerStarted","Data":"7e09a8d2d3c9253176fe0820ab4f9088137f32b90af1091cb6cce5ffa38d98d4"}
Sep 15 21:36:14.500464 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:14.500411    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5cf7bfc64b-bxpx4"
Sep 15 21:36:14.553573 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:14.553468    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9" podStartSLOduration=1.038534467 podStartE2EDuration="1.553450977s" totalImagesPullingTime="514.91651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:14.551342494 +0000 UTC m=+2293.041988225" watchObservedRunningTime="2026-09-15 21:36:14.553450977 +0000 UTC m=+2293.044096703"
Sep 15 21:36:14.606123 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:14.606071    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5cf7bfc64b-bxpx4" podStartSLOduration=1.160383463 podStartE2EDuration="1.606041505s" totalImagesPullingTime="445.658042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:14.603285429 +0000 UTC m=+2293.093931163" watchObservedRunningTime="2026-09-15 21:36:14.606041505 +0000 UTC m=+2293.096687275"
Sep 15 21:36:20.513464 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:20.513429    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9"
Sep 15 21:36:20.513965 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:20.513583    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5cf7bfc64b-bxpx4"
Sep 15 21:36:20.586876 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:20.586843    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9"]
Sep 15 21:36:20.587125 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:20.587080    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9" podUID="a357c326-f932-48cc-89bc-61cd7f19aa8a" containerName="maas-api" containerID="cri-o://fa7daa02d89bfb920d3f5aa72532cda832436f8490f191663655d97f311003f5" gracePeriod=30
Sep 15 21:36:20.691091 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:20.691037    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:36:20.830016 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:20.829988    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9"
Sep 15 21:36:20.926881 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:20.926845    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a357c326-f932-48cc-89bc-61cd7f19aa8a-metrics-certs\") pod \"a357c326-f932-48cc-89bc-61cd7f19aa8a\" (UID: \"a357c326-f932-48cc-89bc-61cd7f19aa8a\") "
Sep 15 21:36:20.927090 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:20.926921    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a357c326-f932-48cc-89bc-61cd7f19aa8a-maas-api-tls\") pod \"a357c326-f932-48cc-89bc-61cd7f19aa8a\" (UID: \"a357c326-f932-48cc-89bc-61cd7f19aa8a\") "
Sep 15 21:36:20.927090 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:20.926948    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a357c326-f932-48cc-89bc-61cd7f19aa8a-openshift-service-ca\") pod \"a357c326-f932-48cc-89bc-61cd7f19aa8a\" (UID: \"a357c326-f932-48cc-89bc-61cd7f19aa8a\") "
Sep 15 21:36:20.927090 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:20.926993    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcdqd\" (UniqueName: \"kubernetes.io/projected/a357c326-f932-48cc-89bc-61cd7f19aa8a-kube-api-access-mcdqd\") pod \"a357c326-f932-48cc-89bc-61cd7f19aa8a\" (UID: \"a357c326-f932-48cc-89bc-61cd7f19aa8a\") "
Sep 15 21:36:20.927401 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:20.927364    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a357c326-f932-48cc-89bc-61cd7f19aa8a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a357c326-f932-48cc-89bc-61cd7f19aa8a" (UID: "a357c326-f932-48cc-89bc-61cd7f19aa8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:36:20.929091 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:20.929042    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a357c326-f932-48cc-89bc-61cd7f19aa8a-kube-api-access-mcdqd" (OuterVolumeSpecName: "kube-api-access-mcdqd") pod "a357c326-f932-48cc-89bc-61cd7f19aa8a" (UID: "a357c326-f932-48cc-89bc-61cd7f19aa8a"). InnerVolumeSpecName "kube-api-access-mcdqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:36:20.929172 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:20.929093    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a357c326-f932-48cc-89bc-61cd7f19aa8a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a357c326-f932-48cc-89bc-61cd7f19aa8a" (UID: "a357c326-f932-48cc-89bc-61cd7f19aa8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:36:20.929172 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:20.929112    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a357c326-f932-48cc-89bc-61cd7f19aa8a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a357c326-f932-48cc-89bc-61cd7f19aa8a" (UID: "a357c326-f932-48cc-89bc-61cd7f19aa8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:36:21.028363 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:21.028332    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a357c326-f932-48cc-89bc-61cd7f19aa8a-maas-api-tls\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:36:21.028363 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:21.028358    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a357c326-f932-48cc-89bc-61cd7f19aa8a-openshift-service-ca\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:36:21.028363 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:21.028368    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcdqd\" (UniqueName: \"kubernetes.io/projected/a357c326-f932-48cc-89bc-61cd7f19aa8a-kube-api-access-mcdqd\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:36:21.028585 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:21.028378    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a357c326-f932-48cc-89bc-61cd7f19aa8a-metrics-certs\") on node \"ip-10-0-142-207.ec2.internal\" DevicePath \"\""
Sep 15 21:36:21.540982 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:21.540948    2558 generic.go:356] "Generic (PLEG): container finished" podID="a357c326-f932-48cc-89bc-61cd7f19aa8a" containerID="fa7daa02d89bfb920d3f5aa72532cda832436f8490f191663655d97f311003f5" exitCode=0
Sep 15 21:36:21.541412 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:21.540994    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9"
Sep 15 21:36:21.541412 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:21.540996    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9" event={"ID":"a357c326-f932-48cc-89bc-61cd7f19aa8a","Type":"ContainerDied","Data":"fa7daa02d89bfb920d3f5aa72532cda832436f8490f191663655d97f311003f5"}
Sep 15 21:36:21.541412 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:21.541132    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9" event={"ID":"a357c326-f932-48cc-89bc-61cd7f19aa8a","Type":"ContainerDied","Data":"0bd0b28527f25c092fe04862c2d1b75ae1d2b8dda37c17c6faea1f98e0a3f5eb"}
Sep 15 21:36:21.541412 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:21.541162    2558 scope.go:117] "RemoveContainer" containerID="fa7daa02d89bfb920d3f5aa72532cda832436f8490f191663655d97f311003f5"
Sep 15 21:36:21.551468 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:21.551145    2558 scope.go:117] "RemoveContainer" containerID="fa7daa02d89bfb920d3f5aa72532cda832436f8490f191663655d97f311003f5"
Sep 15 21:36:21.554149 ip-10-0-142-207 kubenswrapper[2558]: E0915 21:36:21.551608    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa7daa02d89bfb920d3f5aa72532cda832436f8490f191663655d97f311003f5\": container with ID starting with fa7daa02d89bfb920d3f5aa72532cda832436f8490f191663655d97f311003f5 not found: ID does not exist" containerID="fa7daa02d89bfb920d3f5aa72532cda832436f8490f191663655d97f311003f5"
Sep 15 21:36:21.554149 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:21.551648    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa7daa02d89bfb920d3f5aa72532cda832436f8490f191663655d97f311003f5"} err="failed to get container status \"fa7daa02d89bfb920d3f5aa72532cda832436f8490f191663655d97f311003f5\": rpc error: code = NotFound desc = could not find container \"fa7daa02d89bfb920d3f5aa72532cda832436f8490f191663655d97f311003f5\": container with ID starting with fa7daa02d89bfb920d3f5aa72532cda832436f8490f191663655d97f311003f5 not found: ID does not exist"
Sep 15 21:36:21.565884 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:21.565855    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9"]
Sep 15 21:36:21.568906 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:21.568883    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-646ff496d6-ttjq9"]
Sep 15 21:36:21.941915 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:21.941881    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a357c326-f932-48cc-89bc-61cd7f19aa8a" path="/var/lib/kubelet/pods/a357c326-f932-48cc-89bc-61cd7f19aa8a/volumes"
Sep 15 21:36:27.606820 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:27.606787    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:36:33.783587 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:33.783551    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:36:40.090685 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:40.090650    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-stpzk"]
Sep 15 21:36:42.890012 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:42.889974    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cc985957c-4q9c7_73e75579-00ec-424b-8b07-852bae98085b/authorino/0.log"
Sep 15 21:36:48.768823 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:48.768794    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85789df9-5jrpl_b3bd7fa2-5cec-425c-8cc1-428bd89ded5b/copy-manifests/0.log"
Sep 15 21:36:48.775992 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:48.775970    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85789df9-5jrpl_b3bd7fa2-5cec-425c-8cc1-428bd89ded5b/manager/0.log"
Sep 15 21:36:49.465761 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:49.465734    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5cf7bfc64b-bxpx4_bfadc584-e376-425f-b77f-004d6ee049fa/maas-api/0.log"
Sep 15 21:36:50.568607 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:50.568574    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cc985957c-4q9c7_73e75579-00ec-424b-8b07-852bae98085b/authorino/0.log"
Sep 15 21:36:50.898263 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:50.898184    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-tblrx_8ade0084-60cd-454e-8ab8-16aa903c2bda/kuadrant-console-plugin/0.log"
Sep 15 21:36:51.143431 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:51.143407    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-stpzk_e9aa7166-dec9-4129-bc7d-76e4b0ce15c5/limitador/0.log"
Sep 15 21:36:51.821693 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:51.821664    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85f7c7997-p8wzc_c1d7c96f-a35b-4aba-b0b8-5b8939ceeb04/kube-auth-proxy/0.log"
Sep 15 21:36:53.343157 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:53.343128    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-ln2wz_934756fd-2f06-46a2-95f7-1ac46b50d4a2/main/0.log"
Sep 15 21:36:58.708357 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:58.708278    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hwqzn/must-gather-274wv"]
Sep 15 21:36:58.708738 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:58.708629    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a357c326-f932-48cc-89bc-61cd7f19aa8a" containerName="maas-api"
Sep 15 21:36:58.708738 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:58.708642    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a357c326-f932-48cc-89bc-61cd7f19aa8a" containerName="maas-api"
Sep 15 21:36:58.708738 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:58.708708    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a357c326-f932-48cc-89bc-61cd7f19aa8a" containerName="maas-api"
Sep 15 21:36:58.712139 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:58.712109    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hwqzn/must-gather-274wv"
Sep 15 21:36:58.719367 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:58.719348    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hwqzn\"/\"openshift-service-ca.crt\""
Sep 15 21:36:58.719491 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:58.719417    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hwqzn\"/\"default-dockercfg-dzqc9\""
Sep 15 21:36:58.720743 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:58.720724    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hwqzn\"/\"kube-root-ca.crt\""
Sep 15 21:36:58.733582 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:58.733559    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hwqzn/must-gather-274wv"]
Sep 15 21:36:58.853656 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:58.853623    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz9qr\" (UniqueName: \"kubernetes.io/projected/30cc2860-fa4b-45dc-88d3-59332bfb6901-kube-api-access-rz9qr\") pod \"must-gather-274wv\" (UID: \"30cc2860-fa4b-45dc-88d3-59332bfb6901\") " pod="openshift-must-gather-hwqzn/must-gather-274wv"
Sep 15 21:36:58.853858 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:58.853681    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/30cc2860-fa4b-45dc-88d3-59332bfb6901-must-gather-output\") pod \"must-gather-274wv\" (UID: \"30cc2860-fa4b-45dc-88d3-59332bfb6901\") " pod="openshift-must-gather-hwqzn/must-gather-274wv"
Sep 15 21:36:58.954901 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:58.954866    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/30cc2860-fa4b-45dc-88d3-59332bfb6901-must-gather-output\") pod \"must-gather-274wv\" (UID: \"30cc2860-fa4b-45dc-88d3-59332bfb6901\") " pod="openshift-must-gather-hwqzn/must-gather-274wv"
Sep 15 21:36:58.955122 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:58.954966    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz9qr\" (UniqueName: \"kubernetes.io/projected/30cc2860-fa4b-45dc-88d3-59332bfb6901-kube-api-access-rz9qr\") pod \"must-gather-274wv\" (UID: \"30cc2860-fa4b-45dc-88d3-59332bfb6901\") " pod="openshift-must-gather-hwqzn/must-gather-274wv"
Sep 15 21:36:58.955229 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:58.955208    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/30cc2860-fa4b-45dc-88d3-59332bfb6901-must-gather-output\") pod \"must-gather-274wv\" (UID: \"30cc2860-fa4b-45dc-88d3-59332bfb6901\") " pod="openshift-must-gather-hwqzn/must-gather-274wv"
Sep 15 21:36:58.966586 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:58.966524    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz9qr\" (UniqueName: \"kubernetes.io/projected/30cc2860-fa4b-45dc-88d3-59332bfb6901-kube-api-access-rz9qr\") pod \"must-gather-274wv\" (UID: \"30cc2860-fa4b-45dc-88d3-59332bfb6901\") " pod="openshift-must-gather-hwqzn/must-gather-274wv"
Sep 15 21:36:59.021902 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:59.021872    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hwqzn/must-gather-274wv"
Sep 15 21:36:59.317646 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:59.317615    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hwqzn/must-gather-274wv"]
Sep 15 21:36:59.319445 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:36:59.319413    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30cc2860_fa4b_45dc_88d3_59332bfb6901.slice/crio-e95dfc9bdcedc1b6a3535b4b9c4cac8250f23eeda131e8f88cb6987f76c2636e WatchSource:0}: Error finding container e95dfc9bdcedc1b6a3535b4b9c4cac8250f23eeda131e8f88cb6987f76c2636e: Status 404 returned error can't find the container with id e95dfc9bdcedc1b6a3535b4b9c4cac8250f23eeda131e8f88cb6987f76c2636e
Sep 15 21:36:59.748778 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:36:59.748730    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hwqzn/must-gather-274wv" event={"ID":"30cc2860-fa4b-45dc-88d3-59332bfb6901","Type":"ContainerStarted","Data":"e95dfc9bdcedc1b6a3535b4b9c4cac8250f23eeda131e8f88cb6987f76c2636e"}
Sep 15 21:37:00.759035 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:00.758992    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hwqzn/must-gather-274wv" event={"ID":"30cc2860-fa4b-45dc-88d3-59332bfb6901","Type":"ContainerStarted","Data":"28ff11cb2d9f9a4a9cdf6caaee5b4a076fa27082e1e5c29b1adbcb42cfe89ddd"}
Sep 15 21:37:00.759450 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:00.759071    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hwqzn/must-gather-274wv" event={"ID":"30cc2860-fa4b-45dc-88d3-59332bfb6901","Type":"ContainerStarted","Data":"a13ed071bac4abafa7e9bdca1b65e13501af6e64c075ad82f06412667bdc1599"}
Sep 15 21:37:00.777486 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:00.777434    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hwqzn/must-gather-274wv" podStartSLOduration=2.141259136 podStartE2EDuration="2.77741722s" totalImagesPullingTime="636.158084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:37:00.775331426 +0000 UTC m=+2339.265977159" watchObservedRunningTime="2026-09-15 21:37:00.77741722 +0000 UTC m=+2339.268062974"
Sep 15 21:37:01.601722 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:01.601692    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qphq9_932cce37-6a6a-4968-8cbe-d163e1da16f0/global-pull-secret-syncer/0.log"
Sep 15 21:37:01.677182 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:01.677150    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pc79j_ef60e7a1-62c7-46aa-8fd8-4043168ffa94/konnectivity-agent/0.log"
Sep 15 21:37:01.789284 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:01.789255    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-207.ec2.internal_1c3857c19d72e89622f4c04f57a3da21/haproxy/0.log"
Sep 15 21:37:06.930028 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:06.929988    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cc985957c-4q9c7_73e75579-00ec-424b-8b07-852bae98085b/authorino/0.log"
Sep 15 21:37:07.047940 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:07.047910    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-tblrx_8ade0084-60cd-454e-8ab8-16aa903c2bda/kuadrant-console-plugin/0.log"
Sep 15 21:37:07.124504 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:07.124467    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-stpzk_e9aa7166-dec9-4129-bc7d-76e4b0ce15c5/limitador/0.log"
Sep 15 21:37:08.836084 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:08.836012    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d15d6dda-588c-4b50-bd5b-0f4401619f0d/alertmanager/0.log"
Sep 15 21:37:08.865723 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:08.865688    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d15d6dda-588c-4b50-bd5b-0f4401619f0d/config-reloader/0.log"
Sep 15 21:37:08.890371 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:08.890296    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d15d6dda-588c-4b50-bd5b-0f4401619f0d/kube-rbac-proxy-web/0.log"
Sep 15 21:37:08.918523 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:08.918498    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d15d6dda-588c-4b50-bd5b-0f4401619f0d/kube-rbac-proxy/0.log"
Sep 15 21:37:08.941770 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:08.941695    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d15d6dda-588c-4b50-bd5b-0f4401619f0d/kube-rbac-proxy-metric/0.log"
Sep 15 21:37:08.968760 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:08.968674    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d15d6dda-588c-4b50-bd5b-0f4401619f0d/prom-label-proxy/0.log"
Sep 15 21:37:08.999462 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:08.999409    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d15d6dda-588c-4b50-bd5b-0f4401619f0d/init-config-reloader/0.log"
Sep 15 21:37:09.161501 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:09.161477    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-647b67747-469rh_12d35a75-1573-447f-a27b-b337d5069891/metrics-server/0.log"
Sep 15 21:37:09.189698 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:09.189667    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-rwhsp_16c1df55-089f-4b2e-bba5-39c25ea6bd1f/monitoring-plugin/0.log"
Sep 15 21:37:09.326644 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:09.326560    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b8f5c_aa5b7933-e78a-4ca1-b46e-c101a3f025c5/node-exporter/0.log"
Sep 15 21:37:09.343326 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:09.343277    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b8f5c_aa5b7933-e78a-4ca1-b46e-c101a3f025c5/kube-rbac-proxy/0.log"
Sep 15 21:37:09.368900 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:09.368864    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b8f5c_aa5b7933-e78a-4ca1-b46e-c101a3f025c5/init-textfile/0.log"
Sep 15 21:37:09.497280 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:09.496927    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-c86pl_3f77ec77-c110-445c-82d2-229051a5adc6/kube-rbac-proxy-main/0.log"
Sep 15 21:37:09.532168 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:09.531977    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-c86pl_3f77ec77-c110-445c-82d2-229051a5adc6/kube-rbac-proxy-self/0.log"
Sep 15 21:37:09.555503 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:09.555447    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-c86pl_3f77ec77-c110-445c-82d2-229051a5adc6/openshift-state-metrics/0.log"
Sep 15 21:37:09.784166 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:09.784132    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-jbx4k_694bdf81-364f-4d17-90e2-ca9fe991ce11/prometheus-operator/0.log"
Sep 15 21:37:09.809603 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:09.809567    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-jbx4k_694bdf81-364f-4d17-90e2-ca9fe991ce11/kube-rbac-proxy/0.log"
Sep 15 21:37:11.633172 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:11.633135    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn"]
Sep 15 21:37:11.640521 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:11.640493    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn"
Sep 15 21:37:11.648208 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:11.648177    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn"]
Sep 15 21:37:11.692217 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:11.692174    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d91a698b-1edc-40a9-8a18-d5cdcb161150-lib-modules\") pod \"perf-node-gather-daemonset-tkfwn\" (UID: \"d91a698b-1edc-40a9-8a18-d5cdcb161150\") " pod="openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn"
Sep 15 21:37:11.692400 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:11.692235    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d91a698b-1edc-40a9-8a18-d5cdcb161150-proc\") pod \"perf-node-gather-daemonset-tkfwn\" (UID: \"d91a698b-1edc-40a9-8a18-d5cdcb161150\") " pod="openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn"
Sep 15 21:37:11.692400 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:11.692270    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d91a698b-1edc-40a9-8a18-d5cdcb161150-sys\") pod \"perf-node-gather-daemonset-tkfwn\" (UID: \"d91a698b-1edc-40a9-8a18-d5cdcb161150\") " pod="openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn"
Sep 15 21:37:11.692400 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:11.692316    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx544\" (UniqueName: \"kubernetes.io/projected/d91a698b-1edc-40a9-8a18-d5cdcb161150-kube-api-access-hx544\") pod \"perf-node-gather-daemonset-tkfwn\" (UID: \"d91a698b-1edc-40a9-8a18-d5cdcb161150\") " pod="openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn"
Sep 15 21:37:11.692400 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:11.692383    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d91a698b-1edc-40a9-8a18-d5cdcb161150-podres\") pod \"perf-node-gather-daemonset-tkfwn\" (UID: \"d91a698b-1edc-40a9-8a18-d5cdcb161150\") " pod="openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn"
Sep 15 21:37:11.793359 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:11.793312    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d91a698b-1edc-40a9-8a18-d5cdcb161150-lib-modules\") pod \"perf-node-gather-daemonset-tkfwn\" (UID: \"d91a698b-1edc-40a9-8a18-d5cdcb161150\") " pod="openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn"
Sep 15 21:37:11.793569 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:11.793378    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d91a698b-1edc-40a9-8a18-d5cdcb161150-proc\") pod \"perf-node-gather-daemonset-tkfwn\" (UID: \"d91a698b-1edc-40a9-8a18-d5cdcb161150\") " pod="openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn"
Sep 15 21:37:11.793569 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:11.793416    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d91a698b-1edc-40a9-8a18-d5cdcb161150-sys\") pod \"perf-node-gather-daemonset-tkfwn\" (UID: \"d91a698b-1edc-40a9-8a18-d5cdcb161150\") " pod="openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn"
Sep 15 21:37:11.793569 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:11.793464    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx544\" (UniqueName: \"kubernetes.io/projected/d91a698b-1edc-40a9-8a18-d5cdcb161150-kube-api-access-hx544\") pod \"perf-node-gather-daemonset-tkfwn\" (UID: \"d91a698b-1edc-40a9-8a18-d5cdcb161150\") " pod="openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn"
Sep 15 21:37:11.793569 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:11.793503    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d91a698b-1edc-40a9-8a18-d5cdcb161150-podres\") pod \"perf-node-gather-daemonset-tkfwn\" (UID: \"d91a698b-1edc-40a9-8a18-d5cdcb161150\") " pod="openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn"
Sep 15 21:37:11.793835 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:11.793672    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d91a698b-1edc-40a9-8a18-d5cdcb161150-podres\") pod \"perf-node-gather-daemonset-tkfwn\" (UID: \"d91a698b-1edc-40a9-8a18-d5cdcb161150\") " pod="openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn"
Sep 15 21:37:11.793835 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:11.793771    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d91a698b-1edc-40a9-8a18-d5cdcb161150-lib-modules\") pod \"perf-node-gather-daemonset-tkfwn\" (UID: \"d91a698b-1edc-40a9-8a18-d5cdcb161150\") " pod="openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn"
Sep 15 21:37:11.793835 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:11.793825    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d91a698b-1edc-40a9-8a18-d5cdcb161150-proc\") pod \"perf-node-gather-daemonset-tkfwn\" (UID: \"d91a698b-1edc-40a9-8a18-d5cdcb161150\") " pod="openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn"
Sep 15 21:37:11.794003 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:11.793871    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d91a698b-1edc-40a9-8a18-d5cdcb161150-sys\") pod \"perf-node-gather-daemonset-tkfwn\" (UID: \"d91a698b-1edc-40a9-8a18-d5cdcb161150\") " pod="openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn"
Sep 15 21:37:11.804243 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:11.804210    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx544\" (UniqueName: \"kubernetes.io/projected/d91a698b-1edc-40a9-8a18-d5cdcb161150-kube-api-access-hx544\") pod \"perf-node-gather-daemonset-tkfwn\" (UID: \"d91a698b-1edc-40a9-8a18-d5cdcb161150\") " pod="openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn"
Sep 15 21:37:11.955483 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:11.955439    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn"
Sep 15 21:37:12.243934 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:12.243845    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn"]
Sep 15 21:37:12.248730 ip-10-0-142-207 kubenswrapper[2558]: W0915 21:37:12.248692    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd91a698b_1edc_40a9_8a18_d5cdcb161150.slice/crio-e0177095f429625711c20e0a0d42b83b99ea5ef497855f239fb167670f909726 WatchSource:0}: Error finding container e0177095f429625711c20e0a0d42b83b99ea5ef497855f239fb167670f909726: Status 404 returned error can't find the container with id e0177095f429625711c20e0a0d42b83b99ea5ef497855f239fb167670f909726
Sep 15 21:37:12.858807 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:12.858766    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn" event={"ID":"d91a698b-1edc-40a9-8a18-d5cdcb161150","Type":"ContainerStarted","Data":"ccde8f1eb1381e4fa4808789868e64c4947f02a7909afc95b0e61d89686f74b2"}
Sep 15 21:37:12.859420 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:12.859393    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn" event={"ID":"d91a698b-1edc-40a9-8a18-d5cdcb161150","Type":"ContainerStarted","Data":"e0177095f429625711c20e0a0d42b83b99ea5ef497855f239fb167670f909726"}
Sep 15 21:37:12.859557 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:12.859544    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn"
Sep 15 21:37:12.879774 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:12.879710    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn" podStartSLOduration=1.879687619 podStartE2EDuration="1.879687619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:37:12.877313934 +0000 UTC m=+2351.367959669" watchObservedRunningTime="2026-09-15 21:37:12.879687619 +0000 UTC m=+2351.370333352"
Sep 15 21:37:14.163482 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:14.163451    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ftppc_8f3b8bea-139d-46e2-9477-22e1230ca193/dns/0.log"
Sep 15 21:37:14.189619 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:14.189591    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ftppc_8f3b8bea-139d-46e2-9477-22e1230ca193/kube-rbac-proxy/0.log"
Sep 15 21:37:14.286083 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:14.286042    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nwspf_cb9c01cf-5113-402e-a9ff-8f177ab7f018/dns-node-resolver/0.log"
Sep 15 21:37:14.867570 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:14.867523    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8d6b7655-2cwn2_e9273ede-dc5b-4757-a016-61ed4a57e043/registry/0.log"
Sep 15 21:37:14.962287 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:14.962256    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xkmbd_71567bed-fb87-4bb0-9833-a30ff9a16ef9/node-ca/0.log"
Sep 15 21:37:16.266800 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:16.266771    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85f7c7997-p8wzc_c1d7c96f-a35b-4aba-b0b8-5b8939ceeb04/kube-auth-proxy/0.log"
Sep 15 21:37:17.330720 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:17.330687    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m4ncl_6dc59ea6-8d86-456d-b250-e2d0fc7a2d10/serve-healthcheck-canary/0.log"
Sep 15 21:37:17.848202 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:17.848170    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6rn7r_6a2612f1-6b01-421d-b105-f4a20b6356f7/kube-rbac-proxy/0.log"
Sep 15 21:37:17.879617 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:17.879578    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6rn7r_6a2612f1-6b01-421d-b105-f4a20b6356f7/exporter/0.log"
Sep 15 21:37:17.910002 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:17.909978    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6rn7r_6a2612f1-6b01-421d-b105-f4a20b6356f7/extractor/0.log"
Sep 15 21:37:18.874390 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:18.874361    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hwqzn/perf-node-gather-daemonset-tkfwn"
Sep 15 21:37:20.926261 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:20.926214    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85789df9-5jrpl_b3bd7fa2-5cec-425c-8cc1-428bd89ded5b/manager/0.log"
Sep 15 21:37:20.946136 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:20.946109    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85789df9-5jrpl_b3bd7fa2-5cec-425c-8cc1-428bd89ded5b/copy-manifests/0.log"
Sep 15 21:37:22.349860 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:22.349826    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5665dbb77c-wgxl8_e064fdeb-b201-4d8c-8164-a809e0d58b29/manager/0.log"
Sep 15 21:37:22.400021 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:22.399974    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-jp7c7_7d0f3039-6a34-4ea1-92f1-672bbc5c6143/openshift-lws-operator/0.log"
Sep 15 21:37:22.523423 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:22.523374    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85f7c7997-p8wzc_c1d7c96f-a35b-4aba-b0b8-5b8939ceeb04/kube-auth-proxy/0.log"
Sep 15 21:37:24.643662 ip-10-0-142-207 kubenswrapper[2558]: I0915 21:37:24.643577    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85f7c7997-p8wzc_c1d7c96f-a35b-4aba-b0b8-5b8939ceeb04/kube-auth-proxy/0.log"