Sep 17 21:39:17.210439 ip-10-0-129-67 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 21:39:17.210452 ip-10-0-129-67 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 21:39:17.210459 ip-10-0-129-67 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 21:39:17.210614 ip-10-0-129-67 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 21:39:27.459609 ip-10-0-129-67 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 21:39:27.459631 ip-10-0-129-67 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 10e1bf88283349ff8c7f24113f873240 --
Sep 17 21:41:24.086550 ip-10-0-129-67 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 21:41:24.508696 ip-10-0-129-67 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 21:41:24.508696 ip-10-0-129-67 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 21:41:24.508696 ip-10-0-129-67 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 21:41:24.508696 ip-10-0-129-67 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 21:41:24.508696 ip-10-0-129-67 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 21:41:24.509855 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.509752    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 21:41:24.514180 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514150    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:41:24.514180 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514180    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:41:24.514243 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514184    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:41:24.514243 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514188    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:41:24.514243 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514191    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:41:24.514243 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514195    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:41:24.514243 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514198    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:41:24.514243 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514201    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:41:24.514243 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514209    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:41:24.514243 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514212    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:41:24.514243 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514215    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:41:24.514243 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514218    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:41:24.514243 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514220    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:41:24.514243 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514223    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:41:24.514243 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514225    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:41:24.514243 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514230    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:41:24.514243 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514235    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:41:24.514243 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514238    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:41:24.514243 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514241    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:41:24.514243 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514244    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:41:24.514243 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514247    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:41:24.514243 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514250    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:41:24.514713 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514253    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:41:24.514713 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514256    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:41:24.514713 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514259    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:41:24.514713 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514262    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:41:24.514713 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514264    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:41:24.514713 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514267    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:41:24.514713 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514270    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:41:24.514713 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514273    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:41:24.514713 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514275    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:41:24.514713 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514278    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:41:24.514713 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514280    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:41:24.514713 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514283    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:41:24.514713 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514285    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:41:24.514713 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514288    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:41:24.514713 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514291    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:41:24.514713 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514293    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:41:24.514713 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514298    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:41:24.514713 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514301    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:41:24.514713 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514304    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:41:24.514713 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514314    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:41:24.515199 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514316    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:41:24.515199 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514319    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:41:24.515199 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514322    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:41:24.515199 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514324    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:41:24.515199 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514327    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:41:24.515199 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514330    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:41:24.515199 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514332    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:41:24.515199 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514334    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:41:24.515199 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514337    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:41:24.515199 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514339    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:41:24.515199 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514342    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:41:24.515199 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514346    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:41:24.515199 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514350    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:41:24.515199 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514354    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:41:24.515199 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514357    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:41:24.515199 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514360    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:41:24.515199 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514363    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:41:24.515199 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514366    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:41:24.515199 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514369    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:41:24.515658 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514371    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:41:24.515658 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514374    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:41:24.515658 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514377    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:41:24.515658 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514380    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:41:24.515658 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514382    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:41:24.515658 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514385    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:41:24.515658 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514389    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:41:24.515658 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514391    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:41:24.515658 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514394    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:41:24.515658 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514396    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:41:24.515658 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514399    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:41:24.515658 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514401    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:41:24.515658 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514404    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:41:24.515658 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514412    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:41:24.515658 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514415    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:41:24.515658 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514417    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:41:24.515658 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514420    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:41:24.515658 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514422    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:41:24.515658 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514425    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:41:24.515658 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514427    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:41:24.516140 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514429    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:41:24.516140 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514432    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:41:24.516140 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514434    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:41:24.516140 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514437    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:41:24.516140 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514439    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:41:24.516140 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514442    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:41:24.516140 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514446    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:41:24.516140 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514449    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:41:24.516140 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514451    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:41:24.516140 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514454    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:41:24.516140 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514457    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:41:24.516140 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514459    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:41:24.516140 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514463    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:41:24.516140 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514465    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:41:24.516140 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514468    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:41:24.516140 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514470    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:41:24.516140 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514472    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:41:24.516140 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.514475    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:41:24.516140 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515249    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:41:24.516140 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515256    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:41:24.516640 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515259    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:41:24.516640 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515262    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:41:24.516640 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515265    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:41:24.516640 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515268    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:41:24.516640 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515271    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:41:24.516640 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515273    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:41:24.516640 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515276    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:41:24.516640 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515280    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:41:24.516640 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515283    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:41:24.516640 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515285    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:41:24.516640 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515288    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:41:24.516640 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515290    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:41:24.516640 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515293    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:41:24.516640 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515295    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:41:24.516640 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515298    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:41:24.516640 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515301    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:41:24.516640 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515303    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:41:24.516640 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515305    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:41:24.516640 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515310    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:41:24.516640 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515313    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:41:24.517192 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515316    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:41:24.517192 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515319    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:41:24.517192 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515321    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:41:24.517192 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515324    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:41:24.517192 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515327    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:41:24.517192 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515329    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:41:24.517192 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515332    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:41:24.517192 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515334    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:41:24.517192 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515337    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:41:24.517192 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515339    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:41:24.517192 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515342    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:41:24.517192 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515345    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:41:24.517192 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515347    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:41:24.517192 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515349    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:41:24.517192 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515352    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:41:24.517192 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515354    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:41:24.517192 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515357    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:41:24.517192 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515359    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:41:24.517192 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515362    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:41:24.517192 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515365    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:41:24.517688 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515368    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:41:24.517688 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515370    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:41:24.517688 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515372    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:41:24.517688 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515375    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:41:24.517688 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515379    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:41:24.517688 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515383    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:41:24.517688 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515386    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:41:24.517688 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515389    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:41:24.517688 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515392    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:41:24.517688 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515395    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:41:24.517688 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515399    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:41:24.517688 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515404    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:41:24.517688 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515409    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:41:24.517688 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515412    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:41:24.517688 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515415    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:41:24.517688 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515418    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:41:24.517688 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515421    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:41:24.517688 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515424    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:41:24.517688 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515427    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:41:24.518164 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515430    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:41:24.518164 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515433    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:41:24.518164 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515435    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:41:24.518164 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515438    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:41:24.518164 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515440    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:41:24.518164 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515442    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:41:24.518164 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515445    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:41:24.518164 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515448    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:41:24.518164 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515452    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:41:24.518164 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515454    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:41:24.518164 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515457    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:41:24.518164 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515459    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:41:24.518164 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515462    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:41:24.518164 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515465    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:41:24.518164 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515468    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:41:24.518164 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515470    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:41:24.518164 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515473    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:41:24.518164 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515475    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:41:24.518164 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515478    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:41:24.518675 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515480    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:41:24.518675 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515483    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:41:24.518675 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515486    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:41:24.518675 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515488    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:41:24.518675 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515491    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:41:24.518675 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515493    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:41:24.518675 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515496    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:41:24.518675 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515499    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:41:24.518675 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515501    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:41:24.518675 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515504    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:41:24.518675 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515506    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:41:24.518675 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515509    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:41:24.518675 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515511    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:41:24.518675 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515514    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:41:24.518675 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515516    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:41:24.518675 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515519    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:41:24.518675 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515522    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:41:24.518675 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515524    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:41:24.518675 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515527    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:41:24.518675 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515957    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:41:24.519150 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515963    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:41:24.519150 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515966    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:41:24.519150 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515969    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:41:24.519150 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515972    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:41:24.519150 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515975    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:41:24.519150 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515977    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:41:24.519150 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515980    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:41:24.519150 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515983    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:41:24.519150 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515985    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:41:24.519150 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515988    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:41:24.519150 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515991    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:41:24.519150 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515995    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:41:24.519150 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.515999    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:41:24.519150 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516002    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:41:24.519150 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516005    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:41:24.519150 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516007    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:41:24.519150 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516012    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:41:24.519150 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516015    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:41:24.519150 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516018    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:41:24.519622 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516020    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:41:24.519622 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516023    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:41:24.519622 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516025    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:41:24.519622 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516028    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:41:24.519622 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516031    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:41:24.519622 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516034    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:41:24.519622 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516039    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:41:24.519622 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516041    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:41:24.519622 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516044    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:41:24.519622 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516046    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:41:24.519622 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516049    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:41:24.519622 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516052    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:41:24.519622 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516054    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:41:24.519622 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516057    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:41:24.519622 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516059    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:41:24.519622 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516062    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:41:24.519622 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516065    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:41:24.519622 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516070    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:41:24.519622 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516073    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:41:24.520088 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516076    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:41:24.520088 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516079    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:41:24.520088 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516082    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:41:24.520088 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516085    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:41:24.520088 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516089    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:41:24.520088 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516091    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:41:24.520088 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516094    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:41:24.520088 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516097    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:41:24.520088 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516099    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:41:24.520088 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516102    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:41:24.520088 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516106    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:41:24.520088 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516108    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:41:24.520088 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516111    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:41:24.520088 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516113    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:41:24.520088 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516116    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:41:24.520088 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516118    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:41:24.520088 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516121    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:41:24.520088 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516123    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:41:24.520088 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516126    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:41:24.520088 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516133    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:41:24.520578 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516136    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:41:24.520578 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516138    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:41:24.520578 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516141    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:41:24.520578 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516143    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:41:24.520578 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516146    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:41:24.520578 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516150    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:41:24.520578 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516154    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:41:24.520578 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516157    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:41:24.520578 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516160    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:41:24.520578 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516162    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:41:24.520578 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516165    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:41:24.520578 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516181    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:41:24.520578 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516184    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:41:24.520578 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516187    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:41:24.520578 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516190    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:41:24.520578 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516193    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:41:24.520578 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516196    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:41:24.520578 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516198    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:41:24.520578 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516201    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:41:24.520578 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516203    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:41:24.521063 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516206    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:41:24.521063 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516208    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:41:24.521063 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516212    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:41:24.521063 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516216    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:41:24.521063 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516219    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:41:24.521063 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516222    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:41:24.521063 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516224    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:41:24.521063 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516227    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:41:24.521063 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516229    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:41:24.521063 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516232    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:41:24.521063 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516234    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:41:24.521063 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516238    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:41:24.521063 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516241    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:41:24.521063 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516243    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:41:24.521063 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516246    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:41:24.521063 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516248    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:41:24.521063 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516251    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:41:24.521063 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516254    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:41:24.521063 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516256    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:41:24.521063 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.516259    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:41:24.521705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516331    2564 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 21:41:24.521705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516366    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 21:41:24.521705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516373    2564 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 21:41:24.521705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516378    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 21:41:24.521705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516383    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 21:41:24.521705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516387    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 21:41:24.521705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516392    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 21:41:24.521705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516397    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 21:41:24.521705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516401    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 21:41:24.521705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516404    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 21:41:24.521705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516408    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 21:41:24.521705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516411    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 21:41:24.521705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516414    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 21:41:24.521705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516417    2564 flags.go:64] FLAG: --cgroup-root=""
Sep 17 21:41:24.521705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516421    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 21:41:24.521705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516424    2564 flags.go:64] FLAG: --client-ca-file=""
Sep 17 21:41:24.521705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516427    2564 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 21:41:24.521705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516430    2564 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 21:41:24.521705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516434    2564 flags.go:64] FLAG: --cluster-domain=""
Sep 17 21:41:24.521705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516437    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 21:41:24.521705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516441    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 21:41:24.521705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516444    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 21:41:24.521705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516447    2564 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 21:41:24.521705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516453    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516456    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516459    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516463    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516466    2564 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516469    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516472    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516475    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516478    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516482    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516485    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516488    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516491    2564 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516494    2564 flags.go:64] FLAG: --enable-server="true"
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516497    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516502    2564 flags.go:64] FLAG: --event-burst="100"
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516505    2564 flags.go:64] FLAG: --event-qps="50"
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516508    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516512    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516515    2564 flags.go:64] FLAG: --eviction-hard=""
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516518    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516521    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516524    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516529    2564 flags.go:64] FLAG: --eviction-soft=""
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516532    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 21:41:24.522305 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516535    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 21:41:24.522909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516538    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 21:41:24.522909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516541    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 21:41:24.522909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516543    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 21:41:24.522909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516546    2564 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 21:41:24.522909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516549    2564 flags.go:64] FLAG: --feature-gates=""
Sep 17 21:41:24.522909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516553    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 21:41:24.522909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516557    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 21:41:24.522909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516560    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 21:41:24.522909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516563    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 21:41:24.522909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516566    2564 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 21:41:24.522909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516569    2564 flags.go:64] FLAG: --help="false"
Sep 17 21:41:24.522909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516572    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-129-67.ec2.internal"
Sep 17 21:41:24.522909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516575    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 21:41:24.522909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516578    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 21:41:24.522909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516581    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 21:41:24.522909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516584    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 21:41:24.522909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516587    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 21:41:24.522909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516590    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 21:41:24.522909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516593    2564 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 21:41:24.522909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516596    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 21:41:24.522909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516598    2564 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 21:41:24.522909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516601    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 21:41:24.522909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516604    2564 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 21:41:24.522909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516607    2564 flags.go:64] FLAG: --kube-reserved=""
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516611    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516614    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516617    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516619    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516622    2564 flags.go:64] FLAG: --lock-file=""
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516628    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516631    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516634    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516640    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516643    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516646    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516649    2564 flags.go:64] FLAG: --logging-format="text"
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516652    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516655    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516659    2564 flags.go:64] FLAG: --manifest-url=""
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516662    2564 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516667    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516670    2564 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516674    2564 flags.go:64] FLAG: --max-pods="110"
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516677    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516680    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516683    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516686    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516689    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 21:41:24.523521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516692    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516695    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516702    2564 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516705    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516708    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516710    2564 flags.go:64] FLAG: --pod-cidr=""
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516713    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516719    2564 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516722    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516725    2564 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516728    2564 flags.go:64] FLAG: --port="10250"
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516731    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516734    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b3022451661be57b"
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516737    2564 flags.go:64] FLAG: --qos-reserved=""
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516740    2564 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516743    2564 flags.go:64] FLAG: --register-node="true"
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516746    2564 flags.go:64] FLAG: --register-with-taints=""
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516750    2564 flags.go:64] FLAG: --registry-burst="10"
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516752    2564 flags.go:64] FLAG: --registry-qps="5"
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516755    2564 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516758    2564 flags.go:64] FLAG: --reserved-memory=""
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516762    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516765    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516768    2564 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516771    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 21:41:24.524141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516774    2564 flags.go:64] FLAG: --runonce="false"
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516776    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516780    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516783    2564 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516786    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516788    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516791    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516794    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516797    2564 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516800    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516803    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516806    2564 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516808    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516811    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516814    2564 flags.go:64] FLAG: --system-cgroups=""
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516817    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516822    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516825    2564 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516828    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516832    2564 flags.go:64] FLAG: --tls-min-version=""
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516834    2564 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516837    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516840    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516843    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516846    2564 flags.go:64] FLAG: --v="2"
Sep 17 21:41:24.524936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516850    2564 flags.go:64] FLAG: --version="false"
Sep 17 21:41:24.525987 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516854    2564 flags.go:64] FLAG: --vmodule=""
Sep 17 21:41:24.525987 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516858    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 21:41:24.525987 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.516861    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 21:41:24.525987 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517015    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:41:24.525987 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517020    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:41:24.525987 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517023    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:41:24.525987 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517026    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:41:24.525987 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517029    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:41:24.525987 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517032    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:41:24.525987 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517034    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:41:24.525987 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517037    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:41:24.525987 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517040    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:41:24.525987 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517042    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:41:24.525987 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517045    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:41:24.525987 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517047    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:41:24.525987 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517050    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:41:24.525987 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517053    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:41:24.525987 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517055    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:41:24.525987 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517058    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:41:24.525987 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517061    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:41:24.526697 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517063    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:41:24.526697 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517065    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:41:24.526697 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517068    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:41:24.526697 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517070    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:41:24.526697 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517073    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:41:24.526697 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517076    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:41:24.526697 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517079    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:41:24.526697 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517082    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:41:24.526697 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517084    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:41:24.526697 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517087    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:41:24.526697 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517089    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:41:24.526697 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517092    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:41:24.526697 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517094    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:41:24.526697 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517097    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:41:24.526697 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517101    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:41:24.526697 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517105    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:41:24.526697 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517107    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:41:24.526697 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517111    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:41:24.526697 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517115    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:41:24.527183 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517118    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:41:24.527183 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517121    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:41:24.527183 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517124    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:41:24.527183 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517126    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:41:24.527183 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517129    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:41:24.527183 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517132    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:41:24.527183 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517134    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:41:24.527183 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517137    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:41:24.527183 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517140    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:41:24.527183 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517142    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:41:24.527183 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517144    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:41:24.527183 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517147    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:41:24.527183 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517149    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:41:24.527183 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517152    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:41:24.527183 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517156    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:41:24.527183 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517159    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:41:24.527183 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517162    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:41:24.527183 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517165    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:41:24.527183 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517182    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:41:24.527183 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517185    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:41:24.527671 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517188    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:41:24.527671 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517191    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:41:24.527671 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517194    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:41:24.527671 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517197    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:41:24.527671 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517200    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:41:24.527671 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517203    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:41:24.527671 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517206    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:41:24.527671 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517208    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:41:24.527671 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517213    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:41:24.527671 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517215    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:41:24.527671 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517218    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:41:24.527671 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517221    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:41:24.527671 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517223    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:41:24.527671 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517226    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:41:24.527671 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517228    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:41:24.527671 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517231    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:41:24.527671 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517234    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:41:24.527671 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517236    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:41:24.527671 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517239    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:41:24.528145 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517242    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:41:24.528145 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517244    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:41:24.528145 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517246    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:41:24.528145 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517249    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:41:24.528145 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517252    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:41:24.528145 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517254    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:41:24.528145 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517257    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:41:24.528145 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517259    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:41:24.528145 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517262    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:41:24.528145 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517264    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:41:24.528145 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517267    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:41:24.528145 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517269    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:41:24.528145 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517272    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:41:24.528145 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517276    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:41:24.528145 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517279    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:41:24.528145 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517281    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:41:24.528145 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517284    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:41:24.528145 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517286    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:41:24.528145 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517289    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:41:24.528145 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517292    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:41:24.528644 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517294    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:41:24.528644 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517298    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:41:24.528644 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517301    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:41:24.528644 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.517303    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:41:24.528644 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.517961    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 21:41:24.528644 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.527451    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 21:41:24.528644 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.527467    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 21:41:24.528644 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.528371    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 21:41:24.528644 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.528378    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 21:41:24.528644 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528441    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:41:24.528644 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528446    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:41:24.528644 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528451    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:41:24.528644 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528455    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:41:24.528644 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528458    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:41:24.529000 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528462    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:41:24.529000 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528464    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:41:24.529000 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528467    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:41:24.529000 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528470    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:41:24.529000 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528473    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:41:24.529000 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528476    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:41:24.529000 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528479    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:41:24.529000 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528481    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:41:24.529000 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528484    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:41:24.529000 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528486    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:41:24.529000 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528489    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:41:24.529000 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528491    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:41:24.529000 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528494    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:41:24.529000 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528497    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:41:24.529000 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528499    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:41:24.529000 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528502    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:41:24.529000 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528505    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:41:24.529000 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528507    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:41:24.529000 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528510    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:41:24.529507 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528513    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:41:24.529507 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528516    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:41:24.529507 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528519    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:41:24.529507 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528523    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:41:24.529507 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528531    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:41:24.529507 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528534    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:41:24.529507 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528537    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:41:24.529507 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528540    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:41:24.529507 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528542    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:41:24.529507 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528545    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:41:24.529507 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528547    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:41:24.529507 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528550    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:41:24.529507 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528553    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:41:24.529507 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528555    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:41:24.529507 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528558    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:41:24.529507 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528560    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:41:24.529507 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528562    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:41:24.529507 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528565    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:41:24.529507 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528568    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:41:24.529981 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528571    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:41:24.529981 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528574    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:41:24.529981 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528576    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:41:24.529981 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528578    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:41:24.529981 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528581    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:41:24.529981 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528583    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:41:24.529981 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528586    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:41:24.529981 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528588    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:41:24.529981 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528591    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:41:24.529981 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528593    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:41:24.529981 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528596    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:41:24.529981 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528599    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:41:24.529981 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528601    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:41:24.529981 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528603    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:41:24.529981 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528606    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:41:24.529981 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528608    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:41:24.529981 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528612    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:41:24.529981 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528615    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:41:24.529981 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528617    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:41:24.529981 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528620    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:41:24.530492 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528623    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:41:24.530492 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528625    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:41:24.530492 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528628    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:41:24.530492 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528630    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:41:24.530492 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528633    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:41:24.530492 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528635    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:41:24.530492 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528638    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:41:24.530492 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528641    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:41:24.530492 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528645    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:41:24.530492 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528648    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:41:24.530492 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528651    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:41:24.530492 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528654    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:41:24.530492 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528656    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:41:24.530492 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528659    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:41:24.530492 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528661    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:41:24.530492 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528664    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:41:24.530492 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528667    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:41:24.530492 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528670    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:41:24.530492 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528672    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:41:24.530492 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528674    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:41:24.531014 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528677    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:41:24.531014 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528679    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:41:24.531014 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528682    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:41:24.531014 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528684    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:41:24.531014 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528687    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:41:24.531014 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528689    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:41:24.531014 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528692    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:41:24.531014 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528694    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:41:24.531014 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528697    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:41:24.531014 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528700    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:41:24.531014 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528703    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:41:24.531014 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528706    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:41:24.531014 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528709    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:41:24.531014 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528712    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:41:24.531014 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528714    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:41:24.531014 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528717    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:41:24.531442 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.528722    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 21:41:24.531442 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528839    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:41:24.531442 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528844    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:41:24.531442 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528848    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:41:24.531442 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528851    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:41:24.531442 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528854    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:41:24.531442 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528857    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:41:24.531442 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528859    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:41:24.531442 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528862    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:41:24.531442 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528865    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:41:24.531442 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528868    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:41:24.531442 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528870    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:41:24.531442 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528873    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:41:24.531442 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528876    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:41:24.531787 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528879    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:41:24.531787 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528881    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:41:24.531787 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528884    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:41:24.531787 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528886    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:41:24.531787 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528888    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:41:24.531787 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528891    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:41:24.531787 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528893    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:41:24.531787 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528896    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:41:24.531787 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528898    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:41:24.531787 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528901    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:41:24.531787 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528903    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:41:24.531787 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528906    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:41:24.531787 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528909    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:41:24.531787 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528911    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:41:24.531787 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528914    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:41:24.531787 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528916    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:41:24.531787 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528919    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:41:24.531787 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528921    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:41:24.531787 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528924    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:41:24.532310 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528926    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:41:24.532310 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528929    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:41:24.532310 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528931    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:41:24.532310 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528934    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:41:24.532310 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528938    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:41:24.532310 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528941    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:41:24.532310 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528944    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:41:24.532310 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528947    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:41:24.532310 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528950    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:41:24.532310 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528952    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:41:24.532310 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528955    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:41:24.532310 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528957    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:41:24.532310 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528960    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:41:24.532310 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528963    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:41:24.532310 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528966    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:41:24.532310 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528968    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:41:24.532310 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528971    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:41:24.532310 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528973    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:41:24.532310 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528975    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:41:24.532770 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528978    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:41:24.532770 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528980    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:41:24.532770 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528983    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:41:24.532770 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528985    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:41:24.532770 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528987    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:41:24.532770 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528990    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:41:24.532770 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528993    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:41:24.532770 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528996    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:41:24.532770 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.528998    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:41:24.532770 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529001    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:41:24.532770 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529003    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:41:24.532770 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529006    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:41:24.532770 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529009    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:41:24.532770 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529011    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:41:24.532770 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529014    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:41:24.532770 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529016    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:41:24.532770 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529018    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:41:24.532770 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529021    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:41:24.532770 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529025    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:41:24.532770 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529028    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:41:24.533272 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529031    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:41:24.533272 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529033    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:41:24.533272 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529036    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:41:24.533272 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529039    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:41:24.533272 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529041    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:41:24.533272 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529044    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:41:24.533272 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529046    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:41:24.533272 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529049    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:41:24.533272 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529052    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:41:24.533272 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529055    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:41:24.533272 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529057    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:41:24.533272 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529060    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:41:24.533272 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529062    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:41:24.533272 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529064    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:41:24.533272 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529067    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:41:24.533272 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529069    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:41:24.533272 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529072    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:41:24.533272 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529078    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:41:24.533272 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529080    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:41:24.533272 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529083    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:41:24.533750 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529086    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:41:24.533750 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529088    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:41:24.533750 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529091    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:41:24.533750 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529094    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:41:24.533750 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529096    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:41:24.533750 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529099    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:41:24.533750 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529102    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:41:24.533750 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:24.529104    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:41:24.533750 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.529109    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 21:41:24.533750 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.529238    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 21:41:24.534311 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.534296    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 21:41:24.535417 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.535405    2564 server.go:1019] "Starting client certificate rotation"
Sep 17 21:41:24.535529 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.535511    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 21:41:24.535563 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.535546    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 21:41:24.538298 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.538284    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 21:41:24.539465 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.539448    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 21:41:24.549800 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.549785    2564 log.go:25] "Validated CRI v1 runtime API"
Sep 17 21:41:24.551696 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.551681    2564 log.go:25] "Validated CRI v1 image API"
Sep 17 21:41:24.553367 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.553352    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 21:41:24.557500 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.557468    2564 fs.go:135] Filesystem UUIDs: map[7164693d-60b0-41af-92fa-9befbb821d3d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cc7ce436-39c7-48fe-9c44-18b1fbc07518:/dev/nvme0n1p3]
Sep 17 21:41:24.557588 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.557491    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 21:41:24.562598 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.562482    2564 manager.go:216] Machine: {Timestamp:2026-09-17 21:41:24.561128601 +0000 UTC m=+0.379454903 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099687 MemoryCapacity:32812146688 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26a7044626fbf5c3da6a1d46549c04 SystemUUID:ec26a704-4626-fbf5-c3da-6a1d46549c04 BootID:10e1bf88-2833-49ff-8c7f-24113f873240 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406073344 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f1:99:e3:e5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f1:99:e3:e5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:c5:8d:5f:ad:20 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812146688 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 21:41:24.562598 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.562586    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 21:41:24.563846 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.563401    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 21:41:24.568041 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.567360    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 21:41:24.568238 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.568047    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-67.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 21:41:24.569651 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.569639    2564 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 21:41:24.569686 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.569654    2564 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 21:41:24.569686 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.569662    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 21:41:24.570002 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.569984    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 21:41:24.570379 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.570366    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 21:41:24.572237 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.572226    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 21:41:24.575856 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.575844    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 21:41:24.575978 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.575969    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 21:41:24.576019 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.576000    2564 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 21:41:24.576019 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.576012    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 21:41:24.576076 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.576029    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 21:41:24.576076 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.576040    2564 kubelet.go:398] "Adding apiserver pod source"
Sep 17 21:41:24.576076 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.576049    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 21:41:24.576436 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.576423    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 21:41:24.576480 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.576445    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 21:41:24.578817 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.578798    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 21:41:24.581947 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.581934    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 21:41:24.581990 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.581974    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 21:41:24.583480 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.583468    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 21:41:24.583514 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.583486    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 21:41:24.583514 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.583492    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 21:41:24.583514 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.583500    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 21:41:24.583602 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.583522    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 21:41:24.583602 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.583530    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 21:41:24.583602 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.583548    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 21:41:24.583602 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.583556    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 21:41:24.583602 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.583566    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 21:41:24.583602 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.583573    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 21:41:24.583602 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.583585    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 21:41:24.583602 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.583594    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 21:41:24.584535 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.584521    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 21:41:24.584566 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.584539    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 21:41:24.588262 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.588226    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-67.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 21:41:24.588473 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.588461    2564 server.go:1268] "Started kubelet"
Sep 17 21:41:24.588589 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.588547    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 21:41:24.588668 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.588623    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 21:41:24.588704 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.588694    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 21:41:24.589050 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:24.589030    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 21:41:24.589106 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:24.589032    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-67.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 21:41:24.589304 ip-10-0-129-67 systemd[1]: Started Kubernetes Kubelet.
Sep 17 21:41:24.590255 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.590237    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 21:41:24.590766 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.590751    2564 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 21:41:24.593506 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:24.593469    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 21:41:24.595959 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:24.594801    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-67.ec2.internal.18d639a56d0efe77  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-67.ec2.internal,UID:ip-10-0-129-67.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-67.ec2.internal,},FirstTimestamp:2026-09-17 21:41:24.588437111 +0000 UTC m=+0.406763414,LastTimestamp:2026-09-17 21:41:24.588437111 +0000 UTC m=+0.406763414,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-67.ec2.internal,}"
Sep 17 21:41:24.603139 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.603122    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 21:41:24.603812 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.603794    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 21:41:24.603925 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:24.603908    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-67.ec2.internal\" not found"
Sep 17 21:41:24.604669 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.604654    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 21:41:24.604669 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.604658    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 21:41:24.604800 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.604679    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 21:41:24.604800 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.604786    2564 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 21:41:24.604800 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.604796    2564 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 21:41:24.605538 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.605523    2564 factory.go:55] Registering systemd factory
Sep 17 21:41:24.605598 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.605569    2564 factory.go:223] Registration of the systemd container factory successfully
Sep 17 21:41:24.605805 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.605791    2564 factory.go:153] Registering CRI-O factory
Sep 17 21:41:24.605877 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.605808    2564 factory.go:223] Registration of the crio container factory successfully
Sep 17 21:41:24.605877 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.605869    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 21:41:24.605964 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.605888    2564 factory.go:103] Registering Raw factory
Sep 17 21:41:24.605964 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.605905    2564 manager.go:1196] Started watching for new ooms in manager
Sep 17 21:41:24.607092 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.607077    2564 manager.go:349] Starting recovery of all containers
Sep 17 21:41:24.607871 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:24.607850    2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-67.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 21:41:24.607968 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:24.607938    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 21:41:24.613749 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.613709    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-42m59"
Sep 17 21:41:24.618344 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.618314    2564 manager.go:354] Recovery completed
Sep 17 21:41:24.622632 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.622614    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:41:24.625492 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.625469    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-42m59"
Sep 17 21:41:24.625578 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.625470    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:41:24.625578 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.625519    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:41:24.625578 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.625529    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:41:24.626083 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.626067    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 21:41:24.626083 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.626082    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 21:41:24.626249 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.626100    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 21:41:24.626249 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.626120    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:41:24.627876 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.627856    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:41:24.627965 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.627884    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:41:24.627965 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.627898    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:41:24.628459 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:24.628352    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-67.ec2.internal.18d639a56f449b20  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-67.ec2.internal,UID:ip-10-0-129-67.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-67.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-67.ec2.internal,},FirstTimestamp:2026-09-17 21:41:24.625505056 +0000 UTC m=+0.443831359,LastTimestamp:2026-09-17 21:41:24.625505056 +0000 UTC m=+0.443831359,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-67.ec2.internal,}"
Sep 17 21:41:24.628841 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.628824    2564 policy_none.go:49] "None policy: Start"
Sep 17 21:41:24.628841 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.628843    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 21:41:24.628946 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.628853    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 21:41:24.629781 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.629770    2564 policy_none.go:47] "Start"
Sep 17 21:41:24.676625 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.676608    2564 manager.go:337] "Starting Device Plugin manager"
Sep 17 21:41:24.676739 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:24.676647    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 21:41:24.676739 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.676659    2564 server.go:87] "Starting device plugin registration server"
Sep 17 21:41:24.676956 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.676943    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 21:41:24.676998 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.676961    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 21:41:24.677050 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.677032    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 21:41:24.677110 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.677099    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 21:41:24.677153 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.677112    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 21:41:24.677728 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:24.677707    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 21:41:24.677826 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:24.677760    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-67.ec2.internal\" not found"
Sep 17 21:41:24.745539 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.745496    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 21:41:24.746892 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.746870    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 21:41:24.746999 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.746940    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 21:41:24.746999 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.746971    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 21:41:24.747089 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:24.747019    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 21:41:24.749759 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.749739    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:41:24.777703 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.777625    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:41:24.778676 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.778651    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:41:24.778786 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.778692    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:41:24.778786 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.778708    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:41:24.778786 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.778732    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-67.ec2.internal"
Sep 17 21:41:24.785084 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.785068    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-67.ec2.internal"
Sep 17 21:41:24.785165 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:24.785090    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-67.ec2.internal\": node \"ip-10-0-129-67.ec2.internal\" not found"
Sep 17 21:41:24.805778 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:24.805748    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-67.ec2.internal\" not found"
Sep 17 21:41:24.847569 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.847536    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-67.ec2.internal"]
Sep 17 21:41:24.847732 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.847619    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:41:24.849196 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.849162    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:41:24.849313 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.849211    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:41:24.849313 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.849225    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:41:24.850652 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.850634    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:41:24.850752 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.850739    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal"
Sep 17 21:41:24.850801 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.850769    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:41:24.851367 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.851350    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:41:24.851467 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.851378    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:41:24.851467 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.851392    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:41:24.851467 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.851404    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:41:24.851467 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.851421    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:41:24.851467 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.851433    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:41:24.852514 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.852499    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-67.ec2.internal"
Sep 17 21:41:24.852580 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.852528    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:41:24.853220 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.853206    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:41:24.853296 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.853229    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:41:24.853296 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.853239    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:41:24.881874 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:24.881849    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-67.ec2.internal\" not found" node="ip-10-0-129-67.ec2.internal"
Sep 17 21:41:24.886127 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:24.886110    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-67.ec2.internal\" not found" node="ip-10-0-129-67.ec2.internal"
Sep 17 21:41:24.906469 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:24.906446    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-67.ec2.internal\" not found"
Sep 17 21:41:24.906588 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.906515    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/69be7f4fd498a52ca0b57b8db0dca68c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal\" (UID: \"69be7f4fd498a52ca0b57b8db0dca68c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal"
Sep 17 21:41:24.906588 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.906540    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69be7f4fd498a52ca0b57b8db0dca68c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal\" (UID: \"69be7f4fd498a52ca0b57b8db0dca68c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal"
Sep 17 21:41:24.906588 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:24.906558    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c19c959590466136e44fe97bd5d57e26-config\") pod \"kube-apiserver-proxy-ip-10-0-129-67.ec2.internal\" (UID: \"c19c959590466136e44fe97bd5d57e26\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-67.ec2.internal"
Sep 17 21:41:25.007105 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:25.007052    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-67.ec2.internal\" not found"
Sep 17 21:41:25.007105 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:25.007086    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/69be7f4fd498a52ca0b57b8db0dca68c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal\" (UID: \"69be7f4fd498a52ca0b57b8db0dca68c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal"
Sep 17 21:41:25.007105 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:25.007114    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69be7f4fd498a52ca0b57b8db0dca68c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal\" (UID: \"69be7f4fd498a52ca0b57b8db0dca68c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal"
Sep 17 21:41:25.007375 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:25.007129    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c19c959590466136e44fe97bd5d57e26-config\") pod \"kube-apiserver-proxy-ip-10-0-129-67.ec2.internal\" (UID: \"c19c959590466136e44fe97bd5d57e26\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-67.ec2.internal"
Sep 17 21:41:25.007375 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:25.007197    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/69be7f4fd498a52ca0b57b8db0dca68c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal\" (UID: \"69be7f4fd498a52ca0b57b8db0dca68c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal"
Sep 17 21:41:25.007375 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:25.007225    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69be7f4fd498a52ca0b57b8db0dca68c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal\" (UID: \"69be7f4fd498a52ca0b57b8db0dca68c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal"
Sep 17 21:41:25.007375 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:25.007207    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c19c959590466136e44fe97bd5d57e26-config\") pod \"kube-apiserver-proxy-ip-10-0-129-67.ec2.internal\" (UID: \"c19c959590466136e44fe97bd5d57e26\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-67.ec2.internal"
Sep 17 21:41:25.107949 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:25.107880    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-67.ec2.internal\" not found"
Sep 17 21:41:25.185136 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:25.185105    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal"
Sep 17 21:41:25.189703 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:25.189099    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-67.ec2.internal"
Sep 17 21:41:25.208963 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:25.208940    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-67.ec2.internal\" not found"
Sep 17 21:41:25.309347 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:25.309312    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-67.ec2.internal\" not found"
Sep 17 21:41:25.409766 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:25.409746    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-67.ec2.internal\" not found"
Sep 17 21:41:25.510187 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:25.510151    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-67.ec2.internal\" not found"
Sep 17 21:41:25.536338 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:25.536306    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 21:41:25.536491 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:25.536474    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 21:41:25.610455 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:25.610421    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-67.ec2.internal\" not found"
Sep 17 21:41:25.617670 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:25.617649    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:41:25.630917 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:25.630884    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 21:36:25 +0000 UTC" deadline="2028-03-08 07:00:11.14126786 +0000 UTC"
Sep 17 21:41:25.630917 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:25.630913    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12897h18m45.510357964s"
Sep 17 21:41:25.701777 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:25.701707    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:41:25.711291 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:25.711263    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-67.ec2.internal\" not found"
Sep 17 21:41:25.812103 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:25.812070    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-67.ec2.internal\" not found"
Sep 17 21:41:25.878852 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:25.878817    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19c959590466136e44fe97bd5d57e26.slice/crio-6f70f5f842b13f4f13cc57a8f6e65c00ddccff70dd4d3ece015bd1bd521e7705 WatchSource:0}: Error finding container 6f70f5f842b13f4f13cc57a8f6e65c00ddccff70dd4d3ece015bd1bd521e7705: Status 404 returned error can't find the container with id 6f70f5f842b13f4f13cc57a8f6e65c00ddccff70dd4d3ece015bd1bd521e7705
Sep 17 21:41:25.879115 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:25.879098    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69be7f4fd498a52ca0b57b8db0dca68c.slice/crio-6b2305736d6bc1a48a80736ee1a10ea7f83e578bfcf47354a433b58076144aab WatchSource:0}: Error finding container 6b2305736d6bc1a48a80736ee1a10ea7f83e578bfcf47354a433b58076144aab: Status 404 returned error can't find the container with id 6b2305736d6bc1a48a80736ee1a10ea7f83e578bfcf47354a433b58076144aab
Sep 17 21:41:25.883544 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:25.883527    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:41:25.907573 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:25.907551    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:41:26.008803 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.008699    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal"
Sep 17 21:41:26.027809 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.027783    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 21:41:26.028582 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.028568    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-67.ec2.internal"
Sep 17 21:41:26.040955 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.040929    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 21:41:26.577707 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.577671    2564 apiserver.go:52] "Watching apiserver"
Sep 17 21:41:26.587570 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.587537    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 21:41:26.588053 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.588027    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-67.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk","openshift-cluster-node-tuning-operator/tuned-j4fpb","openshift-dns/node-resolver-fdkhb","openshift-image-registry/node-ca-rkgql","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal","openshift-multus/multus-additional-cni-plugins-kjjz4","openshift-multus/multus-hxcs4","openshift-multus/network-metrics-daemon-5gxhr","openshift-network-diagnostics/network-check-target-hmscl","openshift-network-operator/iptables-alerter-p4lhk","openshift-ovn-kubernetes/ovnkube-node-zdcnw","kube-system/global-pull-secret-syncer-hrbr6","kube-system/konnectivity-agent-7ngpc"]
Sep 17 21:41:26.590728 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.590678    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kjjz4"
Sep 17 21:41:26.593430 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.593405    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:26.593512 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:26.593486    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrbr6" podUID="1e76c503-ec71-47a9-b3cd-63d3b12a439a"
Sep 17 21:41:26.594116 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.594095    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hn5gq\""
Sep 17 21:41:26.594371 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.594348    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 21:41:26.594445 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.594395    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 21:41:26.594552 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.594536    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 21:41:26.594677 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.594614    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 21:41:26.594755 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.594348    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 21:41:26.595510 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.595489    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.597809 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.597665    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fdkhb"
Sep 17 21:41:26.598084 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.598063    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:41:26.599289 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.599267    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 21:41:26.599549 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.599530    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-j6cpg\""
Sep 17 21:41:26.600157 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.599987    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rkgql"
Sep 17 21:41:26.600397 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.600378    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 21:41:26.600634 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.600619    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-k8cqt\""
Sep 17 21:41:26.600689 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.600680    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 21:41:26.602550 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.602459    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 21:41:26.603554 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.602894    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7ngpc"
Sep 17 21:41:26.603554 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.603041    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bpgs5\""
Sep 17 21:41:26.603554 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.603353    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 21:41:26.605458 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.603991    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 21:41:26.605458 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.604036    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 21:41:26.605458 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.605453    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:26.605706 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:26.605532    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gxhr" podUID="93119bca-cc51-4b33-964b-63bf4fe5c9e1"
Sep 17 21:41:26.607384 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.607359    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.607841 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.607805    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4kz2f\""
Sep 17 21:41:26.607986 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.607972    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 21:41:26.608233 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.608209    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 21:41:26.609999 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.609980    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 21:41:26.610247 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.610229    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xcdzn\""
Sep 17 21:41:26.610789 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.610766    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk"
Sep 17 21:41:26.612523 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.612502    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:26.612620 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:26.612577    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmscl" podUID="5fd48c9a-8cd8-4743-8dbf-eb20219c893c"
Sep 17 21:41:26.620446 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.614797    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 21:41:26.620446 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.615151    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n8xcc\""
Sep 17 21:41:26.620446 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.615436    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 21:41:26.620446 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.615435    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 21:41:26.620446 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.616143    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p4lhk"
Sep 17 21:41:26.620446 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.617824    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59d7a806-c3e4-48f4-82df-8bfaa212a9db-system-cni-dir\") pod \"multus-additional-cni-plugins-kjjz4\" (UID: \"59d7a806-c3e4-48f4-82df-8bfaa212a9db\") " pod="openshift-multus/multus-additional-cni-plugins-kjjz4"
Sep 17 21:41:26.620446 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.617853    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59115160-02ac-43e8-9a62-0c91775ba337-tmp-dir\") pod \"node-resolver-fdkhb\" (UID: \"59115160-02ac-43e8-9a62-0c91775ba337\") " pod="openshift-dns/node-resolver-fdkhb"
Sep 17 21:41:26.620446 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.617874    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ctcb\" (UniqueName: \"kubernetes.io/projected/59115160-02ac-43e8-9a62-0c91775ba337-kube-api-access-2ctcb\") pod \"node-resolver-fdkhb\" (UID: \"59115160-02ac-43e8-9a62-0c91775ba337\") " pod="openshift-dns/node-resolver-fdkhb"
Sep 17 21:41:26.620446 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.617907    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d49ed8ac-69ff-4b6e-ad3a-3f45026b84ae-konnectivity-ca\") pod \"konnectivity-agent-7ngpc\" (UID: \"d49ed8ac-69ff-4b6e-ad3a-3f45026b84ae\") " pod="kube-system/konnectivity-agent-7ngpc"
Sep 17 21:41:26.620446 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.617942    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8d2b64b-06cf-4a62-b73f-6835eab58938-cni-binary-copy\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.620446 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.617974    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-host-var-lib-cni-multus\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.620446 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618006    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59d7a806-c3e4-48f4-82df-8bfaa212a9db-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kjjz4\" (UID: \"59d7a806-c3e4-48f4-82df-8bfaa212a9db\") " pod="openshift-multus/multus-additional-cni-plugins-kjjz4"
Sep 17 21:41:26.620446 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618040    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-etc-sysctl-conf\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.620446 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618073    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-multus-cni-dir\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.620446 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618100    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59d7a806-c3e4-48f4-82df-8bfaa212a9db-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kjjz4\" (UID: \"59d7a806-c3e4-48f4-82df-8bfaa212a9db\") " pod="openshift-multus/multus-additional-cni-plugins-kjjz4"
Sep 17 21:41:26.620446 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618123    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e76c503-ec71-47a9-b3cd-63d3b12a439a-original-pull-secret\") pod \"global-pull-secret-syncer-hrbr6\" (UID: \"1e76c503-ec71-47a9-b3cd-63d3b12a439a\") " pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:26.620446 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618143    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-etc-modprobe-d\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.620446 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618161    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-etc-sysctl-d\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.620446 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618190    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-run\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.621505 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618219    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-lib-modules\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.621505 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618270    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-host\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.621505 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618304    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-cnibin\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.621505 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618340    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59d7a806-c3e4-48f4-82df-8bfaa212a9db-cni-binary-copy\") pod \"multus-additional-cni-plugins-kjjz4\" (UID: \"59d7a806-c3e4-48f4-82df-8bfaa212a9db\") " pod="openshift-multus/multus-additional-cni-plugins-kjjz4"
Sep 17 21:41:26.621505 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618378    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-var-lib-kubelet\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.621505 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618441    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49f3497d-f458-44e9-8846-92afb8ba26eb-serviceca\") pod \"node-ca-rkgql\" (UID: \"49f3497d-f458-44e9-8846-92afb8ba26eb\") " pod="openshift-image-registry/node-ca-rkgql"
Sep 17 21:41:26.621505 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618474    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h8ks\" (UniqueName: \"kubernetes.io/projected/49f3497d-f458-44e9-8846-92afb8ba26eb-kube-api-access-5h8ks\") pod \"node-ca-rkgql\" (UID: \"49f3497d-f458-44e9-8846-92afb8ba26eb\") " pod="openshift-image-registry/node-ca-rkgql"
Sep 17 21:41:26.621505 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618506    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d49ed8ac-69ff-4b6e-ad3a-3f45026b84ae-agent-certs\") pod \"konnectivity-agent-7ngpc\" (UID: \"d49ed8ac-69ff-4b6e-ad3a-3f45026b84ae\") " pod="kube-system/konnectivity-agent-7ngpc"
Sep 17 21:41:26.621505 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618545    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-system-cni-dir\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.621505 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618640    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-multus-socket-dir-parent\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.621505 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618668    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-host-var-lib-kubelet\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.621505 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618699    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-etc-kubernetes\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.621505 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618729    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-sys\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.621505 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618767    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-multus-conf-dir\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.621505 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618782    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.621505 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618842    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a8d2b64b-06cf-4a62-b73f-6835eab58938-multus-daemon-config\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.621505 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618896    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59115160-02ac-43e8-9a62-0c91775ba337-hosts-file\") pod \"node-resolver-fdkhb\" (UID: \"59115160-02ac-43e8-9a62-0c91775ba337\") " pod="openshift-dns/node-resolver-fdkhb"
Sep 17 21:41:26.622434 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618933    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-host-run-k8s-cni-cncf-io\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.622434 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.618984    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-host-var-lib-cni-bin\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.622434 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.619019    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e76c503-ec71-47a9-b3cd-63d3b12a439a-dbus\") pod \"global-pull-secret-syncer-hrbr6\" (UID: \"1e76c503-ec71-47a9-b3cd-63d3b12a439a\") " pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:26.622434 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.619054    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-etc-sysconfig\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.622434 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.619090    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-etc-kubernetes\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.622434 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.619127    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-host-run-netns\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.622434 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.619153    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-hostroot\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.622434 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.619203    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlxst\" (UniqueName: \"kubernetes.io/projected/a8d2b64b-06cf-4a62-b73f-6835eab58938-kube-api-access-wlxst\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.622434 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.619239    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pmp5\" (UniqueName: \"kubernetes.io/projected/59d7a806-c3e4-48f4-82df-8bfaa212a9db-kube-api-access-2pmp5\") pod \"multus-additional-cni-plugins-kjjz4\" (UID: \"59d7a806-c3e4-48f4-82df-8bfaa212a9db\") " pod="openshift-multus/multus-additional-cni-plugins-kjjz4"
Sep 17 21:41:26.622434 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.619276    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-tmp\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.622434 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.619305    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k288\" (UniqueName: \"kubernetes.io/projected/93119bca-cc51-4b33-964b-63bf4fe5c9e1-kube-api-access-2k288\") pod \"network-metrics-daemon-5gxhr\" (UID: \"93119bca-cc51-4b33-964b-63bf4fe5c9e1\") " pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:26.622434 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.619339    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49f3497d-f458-44e9-8846-92afb8ba26eb-host\") pod \"node-ca-rkgql\" (UID: \"49f3497d-f458-44e9-8846-92afb8ba26eb\") " pod="openshift-image-registry/node-ca-rkgql"
Sep 17 21:41:26.622434 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.619369    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59d7a806-c3e4-48f4-82df-8bfaa212a9db-cnibin\") pod \"multus-additional-cni-plugins-kjjz4\" (UID: \"59d7a806-c3e4-48f4-82df-8bfaa212a9db\") " pod="openshift-multus/multus-additional-cni-plugins-kjjz4"
Sep 17 21:41:26.622434 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.619402    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59d7a806-c3e4-48f4-82df-8bfaa212a9db-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kjjz4\" (UID: \"59d7a806-c3e4-48f4-82df-8bfaa212a9db\") " pod="openshift-multus/multus-additional-cni-plugins-kjjz4"
Sep 17 21:41:26.622434 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.619434    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-etc-systemd\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.622434 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.619461    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-etc-tuned\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.623230 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.619495    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffkl8\" (UniqueName: \"kubernetes.io/projected/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-kube-api-access-ffkl8\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.623230 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.619528    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93119bca-cc51-4b33-964b-63bf4fe5c9e1-metrics-certs\") pod \"network-metrics-daemon-5gxhr\" (UID: \"93119bca-cc51-4b33-964b-63bf4fe5c9e1\") " pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:26.623230 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.619572    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-host-run-multus-certs\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.623230 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.619607    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e76c503-ec71-47a9-b3cd-63d3b12a439a-kubelet-config\") pod \"global-pull-secret-syncer-hrbr6\" (UID: \"1e76c503-ec71-47a9-b3cd-63d3b12a439a\") " pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:26.623230 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.620389    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:41:26.623230 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.620442    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 21:41:26.623230 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.620565    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9mql7\""
Sep 17 21:41:26.623230 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.620694    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 21:41:26.627939 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.626459    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 21:41:26.627939 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.626680    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 21:41:26.627939 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.626856    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 21:41:26.627939 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.627041    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7j2c4\""
Sep 17 21:41:26.627939 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.627201    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 21:41:26.627939 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.627307    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 21:41:26.627939 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.627585    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 21:41:26.627939 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.627776    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 21:41:26.652464 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.652433    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-v44rl"
Sep 17 21:41:26.661185 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.661162    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v44rl"
Sep 17 21:41:26.706223 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.706192    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 21:41:26.719844 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.719812    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-host-slash\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.719994 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.719862    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkd7w\" (UniqueName: \"kubernetes.io/projected/56fa7d57-fa08-457e-8a11-ca93979cdc8a-kube-api-access-fkd7w\") pod \"aws-ebs-csi-driver-node-778pk\" (UID: \"56fa7d57-fa08-457e-8a11-ca93979cdc8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk"
Sep 17 21:41:26.719994 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.719900    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59115160-02ac-43e8-9a62-0c91775ba337-tmp-dir\") pod \"node-resolver-fdkhb\" (UID: \"59115160-02ac-43e8-9a62-0c91775ba337\") " pod="openshift-dns/node-resolver-fdkhb"
Sep 17 21:41:26.719994 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.719936    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ctcb\" (UniqueName: \"kubernetes.io/projected/59115160-02ac-43e8-9a62-0c91775ba337-kube-api-access-2ctcb\") pod \"node-resolver-fdkhb\" (UID: \"59115160-02ac-43e8-9a62-0c91775ba337\") " pod="openshift-dns/node-resolver-fdkhb"
Sep 17 21:41:26.719994 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.719963    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d49ed8ac-69ff-4b6e-ad3a-3f45026b84ae-konnectivity-ca\") pod \"konnectivity-agent-7ngpc\" (UID: \"d49ed8ac-69ff-4b6e-ad3a-3f45026b84ae\") " pod="kube-system/konnectivity-agent-7ngpc"
Sep 17 21:41:26.719994 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.719989    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8d2b64b-06cf-4a62-b73f-6835eab58938-cni-binary-copy\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.720322 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720025    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-host-var-lib-cni-multus\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.720322 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720050    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59d7a806-c3e4-48f4-82df-8bfaa212a9db-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kjjz4\" (UID: \"59d7a806-c3e4-48f4-82df-8bfaa212a9db\") " pod="openshift-multus/multus-additional-cni-plugins-kjjz4"
Sep 17 21:41:26.720322 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720076    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-run\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.720322 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720106    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-run-systemd\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.720322 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720134    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-etc-sysctl-d\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.720322 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720159    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbqmn\" (UniqueName: \"kubernetes.io/projected/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-kube-api-access-vbqmn\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.720322 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720186    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59d7a806-c3e4-48f4-82df-8bfaa212a9db-cni-binary-copy\") pod \"multus-additional-cni-plugins-kjjz4\" (UID: \"59d7a806-c3e4-48f4-82df-8bfaa212a9db\") " pod="openshift-multus/multus-additional-cni-plugins-kjjz4"
Sep 17 21:41:26.720322 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720237    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-var-lib-kubelet\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.720322 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720264    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h8ks\" (UniqueName: \"kubernetes.io/projected/49f3497d-f458-44e9-8846-92afb8ba26eb-kube-api-access-5h8ks\") pod \"node-ca-rkgql\" (UID: \"49f3497d-f458-44e9-8846-92afb8ba26eb\") " pod="openshift-image-registry/node-ca-rkgql"
Sep 17 21:41:26.720322 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720271    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59115160-02ac-43e8-9a62-0c91775ba337-tmp-dir\") pod \"node-resolver-fdkhb\" (UID: \"59115160-02ac-43e8-9a62-0c91775ba337\") " pod="openshift-dns/node-resolver-fdkhb"
Sep 17 21:41:26.720322 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720290    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d49ed8ac-69ff-4b6e-ad3a-3f45026b84ae-agent-certs\") pod \"konnectivity-agent-7ngpc\" (UID: \"d49ed8ac-69ff-4b6e-ad3a-3f45026b84ae\") " pod="kube-system/konnectivity-agent-7ngpc"
Sep 17 21:41:26.720322 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720318    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-multus-socket-dir-parent\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.720922 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720343    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/56fa7d57-fa08-457e-8a11-ca93979cdc8a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-778pk\" (UID: \"56fa7d57-fa08-457e-8a11-ca93979cdc8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk"
Sep 17 21:41:26.720922 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720368    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a8d2b64b-06cf-4a62-b73f-6835eab58938-multus-daemon-config\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.720922 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720392    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59115160-02ac-43e8-9a62-0c91775ba337-hosts-file\") pod \"node-resolver-fdkhb\" (UID: \"59115160-02ac-43e8-9a62-0c91775ba337\") " pod="openshift-dns/node-resolver-fdkhb"
Sep 17 21:41:26.720922 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720414    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-host-var-lib-cni-bin\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.720922 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720437    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e76c503-ec71-47a9-b3cd-63d3b12a439a-dbus\") pod \"global-pull-secret-syncer-hrbr6\" (UID: \"1e76c503-ec71-47a9-b3cd-63d3b12a439a\") " pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:26.720922 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720459    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-etc-kubernetes\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.720922 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720483    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-run-ovn\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.720922 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720507    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-host-run-ovn-kubernetes\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.720922 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720531    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/56fa7d57-fa08-457e-8a11-ca93979cdc8a-socket-dir\") pod \"aws-ebs-csi-driver-node-778pk\" (UID: \"56fa7d57-fa08-457e-8a11-ca93979cdc8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk"
Sep 17 21:41:26.720922 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720557    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-hostroot\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.720922 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720601    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlxst\" (UniqueName: \"kubernetes.io/projected/a8d2b64b-06cf-4a62-b73f-6835eab58938-kube-api-access-wlxst\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.720922 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720627    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pmp5\" (UniqueName: \"kubernetes.io/projected/59d7a806-c3e4-48f4-82df-8bfaa212a9db-kube-api-access-2pmp5\") pod \"multus-additional-cni-plugins-kjjz4\" (UID: \"59d7a806-c3e4-48f4-82df-8bfaa212a9db\") " pod="openshift-multus/multus-additional-cni-plugins-kjjz4"
Sep 17 21:41:26.720922 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720660    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-host-cni-netd\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.720922 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720684    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/56fa7d57-fa08-457e-8a11-ca93979cdc8a-etc-selinux\") pod \"aws-ebs-csi-driver-node-778pk\" (UID: \"56fa7d57-fa08-457e-8a11-ca93979cdc8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk"
Sep 17 21:41:26.720922 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720725    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93119bca-cc51-4b33-964b-63bf4fe5c9e1-metrics-certs\") pod \"network-metrics-daemon-5gxhr\" (UID: \"93119bca-cc51-4b33-964b-63bf4fe5c9e1\") " pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:26.720922 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720774    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-host-run-netns\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.720922 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720799    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-etc-openvswitch\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.721496 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720829    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-ovnkube-script-lib\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.721496 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720871    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-host-run-multus-certs\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.721496 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720897    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd8e67a3-3131-4932-b561-112a63c7e612-iptables-alerter-script\") pod \"iptables-alerter-p4lhk\" (UID: \"bd8e67a3-3131-4932-b561-112a63c7e612\") " pod="openshift-network-operator/iptables-alerter-p4lhk"
Sep 17 21:41:26.721496 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720925    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-run-openvswitch\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.721496 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720949    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-ovnkube-config\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.721496 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.720976    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-etc-tuned\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.721496 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721014    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59d7a806-c3e4-48f4-82df-8bfaa212a9db-system-cni-dir\") pod \"multus-additional-cni-plugins-kjjz4\" (UID: \"59d7a806-c3e4-48f4-82df-8bfaa212a9db\") " pod="openshift-multus/multus-additional-cni-plugins-kjjz4"
Sep 17 21:41:26.721496 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721041    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-etc-sysctl-conf\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.721496 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721067    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-lib-modules\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.721496 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721095    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd8e67a3-3131-4932-b561-112a63c7e612-host-slash\") pod \"iptables-alerter-p4lhk\" (UID: \"bd8e67a3-3131-4932-b561-112a63c7e612\") " pod="openshift-network-operator/iptables-alerter-p4lhk"
Sep 17 21:41:26.721496 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721123    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-systemd-units\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.721496 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721143    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d49ed8ac-69ff-4b6e-ad3a-3f45026b84ae-konnectivity-ca\") pod \"konnectivity-agent-7ngpc\" (UID: \"d49ed8ac-69ff-4b6e-ad3a-3f45026b84ae\") " pod="kube-system/konnectivity-agent-7ngpc"
Sep 17 21:41:26.721496 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721153    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-env-overrides\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.721496 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721177    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/56fa7d57-fa08-457e-8a11-ca93979cdc8a-sys-fs\") pod \"aws-ebs-csi-driver-node-778pk\" (UID: \"56fa7d57-fa08-457e-8a11-ca93979cdc8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk"
Sep 17 21:41:26.721496 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721205    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-multus-cni-dir\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.721496 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721231    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59d7a806-c3e4-48f4-82df-8bfaa212a9db-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kjjz4\" (UID: \"59d7a806-c3e4-48f4-82df-8bfaa212a9db\") " pod="openshift-multus/multus-additional-cni-plugins-kjjz4"
Sep 17 21:41:26.721496 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721278    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e76c503-ec71-47a9-b3cd-63d3b12a439a-original-pull-secret\") pod \"global-pull-secret-syncer-hrbr6\" (UID: \"1e76c503-ec71-47a9-b3cd-63d3b12a439a\") " pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:26.722240 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721307    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-etc-modprobe-d\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.722240 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721333    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-host\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.722240 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721373    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-host-kubelet\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.722240 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721403    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.722240 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721432    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-ovn-node-metrics-cert\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.722240 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721465    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-cnibin\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.722240 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721491    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-var-lib-openvswitch\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.722240 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721532    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-node-log\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.722240 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721559    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/56fa7d57-fa08-457e-8a11-ca93979cdc8a-registration-dir\") pod \"aws-ebs-csi-driver-node-778pk\" (UID: \"56fa7d57-fa08-457e-8a11-ca93979cdc8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk"
Sep 17 21:41:26.722240 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721586    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/56fa7d57-fa08-457e-8a11-ca93979cdc8a-device-dir\") pod \"aws-ebs-csi-driver-node-778pk\" (UID: \"56fa7d57-fa08-457e-8a11-ca93979cdc8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk"
Sep 17 21:41:26.722240 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721618    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k288\" (UniqueName: \"kubernetes.io/projected/93119bca-cc51-4b33-964b-63bf4fe5c9e1-kube-api-access-2k288\") pod \"network-metrics-daemon-5gxhr\" (UID: \"93119bca-cc51-4b33-964b-63bf4fe5c9e1\") " pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:26.722240 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721650    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffkl8\" (UniqueName: \"kubernetes.io/projected/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-kube-api-access-ffkl8\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.722240 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721678    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49f3497d-f458-44e9-8846-92afb8ba26eb-serviceca\") pod \"node-ca-rkgql\" (UID: \"49f3497d-f458-44e9-8846-92afb8ba26eb\") " pod="openshift-image-registry/node-ca-rkgql"
Sep 17 21:41:26.722240 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721684    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8d2b64b-06cf-4a62-b73f-6835eab58938-cni-binary-copy\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.722240 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721706    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-system-cni-dir\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.722240 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721735    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-host-var-lib-kubelet\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.722240 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721748    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-host-var-lib-cni-multus\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.723083 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721777    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-etc-kubernetes\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.723083 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721805    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-sys\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.723083 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721835    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l26xs\" (UniqueName: \"kubernetes.io/projected/bd8e67a3-3131-4932-b561-112a63c7e612-kube-api-access-l26xs\") pod \"iptables-alerter-p4lhk\" (UID: \"bd8e67a3-3131-4932-b561-112a63c7e612\") " pod="openshift-network-operator/iptables-alerter-p4lhk"
Sep 17 21:41:26.723083 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721863    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-tmp\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.723083 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721893    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlhnr\" (UniqueName: \"kubernetes.io/projected/5fd48c9a-8cd8-4743-8dbf-eb20219c893c-kube-api-access-mlhnr\") pod \"network-check-target-hmscl\" (UID: \"5fd48c9a-8cd8-4743-8dbf-eb20219c893c\") " pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:26.723083 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721928    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-multus-conf-dir\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.723083 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721956    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-host-run-k8s-cni-cncf-io\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.723083 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.721984    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-etc-sysconfig\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.723083 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722012    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-host-cni-bin\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.723083 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722043    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-host-run-netns\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.723083 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722074    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-log-socket\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.723083 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722103    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49f3497d-f458-44e9-8846-92afb8ba26eb-host\") pod \"node-ca-rkgql\" (UID: \"49f3497d-f458-44e9-8846-92afb8ba26eb\") " pod="openshift-image-registry/node-ca-rkgql"
Sep 17 21:41:26.723083 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722136    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59d7a806-c3e4-48f4-82df-8bfaa212a9db-cnibin\") pod \"multus-additional-cni-plugins-kjjz4\" (UID: \"59d7a806-c3e4-48f4-82df-8bfaa212a9db\") " pod="openshift-multus/multus-additional-cni-plugins-kjjz4"
Sep 17 21:41:26.723083 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722165    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59d7a806-c3e4-48f4-82df-8bfaa212a9db-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kjjz4\" (UID: \"59d7a806-c3e4-48f4-82df-8bfaa212a9db\") " pod="openshift-multus/multus-additional-cni-plugins-kjjz4"
Sep 17 21:41:26.723083 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722212    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-etc-systemd\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.723083 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722234    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59d7a806-c3e4-48f4-82df-8bfaa212a9db-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kjjz4\" (UID: \"59d7a806-c3e4-48f4-82df-8bfaa212a9db\") " pod="openshift-multus/multus-additional-cni-plugins-kjjz4"
Sep 17 21:41:26.723083 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722267    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e76c503-ec71-47a9-b3cd-63d3b12a439a-kubelet-config\") pod \"global-pull-secret-syncer-hrbr6\" (UID: \"1e76c503-ec71-47a9-b3cd-63d3b12a439a\") " pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:26.724105 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722379    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e76c503-ec71-47a9-b3cd-63d3b12a439a-kubelet-config\") pod \"global-pull-secret-syncer-hrbr6\" (UID: \"1e76c503-ec71-47a9-b3cd-63d3b12a439a\") " pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:26.724105 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722391    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-multus-conf-dir\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.724105 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722416    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-host\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.724105 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722511    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-run\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.724105 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722566    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-var-lib-kubelet\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.724105 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722565    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-multus-cni-dir\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.724105 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722616    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-etc-sysctl-d\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.724105 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722622    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-host-run-k8s-cni-cncf-io\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.724105 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722639    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 21:41:26.724105 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722649    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-cnibin\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.724105 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722659    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-host-var-lib-kubelet\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.724105 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722730    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-etc-sysconfig\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.724105 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722730    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49f3497d-f458-44e9-8846-92afb8ba26eb-host\") pod \"node-ca-rkgql\" (UID: \"49f3497d-f458-44e9-8846-92afb8ba26eb\") " pod="openshift-image-registry/node-ca-rkgql"
Sep 17 21:41:26.724105 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722769    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59d7a806-c3e4-48f4-82df-8bfaa212a9db-cnibin\") pod \"multus-additional-cni-plugins-kjjz4\" (UID: \"59d7a806-c3e4-48f4-82df-8bfaa212a9db\") " pod="openshift-multus/multus-additional-cni-plugins-kjjz4"
Sep 17 21:41:26.724105 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722934    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-etc-kubernetes\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.724105 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722972    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-system-cni-dir\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.724105 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.723035    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49f3497d-f458-44e9-8846-92afb8ba26eb-serviceca\") pod \"node-ca-rkgql\" (UID: \"49f3497d-f458-44e9-8846-92afb8ba26eb\") " pod="openshift-image-registry/node-ca-rkgql"
Sep 17 21:41:26.724105 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:26.723042    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:41:26.725146 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:26.723130    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93119bca-cc51-4b33-964b-63bf4fe5c9e1-metrics-certs podName:93119bca-cc51-4b33-964b-63bf4fe5c9e1 nodeName:}" failed. No retries permitted until 2026-09-17 21:41:27.223111149 +0000 UTC m=+3.041437462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93119bca-cc51-4b33-964b-63bf4fe5c9e1-metrics-certs") pod "network-metrics-daemon-5gxhr" (UID: "93119bca-cc51-4b33-964b-63bf4fe5c9e1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:41:26.725146 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.723155    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59d7a806-c3e4-48f4-82df-8bfaa212a9db-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kjjz4\" (UID: \"59d7a806-c3e4-48f4-82df-8bfaa212a9db\") " pod="openshift-multus/multus-additional-cni-plugins-kjjz4"
Sep 17 21:41:26.725146 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.723168    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59d7a806-c3e4-48f4-82df-8bfaa212a9db-system-cni-dir\") pod \"multus-additional-cni-plugins-kjjz4\" (UID: \"59d7a806-c3e4-48f4-82df-8bfaa212a9db\") " pod="openshift-multus/multus-additional-cni-plugins-kjjz4"
Sep 17 21:41:26.725146 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.723216    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59115160-02ac-43e8-9a62-0c91775ba337-hosts-file\") pod \"node-resolver-fdkhb\" (UID: \"59115160-02ac-43e8-9a62-0c91775ba337\") " pod="openshift-dns/node-resolver-fdkhb"
Sep 17 21:41:26.725146 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.723289    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-multus-socket-dir-parent\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.725146 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:26.723293    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:41:26.725146 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.722393    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-sys\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.725146 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:26.723373    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e76c503-ec71-47a9-b3cd-63d3b12a439a-original-pull-secret podName:1e76c503-ec71-47a9-b3cd-63d3b12a439a nodeName:}" failed. No retries permitted until 2026-09-17 21:41:27.223353407 +0000 UTC m=+3.041679716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e76c503-ec71-47a9-b3cd-63d3b12a439a-original-pull-secret") pod "global-pull-secret-syncer-hrbr6" (UID: "1e76c503-ec71-47a9-b3cd-63d3b12a439a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:41:26.725146 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.723451    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-host-var-lib-cni-bin\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.725146 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.723470    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-lib-modules\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.725146 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.723481    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-etc-systemd\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.725146 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.723529    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e76c503-ec71-47a9-b3cd-63d3b12a439a-dbus\") pod \"global-pull-secret-syncer-hrbr6\" (UID: \"1e76c503-ec71-47a9-b3cd-63d3b12a439a\") " pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:26.725146 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.723531    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-etc-modprobe-d\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.725146 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.723556    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-hostroot\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.725146 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.723591    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-etc-sysctl-conf\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.725146 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.723607    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-etc-kubernetes\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.725146 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.723640    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-host-run-multus-certs\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.726004 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.723728    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a8d2b64b-06cf-4a62-b73f-6835eab58938-multus-daemon-config\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.726004 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.723787    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a8d2b64b-06cf-4a62-b73f-6835eab58938-host-run-netns\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.726004 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.723868    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59d7a806-c3e4-48f4-82df-8bfaa212a9db-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kjjz4\" (UID: \"59d7a806-c3e4-48f4-82df-8bfaa212a9db\") " pod="openshift-multus/multus-additional-cni-plugins-kjjz4"
Sep 17 21:41:26.726004 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.724094    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59d7a806-c3e4-48f4-82df-8bfaa212a9db-cni-binary-copy\") pod \"multus-additional-cni-plugins-kjjz4\" (UID: \"59d7a806-c3e4-48f4-82df-8bfaa212a9db\") " pod="openshift-multus/multus-additional-cni-plugins-kjjz4"
Sep 17 21:41:26.727667 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.727593    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-tmp\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.728758 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.728731    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-etc-tuned\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.730328 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.730007    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d49ed8ac-69ff-4b6e-ad3a-3f45026b84ae-agent-certs\") pod \"konnectivity-agent-7ngpc\" (UID: \"d49ed8ac-69ff-4b6e-ad3a-3f45026b84ae\") " pod="kube-system/konnectivity-agent-7ngpc"
Sep 17 21:41:26.731993 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.731968    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ctcb\" (UniqueName: \"kubernetes.io/projected/59115160-02ac-43e8-9a62-0c91775ba337-kube-api-access-2ctcb\") pod \"node-resolver-fdkhb\" (UID: \"59115160-02ac-43e8-9a62-0c91775ba337\") " pod="openshift-dns/node-resolver-fdkhb"
Sep 17 21:41:26.736566 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.736540    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlxst\" (UniqueName: \"kubernetes.io/projected/a8d2b64b-06cf-4a62-b73f-6835eab58938-kube-api-access-wlxst\") pod \"multus-hxcs4\" (UID: \"a8d2b64b-06cf-4a62-b73f-6835eab58938\") " pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.738526 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.738500    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffkl8\" (UniqueName: \"kubernetes.io/projected/0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d-kube-api-access-ffkl8\") pod \"tuned-j4fpb\" (UID: \"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d\") " pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.740053 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.740028    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k288\" (UniqueName: \"kubernetes.io/projected/93119bca-cc51-4b33-964b-63bf4fe5c9e1-kube-api-access-2k288\") pod \"network-metrics-daemon-5gxhr\" (UID: \"93119bca-cc51-4b33-964b-63bf4fe5c9e1\") " pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:26.746729 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.746688    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h8ks\" (UniqueName: \"kubernetes.io/projected/49f3497d-f458-44e9-8846-92afb8ba26eb-kube-api-access-5h8ks\") pod \"node-ca-rkgql\" (UID: \"49f3497d-f458-44e9-8846-92afb8ba26eb\") " pod="openshift-image-registry/node-ca-rkgql"
Sep 17 21:41:26.752087 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.751954    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pmp5\" (UniqueName: \"kubernetes.io/projected/59d7a806-c3e4-48f4-82df-8bfaa212a9db-kube-api-access-2pmp5\") pod \"multus-additional-cni-plugins-kjjz4\" (UID: \"59d7a806-c3e4-48f4-82df-8bfaa212a9db\") " pod="openshift-multus/multus-additional-cni-plugins-kjjz4"
Sep 17 21:41:26.754509 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.754413    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-67.ec2.internal" event={"ID":"c19c959590466136e44fe97bd5d57e26","Type":"ContainerStarted","Data":"6f70f5f842b13f4f13cc57a8f6e65c00ddccff70dd4d3ece015bd1bd521e7705"}
Sep 17 21:41:26.757167 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.757097    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal" event={"ID":"69be7f4fd498a52ca0b57b8db0dca68c","Type":"ContainerStarted","Data":"6b2305736d6bc1a48a80736ee1a10ea7f83e578bfcf47354a433b58076144aab"}
Sep 17 21:41:26.822745 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.822702    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-run-ovn\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.822995 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.822755    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-host-run-ovn-kubernetes\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.822995 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.822792    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/56fa7d57-fa08-457e-8a11-ca93979cdc8a-socket-dir\") pod \"aws-ebs-csi-driver-node-778pk\" (UID: \"56fa7d57-fa08-457e-8a11-ca93979cdc8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk"
Sep 17 21:41:26.822995 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.822822    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-host-cni-netd\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.822995 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.822848    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/56fa7d57-fa08-457e-8a11-ca93979cdc8a-etc-selinux\") pod \"aws-ebs-csi-driver-node-778pk\" (UID: \"56fa7d57-fa08-457e-8a11-ca93979cdc8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk"
Sep 17 21:41:26.822995 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.822889    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-host-run-netns\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.822995 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.822912    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-etc-openvswitch\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.822995 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.822938    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-ovnkube-script-lib\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.822995 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.822969    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd8e67a3-3131-4932-b561-112a63c7e612-iptables-alerter-script\") pod \"iptables-alerter-p4lhk\" (UID: \"bd8e67a3-3131-4932-b561-112a63c7e612\") " pod="openshift-network-operator/iptables-alerter-p4lhk"
Sep 17 21:41:26.822995 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.822993    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-run-openvswitch\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.823501 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823018    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-ovnkube-config\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.823501 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823048    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd8e67a3-3131-4932-b561-112a63c7e612-host-slash\") pod \"iptables-alerter-p4lhk\" (UID: \"bd8e67a3-3131-4932-b561-112a63c7e612\") " pod="openshift-network-operator/iptables-alerter-p4lhk"
Sep 17 21:41:26.823501 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823064    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-systemd-units\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.823501 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823078    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-env-overrides\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.823501 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823096    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/56fa7d57-fa08-457e-8a11-ca93979cdc8a-sys-fs\") pod \"aws-ebs-csi-driver-node-778pk\" (UID: \"56fa7d57-fa08-457e-8a11-ca93979cdc8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk"
Sep 17 21:41:26.823501 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823132    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-host-kubelet\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.823501 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823150    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.823501 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823179    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-ovn-node-metrics-cert\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.823501 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823207    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-var-lib-openvswitch\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.823501 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823224    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-run-ovn\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.823501 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823247    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-node-log\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.823501 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823275    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/56fa7d57-fa08-457e-8a11-ca93979cdc8a-registration-dir\") pod \"aws-ebs-csi-driver-node-778pk\" (UID: \"56fa7d57-fa08-457e-8a11-ca93979cdc8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk"
Sep 17 21:41:26.823501 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823302    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/56fa7d57-fa08-457e-8a11-ca93979cdc8a-device-dir\") pod \"aws-ebs-csi-driver-node-778pk\" (UID: \"56fa7d57-fa08-457e-8a11-ca93979cdc8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk"
Sep 17 21:41:26.823501 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823331    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-host-kubelet\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.823501 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823345    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd8e67a3-3131-4932-b561-112a63c7e612-host-slash\") pod \"iptables-alerter-p4lhk\" (UID: \"bd8e67a3-3131-4932-b561-112a63c7e612\") " pod="openshift-network-operator/iptables-alerter-p4lhk"
Sep 17 21:41:26.823501 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823394    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/56fa7d57-fa08-457e-8a11-ca93979cdc8a-socket-dir\") pod \"aws-ebs-csi-driver-node-778pk\" (UID: \"56fa7d57-fa08-457e-8a11-ca93979cdc8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk"
Sep 17 21:41:26.823501 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823441    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-host-run-netns\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.824326 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823450    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-etc-openvswitch\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.824326 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823401    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-host-cni-netd\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.824326 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823397    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/56fa7d57-fa08-457e-8a11-ca93979cdc8a-etc-selinux\") pod \"aws-ebs-csi-driver-node-778pk\" (UID: \"56fa7d57-fa08-457e-8a11-ca93979cdc8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk"
Sep 17 21:41:26.824326 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823495    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-run-openvswitch\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.824326 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823515    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/56fa7d57-fa08-457e-8a11-ca93979cdc8a-registration-dir\") pod \"aws-ebs-csi-driver-node-778pk\" (UID: \"56fa7d57-fa08-457e-8a11-ca93979cdc8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk"
Sep 17 21:41:26.824326 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823559    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-var-lib-openvswitch\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.824326 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823566    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/56fa7d57-fa08-457e-8a11-ca93979cdc8a-sys-fs\") pod \"aws-ebs-csi-driver-node-778pk\" (UID: \"56fa7d57-fa08-457e-8a11-ca93979cdc8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk"
Sep 17 21:41:26.824326 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823601    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-node-log\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.824326 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823648    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-systemd-units\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.824326 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823701    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/56fa7d57-fa08-457e-8a11-ca93979cdc8a-device-dir\") pod \"aws-ebs-csi-driver-node-778pk\" (UID: \"56fa7d57-fa08-457e-8a11-ca93979cdc8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk"
Sep 17 21:41:26.824326 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823337    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l26xs\" (UniqueName: \"kubernetes.io/projected/bd8e67a3-3131-4932-b561-112a63c7e612-kube-api-access-l26xs\") pod \"iptables-alerter-p4lhk\" (UID: \"bd8e67a3-3131-4932-b561-112a63c7e612\") " pod="openshift-network-operator/iptables-alerter-p4lhk"
Sep 17 21:41:26.824326 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823746    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlhnr\" (UniqueName: \"kubernetes.io/projected/5fd48c9a-8cd8-4743-8dbf-eb20219c893c-kube-api-access-mlhnr\") pod \"network-check-target-hmscl\" (UID: \"5fd48c9a-8cd8-4743-8dbf-eb20219c893c\") " pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:26.824326 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823776    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-host-cni-bin\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.824326 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823802    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-log-socket\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.824326 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823833    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-host-slash\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.824326 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823860    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkd7w\" (UniqueName: \"kubernetes.io/projected/56fa7d57-fa08-457e-8a11-ca93979cdc8a-kube-api-access-fkd7w\") pod \"aws-ebs-csi-driver-node-778pk\" (UID: \"56fa7d57-fa08-457e-8a11-ca93979cdc8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk"
Sep 17 21:41:26.824326 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823894    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-run-systemd\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.825145 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823921    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbqmn\" (UniqueName: \"kubernetes.io/projected/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-kube-api-access-vbqmn\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.825145 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823937    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd8e67a3-3131-4932-b561-112a63c7e612-iptables-alerter-script\") pod \"iptables-alerter-p4lhk\" (UID: \"bd8e67a3-3131-4932-b561-112a63c7e612\") " pod="openshift-network-operator/iptables-alerter-p4lhk"
Sep 17 21:41:26.825145 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.823956    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/56fa7d57-fa08-457e-8a11-ca93979cdc8a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-778pk\" (UID: \"56fa7d57-fa08-457e-8a11-ca93979cdc8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk"
Sep 17 21:41:26.825145 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.824000    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.825145 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.824004    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-ovnkube-script-lib\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.825145 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.824032    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/56fa7d57-fa08-457e-8a11-ca93979cdc8a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-778pk\" (UID: \"56fa7d57-fa08-457e-8a11-ca93979cdc8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk"
Sep 17 21:41:26.825145 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.824182    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-run-systemd\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.825145 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.824288    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-host-cni-bin\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.825145 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.824299    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-host-slash\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.825145 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.824340    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-log-socket\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.825145 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.824491    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-host-run-ovn-kubernetes\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.825145 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.824867    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-ovnkube-config\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.825771 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.825333    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-env-overrides\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.827318 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.827291    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-ovn-node-metrics-cert\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.832284 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.832061    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:41:26.838739 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:26.838718    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:41:26.838873 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:26.838749    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:41:26.838873 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:26.838763    2564 projected.go:196] Error preparing data for projected volume kube-api-access-mlhnr for pod openshift-network-diagnostics/network-check-target-hmscl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:41:26.838873 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:26.838829    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5fd48c9a-8cd8-4743-8dbf-eb20219c893c-kube-api-access-mlhnr podName:5fd48c9a-8cd8-4743-8dbf-eb20219c893c nodeName:}" failed. No retries permitted until 2026-09-17 21:41:27.338810134 +0000 UTC m=+3.157136440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mlhnr" (UniqueName: "kubernetes.io/projected/5fd48c9a-8cd8-4743-8dbf-eb20219c893c-kube-api-access-mlhnr") pod "network-check-target-hmscl" (UID: "5fd48c9a-8cd8-4743-8dbf-eb20219c893c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:41:26.842289 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.842264    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkd7w\" (UniqueName: \"kubernetes.io/projected/56fa7d57-fa08-457e-8a11-ca93979cdc8a-kube-api-access-fkd7w\") pod \"aws-ebs-csi-driver-node-778pk\" (UID: \"56fa7d57-fa08-457e-8a11-ca93979cdc8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk"
Sep 17 21:41:26.849989 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.849967    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l26xs\" (UniqueName: \"kubernetes.io/projected/bd8e67a3-3131-4932-b561-112a63c7e612-kube-api-access-l26xs\") pod \"iptables-alerter-p4lhk\" (UID: \"bd8e67a3-3131-4932-b561-112a63c7e612\") " pod="openshift-network-operator/iptables-alerter-p4lhk"
Sep 17 21:41:26.851783 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.851746    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbqmn\" (UniqueName: \"kubernetes.io/projected/6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2-kube-api-access-vbqmn\") pod \"ovnkube-node-zdcnw\" (UID: \"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:26.907976 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.907939    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kjjz4"
Sep 17 21:41:26.920536 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.920509    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j4fpb"
Sep 17 21:41:26.930436 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.930406    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fdkhb"
Sep 17 21:41:26.938496 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.938468    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rkgql"
Sep 17 21:41:26.950390 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.950364    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7ngpc"
Sep 17 21:41:26.958744 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.958721    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hxcs4"
Sep 17 21:41:26.966037 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.966012    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk"
Sep 17 21:41:26.975900 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.975876    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p4lhk"
Sep 17 21:41:26.983892 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:26.983864    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:27.226412 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:27.226374    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93119bca-cc51-4b33-964b-63bf4fe5c9e1-metrics-certs\") pod \"network-metrics-daemon-5gxhr\" (UID: \"93119bca-cc51-4b33-964b-63bf4fe5c9e1\") " pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:27.226412 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:27.226418    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e76c503-ec71-47a9-b3cd-63d3b12a439a-original-pull-secret\") pod \"global-pull-secret-syncer-hrbr6\" (UID: \"1e76c503-ec71-47a9-b3cd-63d3b12a439a\") " pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:27.226663 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:27.226550    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:41:27.226663 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:27.226551    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:41:27.226663 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:27.226608    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e76c503-ec71-47a9-b3cd-63d3b12a439a-original-pull-secret podName:1e76c503-ec71-47a9-b3cd-63d3b12a439a nodeName:}" failed. No retries permitted until 2026-09-17 21:41:28.226594614 +0000 UTC m=+4.044920910 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e76c503-ec71-47a9-b3cd-63d3b12a439a-original-pull-secret") pod "global-pull-secret-syncer-hrbr6" (UID: "1e76c503-ec71-47a9-b3cd-63d3b12a439a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:41:27.226663 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:27.226625    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93119bca-cc51-4b33-964b-63bf4fe5c9e1-metrics-certs podName:93119bca-cc51-4b33-964b-63bf4fe5c9e1 nodeName:}" failed. No retries permitted until 2026-09-17 21:41:28.226616392 +0000 UTC m=+4.044942688 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93119bca-cc51-4b33-964b-63bf4fe5c9e1-metrics-certs") pod "network-metrics-daemon-5gxhr" (UID: "93119bca-cc51-4b33-964b-63bf4fe5c9e1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:41:27.427621 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:27.427567    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlhnr\" (UniqueName: \"kubernetes.io/projected/5fd48c9a-8cd8-4743-8dbf-eb20219c893c-kube-api-access-mlhnr\") pod \"network-check-target-hmscl\" (UID: \"5fd48c9a-8cd8-4743-8dbf-eb20219c893c\") " pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:27.427824 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:27.427759    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:41:27.427824 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:27.427786    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:41:27.427824 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:27.427799    2564 projected.go:196] Error preparing data for projected volume kube-api-access-mlhnr for pod openshift-network-diagnostics/network-check-target-hmscl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:41:27.427958 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:27.427869    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5fd48c9a-8cd8-4743-8dbf-eb20219c893c-kube-api-access-mlhnr podName:5fd48c9a-8cd8-4743-8dbf-eb20219c893c nodeName:}" failed. No retries permitted until 2026-09-17 21:41:28.427850254 +0000 UTC m=+4.246176571 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlhnr" (UniqueName: "kubernetes.io/projected/5fd48c9a-8cd8-4743-8dbf-eb20219c893c-kube-api-access-mlhnr") pod "network-check-target-hmscl" (UID: "5fd48c9a-8cd8-4743-8dbf-eb20219c893c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:41:27.664705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:27.664653    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 21:36:27 +0000 UTC" deadline="2028-04-26 22:56:08.939206493 +0000 UTC"
Sep 17 21:41:27.664705 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:27.664689    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14089h14m41.274522161s"
Sep 17 21:41:28.046106 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:28.046076    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef4bf29_cd42_4cb6_8683_6dd3fdab58d2.slice/crio-2c78b92ed605a32ed0edeffe368b80c1c9cdfc32051146413f7ff08ae0165203 WatchSource:0}: Error finding container 2c78b92ed605a32ed0edeffe368b80c1c9cdfc32051146413f7ff08ae0165203: Status 404 returned error can't find the container with id 2c78b92ed605a32ed0edeffe368b80c1c9cdfc32051146413f7ff08ae0165203
Sep 17 21:41:28.048382 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:28.048272    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d7a806_c3e4_48f4_82df_8bfaa212a9db.slice/crio-36d18e19a28ac14cf40f028faec642044c755e0f2233e1783e812bfcad0832c4 WatchSource:0}: Error finding container 36d18e19a28ac14cf40f028faec642044c755e0f2233e1783e812bfcad0832c4: Status 404 returned error can't find the container with id 36d18e19a28ac14cf40f028faec642044c755e0f2233e1783e812bfcad0832c4
Sep 17 21:41:28.050338 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:28.050297    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56fa7d57_fa08_457e_8a11_ca93979cdc8a.slice/crio-684eea31fd0d114ec5be91966fde9ad4f8b83f6f35c61006e57e211e26013b42 WatchSource:0}: Error finding container 684eea31fd0d114ec5be91966fde9ad4f8b83f6f35c61006e57e211e26013b42: Status 404 returned error can't find the container with id 684eea31fd0d114ec5be91966fde9ad4f8b83f6f35c61006e57e211e26013b42
Sep 17 21:41:28.053372 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:28.053345    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49ed8ac_69ff_4b6e_ad3a_3f45026b84ae.slice/crio-c762f75247334e691165c461767932f8045c3a7284b8c9439df53060d66e8778 WatchSource:0}: Error finding container c762f75247334e691165c461767932f8045c3a7284b8c9439df53060d66e8778: Status 404 returned error can't find the container with id c762f75247334e691165c461767932f8045c3a7284b8c9439df53060d66e8778
Sep 17 21:41:28.054459 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:28.054436    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f3497d_f458_44e9_8846_92afb8ba26eb.slice/crio-6d9be23f42b495901a6be8077146bbb18a59da0e38523245cb2d7c4256b479e9 WatchSource:0}: Error finding container 6d9be23f42b495901a6be8077146bbb18a59da0e38523245cb2d7c4256b479e9: Status 404 returned error can't find the container with id 6d9be23f42b495901a6be8077146bbb18a59da0e38523245cb2d7c4256b479e9
Sep 17 21:41:28.055324 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:28.055300    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59115160_02ac_43e8_9a62_0c91775ba337.slice/crio-d530d5c27483fd6fb98ce77b8358d4f10a8695b4b22d8bd19bbacc5a0e4e9222 WatchSource:0}: Error finding container d530d5c27483fd6fb98ce77b8358d4f10a8695b4b22d8bd19bbacc5a0e4e9222: Status 404 returned error can't find the container with id d530d5c27483fd6fb98ce77b8358d4f10a8695b4b22d8bd19bbacc5a0e4e9222
Sep 17 21:41:28.060454 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:41:28.060411    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d2b64b_06cf_4a62_b73f_6835eab58938.slice/crio-40aeedc9c54921904e94ee4d10fc8bd83b5091b230780a5b2e96781ef1648398 WatchSource:0}: Error finding container 40aeedc9c54921904e94ee4d10fc8bd83b5091b230780a5b2e96781ef1648398: Status 404 returned error can't find the container with id 40aeedc9c54921904e94ee4d10fc8bd83b5091b230780a5b2e96781ef1648398
Sep 17 21:41:28.235011 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:28.234959    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93119bca-cc51-4b33-964b-63bf4fe5c9e1-metrics-certs\") pod \"network-metrics-daemon-5gxhr\" (UID: \"93119bca-cc51-4b33-964b-63bf4fe5c9e1\") " pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:28.235168 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:28.235022    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e76c503-ec71-47a9-b3cd-63d3b12a439a-original-pull-secret\") pod \"global-pull-secret-syncer-hrbr6\" (UID: \"1e76c503-ec71-47a9-b3cd-63d3b12a439a\") " pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:28.235168 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:28.235143    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:41:28.235168 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:28.235149    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:41:28.235381 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:28.235222    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e76c503-ec71-47a9-b3cd-63d3b12a439a-original-pull-secret podName:1e76c503-ec71-47a9-b3cd-63d3b12a439a nodeName:}" failed. No retries permitted until 2026-09-17 21:41:30.235188651 +0000 UTC m=+6.053514943 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e76c503-ec71-47a9-b3cd-63d3b12a439a-original-pull-secret") pod "global-pull-secret-syncer-hrbr6" (UID: "1e76c503-ec71-47a9-b3cd-63d3b12a439a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:41:28.235381 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:28.235253    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93119bca-cc51-4b33-964b-63bf4fe5c9e1-metrics-certs podName:93119bca-cc51-4b33-964b-63bf4fe5c9e1 nodeName:}" failed. No retries permitted until 2026-09-17 21:41:30.235232667 +0000 UTC m=+6.053558966 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93119bca-cc51-4b33-964b-63bf4fe5c9e1-metrics-certs") pod "network-metrics-daemon-5gxhr" (UID: "93119bca-cc51-4b33-964b-63bf4fe5c9e1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:41:28.437238 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:28.437183    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlhnr\" (UniqueName: \"kubernetes.io/projected/5fd48c9a-8cd8-4743-8dbf-eb20219c893c-kube-api-access-mlhnr\") pod \"network-check-target-hmscl\" (UID: \"5fd48c9a-8cd8-4743-8dbf-eb20219c893c\") " pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:28.437404 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:28.437369    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:41:28.437477 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:28.437403    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:41:28.437477 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:28.437416    2564 projected.go:196] Error preparing data for projected volume kube-api-access-mlhnr for pod openshift-network-diagnostics/network-check-target-hmscl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:41:28.437477 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:28.437472    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5fd48c9a-8cd8-4743-8dbf-eb20219c893c-kube-api-access-mlhnr podName:5fd48c9a-8cd8-4743-8dbf-eb20219c893c nodeName:}" failed. No retries permitted until 2026-09-17 21:41:30.437458335 +0000 UTC m=+6.255784641 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlhnr" (UniqueName: "kubernetes.io/projected/5fd48c9a-8cd8-4743-8dbf-eb20219c893c-kube-api-access-mlhnr") pod "network-check-target-hmscl" (UID: "5fd48c9a-8cd8-4743-8dbf-eb20219c893c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:41:28.665496 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:28.665309    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 21:36:27 +0000 UTC" deadline="2028-03-05 11:07:13.124703057 +0000 UTC"
Sep 17 21:41:28.665496 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:28.665374    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12829h25m44.459333682s"
Sep 17 21:41:28.751726 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:28.750798    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:28.751726 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:28.750940    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrbr6" podUID="1e76c503-ec71-47a9-b3cd-63d3b12a439a"
Sep 17 21:41:28.751726 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:28.751393    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:28.751726 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:28.751510    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gxhr" podUID="93119bca-cc51-4b33-964b-63bf4fe5c9e1"
Sep 17 21:41:28.751726 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:28.751572    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:28.751726 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:28.751647    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmscl" podUID="5fd48c9a-8cd8-4743-8dbf-eb20219c893c"
Sep 17 21:41:28.789150 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:28.789086    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7ngpc" event={"ID":"d49ed8ac-69ff-4b6e-ad3a-3f45026b84ae","Type":"ContainerStarted","Data":"c762f75247334e691165c461767932f8045c3a7284b8c9439df53060d66e8778"}
Sep 17 21:41:28.796837 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:28.796800    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-67.ec2.internal" event={"ID":"c19c959590466136e44fe97bd5d57e26","Type":"ContainerStarted","Data":"bd5fb9cc8cda59fbb9439d33ae7468546cd2ea42bc61fcad8086880bd43d45fa"}
Sep 17 21:41:28.809570 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:28.809533    2564 generic.go:356] "Generic (PLEG): container finished" podID="69be7f4fd498a52ca0b57b8db0dca68c" containerID="3b9ac58be1f5520d8de8c2c8d1430c65c24fe40da761ecfa1fa4af533b4823f5" exitCode=0
Sep 17 21:41:28.809732 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:28.809625    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal" event={"ID":"69be7f4fd498a52ca0b57b8db0dca68c","Type":"ContainerDied","Data":"3b9ac58be1f5520d8de8c2c8d1430c65c24fe40da761ecfa1fa4af533b4823f5"}
Sep 17 21:41:28.816579 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:28.816553    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hxcs4" event={"ID":"a8d2b64b-06cf-4a62-b73f-6835eab58938","Type":"ContainerStarted","Data":"40aeedc9c54921904e94ee4d10fc8bd83b5091b230780a5b2e96781ef1648398"}
Sep 17 21:41:28.819712 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:28.819637    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-67.ec2.internal" podStartSLOduration=2.81961533 podStartE2EDuration="2.81961533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:41:28.81955525 +0000 UTC m=+4.637881566" watchObservedRunningTime="2026-09-17 21:41:28.81961533 +0000 UTC m=+4.637941645"
Sep 17 21:41:28.833098 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:28.833052    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p4lhk" event={"ID":"bd8e67a3-3131-4932-b561-112a63c7e612","Type":"ContainerStarted","Data":"4df1d1c9724d900c122b5de9723744172e502254cbca7a2fdb9c97d3509b1dab"}
Sep 17 21:41:28.850466 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:28.850425    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rkgql" event={"ID":"49f3497d-f458-44e9-8846-92afb8ba26eb","Type":"ContainerStarted","Data":"6d9be23f42b495901a6be8077146bbb18a59da0e38523245cb2d7c4256b479e9"}
Sep 17 21:41:28.857253 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:28.857213    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk" event={"ID":"56fa7d57-fa08-457e-8a11-ca93979cdc8a","Type":"ContainerStarted","Data":"684eea31fd0d114ec5be91966fde9ad4f8b83f6f35c61006e57e211e26013b42"}
Sep 17 21:41:28.880634 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:28.880588    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kjjz4" event={"ID":"59d7a806-c3e4-48f4-82df-8bfaa212a9db","Type":"ContainerStarted","Data":"36d18e19a28ac14cf40f028faec642044c755e0f2233e1783e812bfcad0832c4"}
Sep 17 21:41:28.890329 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:28.890289    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw" event={"ID":"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2","Type":"ContainerStarted","Data":"2c78b92ed605a32ed0edeffe368b80c1c9cdfc32051146413f7ff08ae0165203"}
Sep 17 21:41:28.898709 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:28.898665    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j4fpb" event={"ID":"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d","Type":"ContainerStarted","Data":"33233927d1b23b78cc07b5420a3617340130d17805b43cd7fce3f4266c869515"}
Sep 17 21:41:28.919265 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:28.919096    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fdkhb" event={"ID":"59115160-02ac-43e8-9a62-0c91775ba337","Type":"ContainerStarted","Data":"d530d5c27483fd6fb98ce77b8358d4f10a8695b4b22d8bd19bbacc5a0e4e9222"}
Sep 17 21:41:29.949878 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:29.949782    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal" event={"ID":"69be7f4fd498a52ca0b57b8db0dca68c","Type":"ContainerStarted","Data":"38bdac3bb21ec0fda0ba3bfc30830f41e0ed22290cfea9ec620d311b9c240733"}
Sep 17 21:41:30.255890 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:30.255779    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93119bca-cc51-4b33-964b-63bf4fe5c9e1-metrics-certs\") pod \"network-metrics-daemon-5gxhr\" (UID: \"93119bca-cc51-4b33-964b-63bf4fe5c9e1\") " pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:30.255890 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:30.255849    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e76c503-ec71-47a9-b3cd-63d3b12a439a-original-pull-secret\") pod \"global-pull-secret-syncer-hrbr6\" (UID: \"1e76c503-ec71-47a9-b3cd-63d3b12a439a\") " pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:30.256134 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:30.255988    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:41:30.256134 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:30.256055    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e76c503-ec71-47a9-b3cd-63d3b12a439a-original-pull-secret podName:1e76c503-ec71-47a9-b3cd-63d3b12a439a nodeName:}" failed. No retries permitted until 2026-09-17 21:41:34.256036646 +0000 UTC m=+10.074362952 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e76c503-ec71-47a9-b3cd-63d3b12a439a-original-pull-secret") pod "global-pull-secret-syncer-hrbr6" (UID: "1e76c503-ec71-47a9-b3cd-63d3b12a439a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:41:30.256598 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:30.256471    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:41:30.256598 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:30.256541    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93119bca-cc51-4b33-964b-63bf4fe5c9e1-metrics-certs podName:93119bca-cc51-4b33-964b-63bf4fe5c9e1 nodeName:}" failed. No retries permitted until 2026-09-17 21:41:34.256522262 +0000 UTC m=+10.074848557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93119bca-cc51-4b33-964b-63bf4fe5c9e1-metrics-certs") pod "network-metrics-daemon-5gxhr" (UID: "93119bca-cc51-4b33-964b-63bf4fe5c9e1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:41:30.458376 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:30.458334    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlhnr\" (UniqueName: \"kubernetes.io/projected/5fd48c9a-8cd8-4743-8dbf-eb20219c893c-kube-api-access-mlhnr\") pod \"network-check-target-hmscl\" (UID: \"5fd48c9a-8cd8-4743-8dbf-eb20219c893c\") " pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:30.458575 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:30.458498    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:41:30.458575 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:30.458522    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:41:30.458575 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:30.458536    2564 projected.go:196] Error preparing data for projected volume kube-api-access-mlhnr for pod openshift-network-diagnostics/network-check-target-hmscl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:41:30.458748 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:30.458599    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5fd48c9a-8cd8-4743-8dbf-eb20219c893c-kube-api-access-mlhnr podName:5fd48c9a-8cd8-4743-8dbf-eb20219c893c nodeName:}" failed. No retries permitted until 2026-09-17 21:41:34.458581422 +0000 UTC m=+10.276907726 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlhnr" (UniqueName: "kubernetes.io/projected/5fd48c9a-8cd8-4743-8dbf-eb20219c893c-kube-api-access-mlhnr") pod "network-check-target-hmscl" (UID: "5fd48c9a-8cd8-4743-8dbf-eb20219c893c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:41:30.751947 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:30.750950    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:30.751947 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:30.751127    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrbr6" podUID="1e76c503-ec71-47a9-b3cd-63d3b12a439a"
Sep 17 21:41:30.751947 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:30.751613    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:30.751947 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:30.751732    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gxhr" podUID="93119bca-cc51-4b33-964b-63bf4fe5c9e1"
Sep 17 21:41:30.751947 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:30.751800    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:30.751947 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:30.751871    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmscl" podUID="5fd48c9a-8cd8-4743-8dbf-eb20219c893c"
Sep 17 21:41:32.750608 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:32.750572    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:32.751096 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:32.750707    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrbr6" podUID="1e76c503-ec71-47a9-b3cd-63d3b12a439a"
Sep 17 21:41:32.751174 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:32.751110    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:32.751273 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:32.751225    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gxhr" podUID="93119bca-cc51-4b33-964b-63bf4fe5c9e1"
Sep 17 21:41:32.751343 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:32.751303    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:32.751400 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:32.751384    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmscl" podUID="5fd48c9a-8cd8-4743-8dbf-eb20219c893c"
Sep 17 21:41:34.297390 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:34.297343    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93119bca-cc51-4b33-964b-63bf4fe5c9e1-metrics-certs\") pod \"network-metrics-daemon-5gxhr\" (UID: \"93119bca-cc51-4b33-964b-63bf4fe5c9e1\") " pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:34.297921 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:34.297409    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e76c503-ec71-47a9-b3cd-63d3b12a439a-original-pull-secret\") pod \"global-pull-secret-syncer-hrbr6\" (UID: \"1e76c503-ec71-47a9-b3cd-63d3b12a439a\") " pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:34.297921 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:34.297525    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:41:34.297921 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:34.297536    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:41:34.297921 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:34.297617    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93119bca-cc51-4b33-964b-63bf4fe5c9e1-metrics-certs podName:93119bca-cc51-4b33-964b-63bf4fe5c9e1 nodeName:}" failed. No retries permitted until 2026-09-17 21:41:42.297594254 +0000 UTC m=+18.115920559 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93119bca-cc51-4b33-964b-63bf4fe5c9e1-metrics-certs") pod "network-metrics-daemon-5gxhr" (UID: "93119bca-cc51-4b33-964b-63bf4fe5c9e1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:41:34.297921 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:34.297638    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e76c503-ec71-47a9-b3cd-63d3b12a439a-original-pull-secret podName:1e76c503-ec71-47a9-b3cd-63d3b12a439a nodeName:}" failed. No retries permitted until 2026-09-17 21:41:42.297627269 +0000 UTC m=+18.115953566 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e76c503-ec71-47a9-b3cd-63d3b12a439a-original-pull-secret") pod "global-pull-secret-syncer-hrbr6" (UID: "1e76c503-ec71-47a9-b3cd-63d3b12a439a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:41:34.499095 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:34.498501    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlhnr\" (UniqueName: \"kubernetes.io/projected/5fd48c9a-8cd8-4743-8dbf-eb20219c893c-kube-api-access-mlhnr\") pod \"network-check-target-hmscl\" (UID: \"5fd48c9a-8cd8-4743-8dbf-eb20219c893c\") " pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:34.499095 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:34.498662    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:41:34.499095 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:34.498684    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:41:34.499095 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:34.498697    2564 projected.go:196] Error preparing data for projected volume kube-api-access-mlhnr for pod openshift-network-diagnostics/network-check-target-hmscl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:41:34.499095 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:34.498754    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5fd48c9a-8cd8-4743-8dbf-eb20219c893c-kube-api-access-mlhnr podName:5fd48c9a-8cd8-4743-8dbf-eb20219c893c nodeName:}" failed. No retries permitted until 2026-09-17 21:41:42.498739685 +0000 UTC m=+18.317065991 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlhnr" (UniqueName: "kubernetes.io/projected/5fd48c9a-8cd8-4743-8dbf-eb20219c893c-kube-api-access-mlhnr") pod "network-check-target-hmscl" (UID: "5fd48c9a-8cd8-4743-8dbf-eb20219c893c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:41:34.749329 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:34.748754    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:34.749329 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:34.748882    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:34.749329 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:34.748904    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gxhr" podUID="93119bca-cc51-4b33-964b-63bf4fe5c9e1"
Sep 17 21:41:34.749329 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:34.748948    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:34.749329 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:34.749063    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrbr6" podUID="1e76c503-ec71-47a9-b3cd-63d3b12a439a"
Sep 17 21:41:34.749329 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:34.749126    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmscl" podUID="5fd48c9a-8cd8-4743-8dbf-eb20219c893c"
Sep 17 21:41:36.748950 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:36.748920    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:36.749398 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:36.749028    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gxhr" podUID="93119bca-cc51-4b33-964b-63bf4fe5c9e1"
Sep 17 21:41:36.749398 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:36.749140    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:36.749398 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:36.749265    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrbr6" podUID="1e76c503-ec71-47a9-b3cd-63d3b12a439a"
Sep 17 21:41:36.749398 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:36.749140    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:36.749668 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:36.749403    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmscl" podUID="5fd48c9a-8cd8-4743-8dbf-eb20219c893c"
Sep 17 21:41:38.747672 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:38.747253    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:38.747672 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:38.747253    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:38.747672 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:38.747392    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gxhr" podUID="93119bca-cc51-4b33-964b-63bf4fe5c9e1"
Sep 17 21:41:38.747672 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:38.747259    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:38.747672 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:38.747500    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrbr6" podUID="1e76c503-ec71-47a9-b3cd-63d3b12a439a"
Sep 17 21:41:38.747672 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:38.747576    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmscl" podUID="5fd48c9a-8cd8-4743-8dbf-eb20219c893c"
Sep 17 21:41:40.747946 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:40.747907    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:40.748384 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:40.747924    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:40.748384 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:40.748044    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrbr6" podUID="1e76c503-ec71-47a9-b3cd-63d3b12a439a"
Sep 17 21:41:40.748384 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:40.748057    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:40.748384 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:40.748156    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gxhr" podUID="93119bca-cc51-4b33-964b-63bf4fe5c9e1"
Sep 17 21:41:40.748384 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:40.748262    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmscl" podUID="5fd48c9a-8cd8-4743-8dbf-eb20219c893c"
Sep 17 21:41:42.355378 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:42.355335    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e76c503-ec71-47a9-b3cd-63d3b12a439a-original-pull-secret\") pod \"global-pull-secret-syncer-hrbr6\" (UID: \"1e76c503-ec71-47a9-b3cd-63d3b12a439a\") " pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:42.355736 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:42.355425    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93119bca-cc51-4b33-964b-63bf4fe5c9e1-metrics-certs\") pod \"network-metrics-daemon-5gxhr\" (UID: \"93119bca-cc51-4b33-964b-63bf4fe5c9e1\") " pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:42.355736 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:42.355491    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:41:42.355736 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:42.355504    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:41:42.355736 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:42.355557    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93119bca-cc51-4b33-964b-63bf4fe5c9e1-metrics-certs podName:93119bca-cc51-4b33-964b-63bf4fe5c9e1 nodeName:}" failed. No retries permitted until 2026-09-17 21:41:58.355543678 +0000 UTC m=+34.173869972 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93119bca-cc51-4b33-964b-63bf4fe5c9e1-metrics-certs") pod "network-metrics-daemon-5gxhr" (UID: "93119bca-cc51-4b33-964b-63bf4fe5c9e1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:41:42.355736 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:42.355572    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e76c503-ec71-47a9-b3cd-63d3b12a439a-original-pull-secret podName:1e76c503-ec71-47a9-b3cd-63d3b12a439a nodeName:}" failed. No retries permitted until 2026-09-17 21:41:58.355565835 +0000 UTC m=+34.173892129 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e76c503-ec71-47a9-b3cd-63d3b12a439a-original-pull-secret") pod "global-pull-secret-syncer-hrbr6" (UID: "1e76c503-ec71-47a9-b3cd-63d3b12a439a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:41:42.556923 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:42.556884    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlhnr\" (UniqueName: \"kubernetes.io/projected/5fd48c9a-8cd8-4743-8dbf-eb20219c893c-kube-api-access-mlhnr\") pod \"network-check-target-hmscl\" (UID: \"5fd48c9a-8cd8-4743-8dbf-eb20219c893c\") " pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:42.557065 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:42.557008    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:41:42.557065 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:42.557025    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:41:42.557065 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:42.557037    2564 projected.go:196] Error preparing data for projected volume kube-api-access-mlhnr for pod openshift-network-diagnostics/network-check-target-hmscl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:41:42.557195 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:42.557088    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5fd48c9a-8cd8-4743-8dbf-eb20219c893c-kube-api-access-mlhnr podName:5fd48c9a-8cd8-4743-8dbf-eb20219c893c nodeName:}" failed. No retries permitted until 2026-09-17 21:41:58.557074205 +0000 UTC m=+34.375400495 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlhnr" (UniqueName: "kubernetes.io/projected/5fd48c9a-8cd8-4743-8dbf-eb20219c893c-kube-api-access-mlhnr") pod "network-check-target-hmscl" (UID: "5fd48c9a-8cd8-4743-8dbf-eb20219c893c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:41:42.747255 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:42.747218    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:42.747255 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:42.747250    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:42.747506 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:42.747226    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:42.747506 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:42.747340    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrbr6" podUID="1e76c503-ec71-47a9-b3cd-63d3b12a439a"
Sep 17 21:41:42.747506 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:42.747427    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gxhr" podUID="93119bca-cc51-4b33-964b-63bf4fe5c9e1"
Sep 17 21:41:42.747665 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:42.747512    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmscl" podUID="5fd48c9a-8cd8-4743-8dbf-eb20219c893c"
Sep 17 21:41:44.748219 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:44.748164    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:44.748632 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:44.748231    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:44.748632 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:44.748286    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:44.748632 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:44.748311    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrbr6" podUID="1e76c503-ec71-47a9-b3cd-63d3b12a439a"
Sep 17 21:41:44.748632 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:44.748363    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmscl" podUID="5fd48c9a-8cd8-4743-8dbf-eb20219c893c"
Sep 17 21:41:44.748867 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:44.748406    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gxhr" podUID="93119bca-cc51-4b33-964b-63bf4fe5c9e1"
Sep 17 21:41:46.751844 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:46.751428    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:46.751844 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:46.751554    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:46.751844 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:46.751621    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmscl" podUID="5fd48c9a-8cd8-4743-8dbf-eb20219c893c"
Sep 17 21:41:46.751844 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:46.751547    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrbr6" podUID="1e76c503-ec71-47a9-b3cd-63d3b12a439a"
Sep 17 21:41:46.751844 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:46.751681    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:46.751844 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:46.751754    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gxhr" podUID="93119bca-cc51-4b33-964b-63bf4fe5c9e1"
Sep 17 21:41:46.994808 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:46.994541    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hxcs4" event={"ID":"a8d2b64b-06cf-4a62-b73f-6835eab58938","Type":"ContainerStarted","Data":"11b96a6f2111dc749dc851c1d68a9405b8217e3f6ab4f831a296c16872df2391"}
Sep 17 21:41:46.996566 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:46.996199    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rkgql" event={"ID":"49f3497d-f458-44e9-8846-92afb8ba26eb","Type":"ContainerStarted","Data":"49d3a56dd85a7d8e010aec3a117a3db0c24b722a03dd9ed1148fdb4c56a8ec82"}
Sep 17 21:41:46.997747 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:46.997723    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk" event={"ID":"56fa7d57-fa08-457e-8a11-ca93979cdc8a","Type":"ContainerStarted","Data":"6299ea3272b1b5ddd44dc53525af73f18a78ecabb56ace6e3f82ceaaee6d6495"}
Sep 17 21:41:47.003693 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:47.003665    2564 generic.go:356] "Generic (PLEG): container finished" podID="59d7a806-c3e4-48f4-82df-8bfaa212a9db" containerID="51d38a8c3d6161841e9ecf81f90842feafde106b4119555967b71044a7301940" exitCode=0
Sep 17 21:41:47.003864 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:47.003814    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kjjz4" event={"ID":"59d7a806-c3e4-48f4-82df-8bfaa212a9db","Type":"ContainerDied","Data":"51d38a8c3d6161841e9ecf81f90842feafde106b4119555967b71044a7301940"}
Sep 17 21:41:47.006991 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:47.006969    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdcnw_6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2/ovn-acl-logging/0.log"
Sep 17 21:41:47.008311 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:47.007492    2564 generic.go:356] "Generic (PLEG): container finished" podID="6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2" containerID="0fd8073071194aca8da84f280ec78b3519922374de0dd64479d9bc3e64234de3" exitCode=1
Sep 17 21:41:47.008311 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:47.007587    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw" event={"ID":"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2","Type":"ContainerDied","Data":"0fd8073071194aca8da84f280ec78b3519922374de0dd64479d9bc3e64234de3"}
Sep 17 21:41:47.008311 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:47.007676    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw" event={"ID":"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2","Type":"ContainerStarted","Data":"f683600759f20ae99a11665b96f80c4ed843d982f821139b2eb545b589521db6"}
Sep 17 21:41:47.011116 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:47.011086    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j4fpb" event={"ID":"0cf60ac6-5179-4cee-bbe5-92ab9eb1ad5d","Type":"ContainerStarted","Data":"1b5c99ad90a8318dad5fe49f4f82956018c3d60dae7c7b8802dc1b1c7ebf971e"}
Sep 17 21:41:47.013459 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:47.013435    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fdkhb" event={"ID":"59115160-02ac-43e8-9a62-0c91775ba337","Type":"ContainerStarted","Data":"626f88fc8c7deb4fc8420baeb277760ac36678bfc5b749b20d124a672a2a374a"}
Sep 17 21:41:47.015025 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:47.015004    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7ngpc" event={"ID":"d49ed8ac-69ff-4b6e-ad3a-3f45026b84ae","Type":"ContainerStarted","Data":"0b0df5e766330779743d512358c42a3c727d12cd0858f77bc8aa7b3578bef6a8"}
Sep 17 21:41:47.039345 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:47.039305    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal" podStartSLOduration=21.039290074 podStartE2EDuration="21.039290074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:41:29.967053183 +0000 UTC m=+5.785379499" watchObservedRunningTime="2026-09-17 21:41:47.039290074 +0000 UTC m=+22.857616386"
Sep 17 21:41:47.039450 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:47.039431    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hxcs4" podStartSLOduration=3.412287354 podStartE2EDuration="22.03942563s" totalImagesPullingTime="18.627138276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:41:47.038655913 +0000 UTC m=+22.856982224" watchObservedRunningTime="2026-09-17 21:41:47.03942563 +0000 UTC m=+22.857751942"
Sep 17 21:41:47.114597 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:47.114411    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rkgql" podStartSLOduration=3.561141093 podStartE2EDuration="22.11439585s" totalImagesPullingTime="18.553254757s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:41:47.066936393 +0000 UTC m=+22.885262706" watchObservedRunningTime="2026-09-17 21:41:47.11439585 +0000 UTC m=+22.932722162"
Sep 17 21:41:47.170566 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:47.170521    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-j4fpb" podStartSLOduration=3.6202030819999997 podStartE2EDuration="22.170504983s" totalImagesPullingTime="18.550301901s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:41:47.116470119 +0000 UTC m=+22.934796430" watchObservedRunningTime="2026-09-17 21:41:47.170504983 +0000 UTC m=+22.988831296"
Sep 17 21:41:47.210793 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:47.210735    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fdkhb" podStartSLOduration=3.673528336 podStartE2EDuration="22.210715159s" totalImagesPullingTime="18.537186823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:41:47.171277447 +0000 UTC m=+22.989603758" watchObservedRunningTime="2026-09-17 21:41:47.210715159 +0000 UTC m=+23.029041479"
Sep 17 21:41:47.273575 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:47.273533    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7ngpc" podStartSLOduration=11.346983827 podStartE2EDuration="22.273516305s" totalImagesPullingTime="10.926532478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:41:47.21084667 +0000 UTC m=+23.029172983" watchObservedRunningTime="2026-09-17 21:41:47.273516305 +0000 UTC m=+23.091842616"
Sep 17 21:41:48.019402 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:48.019366    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p4lhk" event={"ID":"bd8e67a3-3131-4932-b561-112a63c7e612","Type":"ContainerStarted","Data":"c39b5f56c5a78ad17777fe821c58d15f84fe4fb8209063130f352ee0923a407d"}
Sep 17 21:41:48.022725 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:48.022704    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdcnw_6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2/ovn-acl-logging/0.log"
Sep 17 21:41:48.023283 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:48.023251    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw" event={"ID":"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2","Type":"ContainerStarted","Data":"469c94c85dde0712daa3755e90146a590ac50d5a2239d32e1eea903e1f158f98"}
Sep 17 21:41:48.023383 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:48.023322    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw" event={"ID":"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2","Type":"ContainerStarted","Data":"4963499efc5079ffe4f66cee658543f559bb5adef8cd6eb79381726ce0323263"}
Sep 17 21:41:48.023383 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:48.023363    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw" event={"ID":"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2","Type":"ContainerStarted","Data":"08f76d0e7e27dbd26c59f9ac1381d13bc113d02c9be967c939b7263cf0000c0a"}
Sep 17 21:41:48.023465 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:48.023391    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw" event={"ID":"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2","Type":"ContainerStarted","Data":"9d0f029d605b83d0b694f1f81adf9c63c6463afec3ee99dc16c2e0b1ed9c9c41"}
Sep 17 21:41:48.354659 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:48.354610    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 21:41:48.689193 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:48.689151    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7ngpc"
Sep 17 21:41:48.689358 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:48.689125    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T21:41:48.354635889Z","UUID":"cc56aacc-ea5c-4a92-985c-714c1d99ec68","Handler":null,"Name":"","Endpoint":""}
Sep 17 21:41:48.690677 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:48.690654    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 21:41:48.690677 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:48.690680    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 21:41:48.747300 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:48.747269    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:48.747300 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:48.747293    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:48.747513 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:48.747294    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:48.747513 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:48.747398    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gxhr" podUID="93119bca-cc51-4b33-964b-63bf4fe5c9e1"
Sep 17 21:41:48.747513 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:48.747488    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrbr6" podUID="1e76c503-ec71-47a9-b3cd-63d3b12a439a"
Sep 17 21:41:48.747632 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:48.747561    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmscl" podUID="5fd48c9a-8cd8-4743-8dbf-eb20219c893c"
Sep 17 21:41:48.990534 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:48.990446    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7ngpc"
Sep 17 21:41:48.991054 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:48.991025    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7ngpc"
Sep 17 21:41:49.027674 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:49.027633    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk" event={"ID":"56fa7d57-fa08-457e-8a11-ca93979cdc8a","Type":"ContainerStarted","Data":"6aa6f5ece0a39e680574434fd5328361020682c9459342ff1dca1449a7cd60e6"}
Sep 17 21:41:49.028754 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:49.028734    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7ngpc"
Sep 17 21:41:49.067836 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:49.067782    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-p4lhk" podStartSLOduration=5.5163698310000004 podStartE2EDuration="24.067767022s" totalImagesPullingTime="18.551397191s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:41:49.049412389 +0000 UTC m=+24.867738702" watchObservedRunningTime="2026-09-17 21:41:49.067767022 +0000 UTC m=+24.886093334"
Sep 17 21:41:50.033188 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:50.033143    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk" event={"ID":"56fa7d57-fa08-457e-8a11-ca93979cdc8a","Type":"ContainerStarted","Data":"cfb11dd501b84595f4df1c84381d4225d792ffd3efc878351d5a6b1e900cce23"}
Sep 17 21:41:50.037549 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:50.037529    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdcnw_6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2/ovn-acl-logging/0.log"
Sep 17 21:41:50.037966 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:50.037936    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw" event={"ID":"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2","Type":"ContainerStarted","Data":"04dd44be6b8e280798a2cf550d91fa8ba083951e4cf0a46ab8f13d788f449f68"}
Sep 17 21:41:50.057677 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:50.057632    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778pk" podStartSLOduration=4.232951714 podStartE2EDuration="25.057616004s" totalImagesPullingTime="20.82466429s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:41:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:41:50.056459978 +0000 UTC m=+25.874786289" watchObservedRunningTime="2026-09-17 21:41:50.057616004 +0000 UTC m=+25.875942315"
Sep 17 21:41:50.749222 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:50.749192    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:50.749377 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:50.749303    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrbr6" podUID="1e76c503-ec71-47a9-b3cd-63d3b12a439a"
Sep 17 21:41:50.749654 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:50.749639    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:50.749751 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:50.749736    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gxhr" podUID="93119bca-cc51-4b33-964b-63bf4fe5c9e1"
Sep 17 21:41:50.749799 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:50.749773    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:50.749844 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:50.749831    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmscl" podUID="5fd48c9a-8cd8-4743-8dbf-eb20219c893c"
Sep 17 21:41:52.044820 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:52.044632    2564 generic.go:356] "Generic (PLEG): container finished" podID="59d7a806-c3e4-48f4-82df-8bfaa212a9db" containerID="d1daf03804786e0e3e96b9a216c07f0fae6900e341d9978404673e8f1d0ab4f1" exitCode=0
Sep 17 21:41:52.045557 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:52.044676    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kjjz4" event={"ID":"59d7a806-c3e4-48f4-82df-8bfaa212a9db","Type":"ContainerDied","Data":"d1daf03804786e0e3e96b9a216c07f0fae6900e341d9978404673e8f1d0ab4f1"}
Sep 17 21:41:52.048784 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:52.048768    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdcnw_6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2/ovn-acl-logging/0.log"
Sep 17 21:41:52.049104 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:52.049081    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw" event={"ID":"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2","Type":"ContainerStarted","Data":"111ed60f118ed2b0955da956dbef70d1b7b564a2f3f9e658ee56d4133a86460f"}
Sep 17 21:41:52.049444 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:52.049410    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:52.049608 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:52.049499    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:52.049608 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:52.049541    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:52.049756 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:52.049677    2564 scope.go:117] "RemoveContainer" containerID="0fd8073071194aca8da84f280ec78b3519922374de0dd64479d9bc3e64234de3"
Sep 17 21:41:52.066276 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:52.066249    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:52.066835 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:52.066817    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:41:52.747937 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:52.747906    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:52.748108 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:52.748045    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gxhr" podUID="93119bca-cc51-4b33-964b-63bf4fe5c9e1"
Sep 17 21:41:52.748165 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:52.748111    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:52.748229 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:52.748192    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmscl" podUID="5fd48c9a-8cd8-4743-8dbf-eb20219c893c"
Sep 17 21:41:52.748276 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:52.748252    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:52.748378 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:52.748358    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrbr6" podUID="1e76c503-ec71-47a9-b3cd-63d3b12a439a"
Sep 17 21:41:53.053069 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:53.053038    2564 generic.go:356] "Generic (PLEG): container finished" podID="59d7a806-c3e4-48f4-82df-8bfaa212a9db" containerID="41ef33094cb4f200872b01ec3422851b661fdd1d05de536d13f31b71075bae77" exitCode=0
Sep 17 21:41:53.053522 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:53.053118    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kjjz4" event={"ID":"59d7a806-c3e4-48f4-82df-8bfaa212a9db","Type":"ContainerDied","Data":"41ef33094cb4f200872b01ec3422851b661fdd1d05de536d13f31b71075bae77"}
Sep 17 21:41:53.057207 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:53.057188    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdcnw_6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2/ovn-acl-logging/0.log"
Sep 17 21:41:53.057570 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:53.057548    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw" event={"ID":"6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2","Type":"ContainerStarted","Data":"f3f320ff34f31f027c7075b209903c7eb0c5aeb8b8e4c8974f8e08d26c885eea"}
Sep 17 21:41:53.101294 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:53.101246    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw" podStartSLOduration=9.462492232 podStartE2EDuration="28.101229024s" totalImagesPullingTime="18.638736792s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:41:53.100724105 +0000 UTC m=+28.919050416" watchObservedRunningTime="2026-09-17 21:41:53.101229024 +0000 UTC m=+28.919555335"
Sep 17 21:41:53.573985 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:53.573954    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5gxhr"]
Sep 17 21:41:53.574237 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:53.574077    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:53.574237 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:53.574201    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gxhr" podUID="93119bca-cc51-4b33-964b-63bf4fe5c9e1"
Sep 17 21:41:53.577253 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:53.577227    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hmscl"]
Sep 17 21:41:53.577375 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:53.577345    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:53.577511 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:53.577444    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmscl" podUID="5fd48c9a-8cd8-4743-8dbf-eb20219c893c"
Sep 17 21:41:53.578055 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:53.578018    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hrbr6"]
Sep 17 21:41:53.578143 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:53.578128    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:53.578252 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:53.578234    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrbr6" podUID="1e76c503-ec71-47a9-b3cd-63d3b12a439a"
Sep 17 21:41:54.062063 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:54.061976    2564 generic.go:356] "Generic (PLEG): container finished" podID="59d7a806-c3e4-48f4-82df-8bfaa212a9db" containerID="56af712b4a857645e62cc46766b95034026783b088b8b0f4d50f930d1bcd7732" exitCode=0
Sep 17 21:41:54.062433 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:54.062054    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kjjz4" event={"ID":"59d7a806-c3e4-48f4-82df-8bfaa212a9db","Type":"ContainerDied","Data":"56af712b4a857645e62cc46766b95034026783b088b8b0f4d50f930d1bcd7732"}
Sep 17 21:41:54.749359 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:54.749321    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:54.749517 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:54.749440    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmscl" podUID="5fd48c9a-8cd8-4743-8dbf-eb20219c893c"
Sep 17 21:41:54.749517 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:54.749497    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:54.749619 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:54.749601    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gxhr" podUID="93119bca-cc51-4b33-964b-63bf4fe5c9e1"
Sep 17 21:41:55.747409 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:55.747381    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:55.747806 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:55.747506    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrbr6" podUID="1e76c503-ec71-47a9-b3cd-63d3b12a439a"
Sep 17 21:41:56.747903 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:56.747862    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:56.748387 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:56.747863    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:56.748387 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:56.748008    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmscl" podUID="5fd48c9a-8cd8-4743-8dbf-eb20219c893c"
Sep 17 21:41:56.748387 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:56.748091    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gxhr" podUID="93119bca-cc51-4b33-964b-63bf4fe5c9e1"
Sep 17 21:41:57.747942 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:57.747907    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:57.748478 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:57.748029    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrbr6" podUID="1e76c503-ec71-47a9-b3cd-63d3b12a439a"
Sep 17 21:41:58.378839 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:58.378786    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93119bca-cc51-4b33-964b-63bf4fe5c9e1-metrics-certs\") pod \"network-metrics-daemon-5gxhr\" (UID: \"93119bca-cc51-4b33-964b-63bf4fe5c9e1\") " pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:58.379024 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:58.378857    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e76c503-ec71-47a9-b3cd-63d3b12a439a-original-pull-secret\") pod \"global-pull-secret-syncer-hrbr6\" (UID: \"1e76c503-ec71-47a9-b3cd-63d3b12a439a\") " pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:58.379024 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:58.378957    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:41:58.379024 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:58.378969    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:41:58.379124 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:58.379030    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93119bca-cc51-4b33-964b-63bf4fe5c9e1-metrics-certs podName:93119bca-cc51-4b33-964b-63bf4fe5c9e1 nodeName:}" failed. No retries permitted until 2026-09-17 21:42:30.379013461 +0000 UTC m=+66.197339755 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93119bca-cc51-4b33-964b-63bf4fe5c9e1-metrics-certs") pod "network-metrics-daemon-5gxhr" (UID: "93119bca-cc51-4b33-964b-63bf4fe5c9e1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:41:58.379124 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:58.379045    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e76c503-ec71-47a9-b3cd-63d3b12a439a-original-pull-secret podName:1e76c503-ec71-47a9-b3cd-63d3b12a439a nodeName:}" failed. No retries permitted until 2026-09-17 21:42:30.379038427 +0000 UTC m=+66.197364720 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e76c503-ec71-47a9-b3cd-63d3b12a439a-original-pull-secret") pod "global-pull-secret-syncer-hrbr6" (UID: "1e76c503-ec71-47a9-b3cd-63d3b12a439a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:41:58.580966 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:58.580881    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlhnr\" (UniqueName: \"kubernetes.io/projected/5fd48c9a-8cd8-4743-8dbf-eb20219c893c-kube-api-access-mlhnr\") pod \"network-check-target-hmscl\" (UID: \"5fd48c9a-8cd8-4743-8dbf-eb20219c893c\") " pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:58.581118 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:58.581080    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:41:58.581118 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:58.581106    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:41:58.581118 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:58.581119    2564 projected.go:196] Error preparing data for projected volume kube-api-access-mlhnr for pod openshift-network-diagnostics/network-check-target-hmscl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:41:58.581258 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:58.581198    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5fd48c9a-8cd8-4743-8dbf-eb20219c893c-kube-api-access-mlhnr podName:5fd48c9a-8cd8-4743-8dbf-eb20219c893c nodeName:}" failed. No retries permitted until 2026-09-17 21:42:30.581159877 +0000 UTC m=+66.399486183 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlhnr" (UniqueName: "kubernetes.io/projected/5fd48c9a-8cd8-4743-8dbf-eb20219c893c-kube-api-access-mlhnr") pod "network-check-target-hmscl" (UID: "5fd48c9a-8cd8-4743-8dbf-eb20219c893c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:41:58.747649 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:58.747614    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:41:58.747841 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:58.747628    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:41:58.747841 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:58.747770    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gxhr" podUID="93119bca-cc51-4b33-964b-63bf4fe5c9e1"
Sep 17 21:41:58.747841 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:58.747828    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmscl" podUID="5fd48c9a-8cd8-4743-8dbf-eb20219c893c"
Sep 17 21:41:59.747677 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:41:59.747636    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:41:59.748147 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:41:59.747757    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrbr6" podUID="1e76c503-ec71-47a9-b3cd-63d3b12a439a"
Sep 17 21:42:00.519773 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.519433    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeReady"
Sep 17 21:42:00.519859 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.519776    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 21:42:00.574853 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.574823    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hts6f"]
Sep 17 21:42:00.589365 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.589342    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zht2d"]
Sep 17 21:42:00.589497 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.589477    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hts6f"
Sep 17 21:42:00.592388 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.592362    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vmz7f\""
Sep 17 21:42:00.592732 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.592717    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 21:42:00.592821 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.592755    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 21:42:00.593020 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.592998    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 21:42:00.605322 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.605301    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hts6f"]
Sep 17 21:42:00.605425 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.605345    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zht2d"]
Sep 17 21:42:00.605425 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.605307    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zht2d"
Sep 17 21:42:00.607796 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.607776    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 21:42:00.607899 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.607801    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 21:42:00.607967 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.607928    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-csvq4\""
Sep 17 21:42:00.694433 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.694395    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ae24773-6451-4cfd-a768-8548a684e09f-metrics-tls\") pod \"dns-default-zht2d\" (UID: \"3ae24773-6451-4cfd-a768-8548a684e09f\") " pod="openshift-dns/dns-default-zht2d"
Sep 17 21:42:00.694602 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.694451    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d38d2d6b-ead0-4052-ae97-d23439d4b9c9-cert\") pod \"ingress-canary-hts6f\" (UID: \"d38d2d6b-ead0-4052-ae97-d23439d4b9c9\") " pod="openshift-ingress-canary/ingress-canary-hts6f"
Sep 17 21:42:00.694602 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.694481    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ae24773-6451-4cfd-a768-8548a684e09f-tmp-dir\") pod \"dns-default-zht2d\" (UID: \"3ae24773-6451-4cfd-a768-8548a684e09f\") " pod="openshift-dns/dns-default-zht2d"
Sep 17 21:42:00.694602 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.694527    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmh55\" (UniqueName: \"kubernetes.io/projected/3ae24773-6451-4cfd-a768-8548a684e09f-kube-api-access-lmh55\") pod \"dns-default-zht2d\" (UID: \"3ae24773-6451-4cfd-a768-8548a684e09f\") " pod="openshift-dns/dns-default-zht2d"
Sep 17 21:42:00.694602 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.694588    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhrtb\" (UniqueName: \"kubernetes.io/projected/d38d2d6b-ead0-4052-ae97-d23439d4b9c9-kube-api-access-mhrtb\") pod \"ingress-canary-hts6f\" (UID: \"d38d2d6b-ead0-4052-ae97-d23439d4b9c9\") " pod="openshift-ingress-canary/ingress-canary-hts6f"
Sep 17 21:42:00.694751 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.694618    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae24773-6451-4cfd-a768-8548a684e09f-config-volume\") pod \"dns-default-zht2d\" (UID: \"3ae24773-6451-4cfd-a768-8548a684e09f\") " pod="openshift-dns/dns-default-zht2d"
Sep 17 21:42:00.747287 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.747251    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:42:00.747438 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.747257    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:42:00.750226 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.750147    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 21:42:00.750226 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.750157    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-44pmn\""
Sep 17 21:42:00.750226 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.750156    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 21:42:00.750226 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.750199    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 21:42:00.750612 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.750273    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7sc2s\""
Sep 17 21:42:00.795285 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.795252    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d38d2d6b-ead0-4052-ae97-d23439d4b9c9-cert\") pod \"ingress-canary-hts6f\" (UID: \"d38d2d6b-ead0-4052-ae97-d23439d4b9c9\") " pod="openshift-ingress-canary/ingress-canary-hts6f"
Sep 17 21:42:00.795285 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.795284    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ae24773-6451-4cfd-a768-8548a684e09f-tmp-dir\") pod \"dns-default-zht2d\" (UID: \"3ae24773-6451-4cfd-a768-8548a684e09f\") " pod="openshift-dns/dns-default-zht2d"
Sep 17 21:42:00.795451 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.795304    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmh55\" (UniqueName: \"kubernetes.io/projected/3ae24773-6451-4cfd-a768-8548a684e09f-kube-api-access-lmh55\") pod \"dns-default-zht2d\" (UID: \"3ae24773-6451-4cfd-a768-8548a684e09f\") " pod="openshift-dns/dns-default-zht2d"
Sep 17 21:42:00.795451 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.795433    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhrtb\" (UniqueName: \"kubernetes.io/projected/d38d2d6b-ead0-4052-ae97-d23439d4b9c9-kube-api-access-mhrtb\") pod \"ingress-canary-hts6f\" (UID: \"d38d2d6b-ead0-4052-ae97-d23439d4b9c9\") " pod="openshift-ingress-canary/ingress-canary-hts6f"
Sep 17 21:42:00.795543 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.795469    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae24773-6451-4cfd-a768-8548a684e09f-config-volume\") pod \"dns-default-zht2d\" (UID: \"3ae24773-6451-4cfd-a768-8548a684e09f\") " pod="openshift-dns/dns-default-zht2d"
Sep 17 21:42:00.795602 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.795581    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ae24773-6451-4cfd-a768-8548a684e09f-tmp-dir\") pod \"dns-default-zht2d\" (UID: \"3ae24773-6451-4cfd-a768-8548a684e09f\") " pod="openshift-dns/dns-default-zht2d"
Sep 17 21:42:00.795655 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.795636    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ae24773-6451-4cfd-a768-8548a684e09f-metrics-tls\") pod \"dns-default-zht2d\" (UID: \"3ae24773-6451-4cfd-a768-8548a684e09f\") " pod="openshift-dns/dns-default-zht2d"
Sep 17 21:42:00.795740 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:00.795724    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:42:00.795809 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:00.795790    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae24773-6451-4cfd-a768-8548a684e09f-metrics-tls podName:3ae24773-6451-4cfd-a768-8548a684e09f nodeName:}" failed. No retries permitted until 2026-09-17 21:42:01.295777312 +0000 UTC m=+37.114103602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ae24773-6451-4cfd-a768-8548a684e09f-metrics-tls") pod "dns-default-zht2d" (UID: "3ae24773-6451-4cfd-a768-8548a684e09f") : secret "dns-default-metrics-tls" not found
Sep 17 21:42:00.795950 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.795930    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae24773-6451-4cfd-a768-8548a684e09f-config-volume\") pod \"dns-default-zht2d\" (UID: \"3ae24773-6451-4cfd-a768-8548a684e09f\") " pod="openshift-dns/dns-default-zht2d"
Sep 17 21:42:00.799641 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.799623    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d38d2d6b-ead0-4052-ae97-d23439d4b9c9-cert\") pod \"ingress-canary-hts6f\" (UID: \"d38d2d6b-ead0-4052-ae97-d23439d4b9c9\") " pod="openshift-ingress-canary/ingress-canary-hts6f"
Sep 17 21:42:00.806650 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.806627    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmh55\" (UniqueName: \"kubernetes.io/projected/3ae24773-6451-4cfd-a768-8548a684e09f-kube-api-access-lmh55\") pod \"dns-default-zht2d\" (UID: \"3ae24773-6451-4cfd-a768-8548a684e09f\") " pod="openshift-dns/dns-default-zht2d"
Sep 17 21:42:00.806932 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.806912    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhrtb\" (UniqueName: \"kubernetes.io/projected/d38d2d6b-ead0-4052-ae97-d23439d4b9c9-kube-api-access-mhrtb\") pod \"ingress-canary-hts6f\" (UID: \"d38d2d6b-ead0-4052-ae97-d23439d4b9c9\") " pod="openshift-ingress-canary/ingress-canary-hts6f"
Sep 17 21:42:00.899151 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:00.899113    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hts6f"
Sep 17 21:42:01.081097 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:01.081065    2564 generic.go:356] "Generic (PLEG): container finished" podID="59d7a806-c3e4-48f4-82df-8bfaa212a9db" containerID="8ab47b58750741599c95c81490df07aa3c5ad1c265dec6743d7eae74b1a3e22e" exitCode=0
Sep 17 21:42:01.081294 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:01.081125    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kjjz4" event={"ID":"59d7a806-c3e4-48f4-82df-8bfaa212a9db","Type":"ContainerDied","Data":"8ab47b58750741599c95c81490df07aa3c5ad1c265dec6743d7eae74b1a3e22e"}
Sep 17 21:42:01.226608 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:01.226541    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hts6f"]
Sep 17 21:42:01.231741 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:42:01.231716    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38d2d6b_ead0_4052_ae97_d23439d4b9c9.slice/crio-b7cb6c4700ed532aed0d6f88c0fdb985db399864f558b4259e04f7b273ff9af9 WatchSource:0}: Error finding container b7cb6c4700ed532aed0d6f88c0fdb985db399864f558b4259e04f7b273ff9af9: Status 404 returned error can't find the container with id b7cb6c4700ed532aed0d6f88c0fdb985db399864f558b4259e04f7b273ff9af9
Sep 17 21:42:01.299830 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:01.299650    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ae24773-6451-4cfd-a768-8548a684e09f-metrics-tls\") pod \"dns-default-zht2d\" (UID: \"3ae24773-6451-4cfd-a768-8548a684e09f\") " pod="openshift-dns/dns-default-zht2d"
Sep 17 21:42:01.299981 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:01.299810    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:42:01.299981 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:01.299941    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae24773-6451-4cfd-a768-8548a684e09f-metrics-tls podName:3ae24773-6451-4cfd-a768-8548a684e09f nodeName:}" failed. No retries permitted until 2026-09-17 21:42:02.29992143 +0000 UTC m=+38.118247740 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ae24773-6451-4cfd-a768-8548a684e09f-metrics-tls") pod "dns-default-zht2d" (UID: "3ae24773-6451-4cfd-a768-8548a684e09f") : secret "dns-default-metrics-tls" not found
Sep 17 21:42:01.747556 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:01.747522    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:42:01.751386 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:01.751365    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 21:42:02.087619 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:02.087263    2564 generic.go:356] "Generic (PLEG): container finished" podID="59d7a806-c3e4-48f4-82df-8bfaa212a9db" containerID="f26d449d04335f5f6343e41b7a0e3ad1b4a15dc57e18b6b55a26483b3c3783df" exitCode=0
Sep 17 21:42:02.087619 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:02.087357    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kjjz4" event={"ID":"59d7a806-c3e4-48f4-82df-8bfaa212a9db","Type":"ContainerDied","Data":"f26d449d04335f5f6343e41b7a0e3ad1b4a15dc57e18b6b55a26483b3c3783df"}
Sep 17 21:42:02.088743 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:02.088721    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hts6f" event={"ID":"d38d2d6b-ead0-4052-ae97-d23439d4b9c9","Type":"ContainerStarted","Data":"b7cb6c4700ed532aed0d6f88c0fdb985db399864f558b4259e04f7b273ff9af9"}
Sep 17 21:42:02.308015 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:02.307983    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ae24773-6451-4cfd-a768-8548a684e09f-metrics-tls\") pod \"dns-default-zht2d\" (UID: \"3ae24773-6451-4cfd-a768-8548a684e09f\") " pod="openshift-dns/dns-default-zht2d"
Sep 17 21:42:02.308207 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:02.308097    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:42:02.308207 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:02.308154    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae24773-6451-4cfd-a768-8548a684e09f-metrics-tls podName:3ae24773-6451-4cfd-a768-8548a684e09f nodeName:}" failed. No retries permitted until 2026-09-17 21:42:04.308141224 +0000 UTC m=+40.126467514 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ae24773-6451-4cfd-a768-8548a684e09f-metrics-tls") pod "dns-default-zht2d" (UID: "3ae24773-6451-4cfd-a768-8548a684e09f") : secret "dns-default-metrics-tls" not found
Sep 17 21:42:03.095394 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:03.095361    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kjjz4" event={"ID":"59d7a806-c3e4-48f4-82df-8bfaa212a9db","Type":"ContainerStarted","Data":"c6d2654024dac4780e61c2275b0b4f5a06e67e2f3f587dd65052c47306080dcb"}
Sep 17 21:42:03.120941 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:03.120877    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kjjz4" podStartSLOduration=7.120620528 podStartE2EDuration="38.120860294s" totalImagesPullingTime="31.000239766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:41:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:42:03.120591844 +0000 UTC m=+38.938918159" watchObservedRunningTime="2026-09-17 21:42:03.120860294 +0000 UTC m=+38.939186605"
Sep 17 21:42:04.099726 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:04.099669    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hts6f" event={"ID":"d38d2d6b-ead0-4052-ae97-d23439d4b9c9","Type":"ContainerStarted","Data":"5a28abf889867e333e3b942c5d15acb65f3d344a2a367bfa3aada63009aaf7ea"}
Sep 17 21:42:04.116907 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:04.116783    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hts6f" podStartSLOduration=1.45577383 podStartE2EDuration="4.116764102s" totalImagesPullingTime="2.660990272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:42:04.115828132 +0000 UTC m=+39.934154442" watchObservedRunningTime="2026-09-17 21:42:04.116764102 +0000 UTC m=+39.935090415"
Sep 17 21:42:04.323519 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:04.323426    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ae24773-6451-4cfd-a768-8548a684e09f-metrics-tls\") pod \"dns-default-zht2d\" (UID: \"3ae24773-6451-4cfd-a768-8548a684e09f\") " pod="openshift-dns/dns-default-zht2d"
Sep 17 21:42:04.323672 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:04.323545    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:42:04.323672 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:04.323604    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae24773-6451-4cfd-a768-8548a684e09f-metrics-tls podName:3ae24773-6451-4cfd-a768-8548a684e09f nodeName:}" failed. No retries permitted until 2026-09-17 21:42:08.32359117 +0000 UTC m=+44.141917460 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ae24773-6451-4cfd-a768-8548a684e09f-metrics-tls") pod "dns-default-zht2d" (UID: "3ae24773-6451-4cfd-a768-8548a684e09f") : secret "dns-default-metrics-tls" not found
Sep 17 21:42:05.528488 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:05.528449    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sdnkh"]
Sep 17 21:42:05.551228 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:05.551199    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:05.551479 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:05.551421    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sdnkh"]
Sep 17 21:42:05.554647 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:05.554625    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5wn52\""
Sep 17 21:42:05.554792 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:05.554651    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 21:42:05.554850 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:05.554843    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 21:42:05.555765 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:05.555748    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 21:42:05.557609 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:05.557587    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 21:42:05.733009 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:05.732974    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:05.733009 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:05.733010    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-data-volume\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:05.733224 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:05.733035    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8nrg\" (UniqueName: \"kubernetes.io/projected/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-kube-api-access-d8nrg\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:05.733224 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:05.733100    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:05.733224 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:05.733152    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-crio-socket\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:05.834455 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:05.834361    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-crio-socket\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:05.834455 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:05.834426    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:05.834656 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:05.834485    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:42:05.834656 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:05.834535    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-kube-rbac-proxy-cm podName:87a36c1f-0c4f-4875-8d1e-f562ac1a19a2 nodeName:}" failed. No retries permitted until 2026-09-17 21:42:06.334521932 +0000 UTC m=+42.152848235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sdnkh" (UID: "87a36c1f-0c4f-4875-8d1e-f562ac1a19a2") : configmap "kube-rbac-proxy" not found
Sep 17 21:42:05.834656 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:05.834547    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-data-volume\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:05.834656 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:05.834564    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8nrg\" (UniqueName: \"kubernetes.io/projected/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-kube-api-access-d8nrg\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:05.834656 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:05.834587    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:05.834656 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:05.834624    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-crio-socket\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:05.834910 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:05.834682    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:42:05.834910 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:05.834717    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-insights-runtime-extractor-tls podName:87a36c1f-0c4f-4875-8d1e-f562ac1a19a2 nodeName:}" failed. No retries permitted until 2026-09-17 21:42:06.334708434 +0000 UTC m=+42.153034728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sdnkh" (UID: "87a36c1f-0c4f-4875-8d1e-f562ac1a19a2") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:42:05.834910 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:05.834849    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-data-volume\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:05.855542 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:05.855511    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8nrg\" (UniqueName: \"kubernetes.io/projected/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-kube-api-access-d8nrg\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:06.339010 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:06.338971    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:06.339197 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:06.339016    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:06.339197 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:06.339099    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:42:06.339197 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:06.339163    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:42:06.339197 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:06.339188    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-kube-rbac-proxy-cm podName:87a36c1f-0c4f-4875-8d1e-f562ac1a19a2 nodeName:}" failed. No retries permitted until 2026-09-17 21:42:07.339149494 +0000 UTC m=+43.157475797 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sdnkh" (UID: "87a36c1f-0c4f-4875-8d1e-f562ac1a19a2") : configmap "kube-rbac-proxy" not found
Sep 17 21:42:06.339337 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:06.339237    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-insights-runtime-extractor-tls podName:87a36c1f-0c4f-4875-8d1e-f562ac1a19a2 nodeName:}" failed. No retries permitted until 2026-09-17 21:42:07.339226256 +0000 UTC m=+43.157552559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sdnkh" (UID: "87a36c1f-0c4f-4875-8d1e-f562ac1a19a2") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:42:06.972708 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:06.972672    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-z2x7n"]
Sep 17 21:42:06.994776 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:06.994740    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-z2x7n"
Sep 17 21:42:07.004434 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:07.004410    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 21:42:07.004688 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:07.004667    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-z2x7n"]
Sep 17 21:42:07.007603 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:07.007583    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-nlvhn\""
Sep 17 21:42:07.008005 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:07.007990    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 21:42:07.050315 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:07.050116    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fdkhb_59115160-02ac-43e8-9a62-0c91775ba337/dns-node-resolver/0.log"
Sep 17 21:42:07.145790 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:07.145762    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr7b7\" (UniqueName: \"kubernetes.io/projected/5965f2cf-8d5f-4c57-bd7b-ba79842632d2-kube-api-access-jr7b7\") pod \"migrator-76794cbd6c-z2x7n\" (UID: \"5965f2cf-8d5f-4c57-bd7b-ba79842632d2\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-z2x7n"
Sep 17 21:42:07.246442 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:07.246411    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7b7\" (UniqueName: \"kubernetes.io/projected/5965f2cf-8d5f-4c57-bd7b-ba79842632d2-kube-api-access-jr7b7\") pod \"migrator-76794cbd6c-z2x7n\" (UID: \"5965f2cf-8d5f-4c57-bd7b-ba79842632d2\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-z2x7n"
Sep 17 21:42:07.263888 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:07.263849    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr7b7\" (UniqueName: \"kubernetes.io/projected/5965f2cf-8d5f-4c57-bd7b-ba79842632d2-kube-api-access-jr7b7\") pod \"migrator-76794cbd6c-z2x7n\" (UID: \"5965f2cf-8d5f-4c57-bd7b-ba79842632d2\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-z2x7n"
Sep 17 21:42:07.302827 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:07.302749    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-z2x7n"
Sep 17 21:42:07.347103 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:07.347059    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:07.347299 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:07.347131    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:07.347299 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:07.347225    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:42:07.347299 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:07.347265    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:42:07.347408 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:07.347314    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-kube-rbac-proxy-cm podName:87a36c1f-0c4f-4875-8d1e-f562ac1a19a2 nodeName:}" failed. No retries permitted until 2026-09-17 21:42:09.347293742 +0000 UTC m=+45.165620049 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sdnkh" (UID: "87a36c1f-0c4f-4875-8d1e-f562ac1a19a2") : configmap "kube-rbac-proxy" not found
Sep 17 21:42:07.347408 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:07.347333    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-insights-runtime-extractor-tls podName:87a36c1f-0c4f-4875-8d1e-f562ac1a19a2 nodeName:}" failed. No retries permitted until 2026-09-17 21:42:09.347323565 +0000 UTC m=+45.165649858 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sdnkh" (UID: "87a36c1f-0c4f-4875-8d1e-f562ac1a19a2") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:42:07.604087 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:07.604007    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-z2x7n"]
Sep 17 21:42:07.607706 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:42:07.607679    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5965f2cf_8d5f_4c57_bd7b_ba79842632d2.slice/crio-caccd00547182af1c7400003394e6ab326907b15e200889cf0f5f9671ded1751 WatchSource:0}: Error finding container caccd00547182af1c7400003394e6ab326907b15e200889cf0f5f9671ded1751: Status 404 returned error can't find the container with id caccd00547182af1c7400003394e6ab326907b15e200889cf0f5f9671ded1751
Sep 17 21:42:07.978641 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:07.978608    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-5hdsn"]
Sep 17 21:42:08.010525 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:08.010500    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-5hdsn"
Sep 17 21:42:08.018689 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:08.018671    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 21:42:08.018777 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:08.018675    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 21:42:08.022045 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:08.022030    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 21:42:08.022214 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:08.022202    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 21:42:08.024083 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:08.024071    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ntvq6\""
Sep 17 21:42:08.029801 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:08.029777    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-5hdsn"]
Sep 17 21:42:08.065282 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:08.065262    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rkgql_49f3497d-f458-44e9-8846-92afb8ba26eb/node-ca/0.log"
Sep 17 21:42:08.113058 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:08.113028    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-z2x7n" event={"ID":"5965f2cf-8d5f-4c57-bd7b-ba79842632d2","Type":"ContainerStarted","Data":"caccd00547182af1c7400003394e6ab326907b15e200889cf0f5f9671ded1751"}
Sep 17 21:42:08.153411 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:08.153375    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/82fa7360-125b-4ec6-b5ce-d76daa51e528-signing-key\") pod \"service-ca-55f6fc7b86-5hdsn\" (UID: \"82fa7360-125b-4ec6-b5ce-d76daa51e528\") " pod="openshift-service-ca/service-ca-55f6fc7b86-5hdsn"
Sep 17 21:42:08.153624 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:08.153429    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/82fa7360-125b-4ec6-b5ce-d76daa51e528-signing-cabundle\") pod \"service-ca-55f6fc7b86-5hdsn\" (UID: \"82fa7360-125b-4ec6-b5ce-d76daa51e528\") " pod="openshift-service-ca/service-ca-55f6fc7b86-5hdsn"
Sep 17 21:42:08.153624 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:08.153499    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xkws\" (UniqueName: \"kubernetes.io/projected/82fa7360-125b-4ec6-b5ce-d76daa51e528-kube-api-access-2xkws\") pod \"service-ca-55f6fc7b86-5hdsn\" (UID: \"82fa7360-125b-4ec6-b5ce-d76daa51e528\") " pod="openshift-service-ca/service-ca-55f6fc7b86-5hdsn"
Sep 17 21:42:08.254895 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:08.254807    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xkws\" (UniqueName: \"kubernetes.io/projected/82fa7360-125b-4ec6-b5ce-d76daa51e528-kube-api-access-2xkws\") pod \"service-ca-55f6fc7b86-5hdsn\" (UID: \"82fa7360-125b-4ec6-b5ce-d76daa51e528\") " pod="openshift-service-ca/service-ca-55f6fc7b86-5hdsn"
Sep 17 21:42:08.255052 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:08.254922    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/82fa7360-125b-4ec6-b5ce-d76daa51e528-signing-key\") pod \"service-ca-55f6fc7b86-5hdsn\" (UID: \"82fa7360-125b-4ec6-b5ce-d76daa51e528\") " pod="openshift-service-ca/service-ca-55f6fc7b86-5hdsn"
Sep 17 21:42:08.255052 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:08.254970    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/82fa7360-125b-4ec6-b5ce-d76daa51e528-signing-cabundle\") pod \"service-ca-55f6fc7b86-5hdsn\" (UID: \"82fa7360-125b-4ec6-b5ce-d76daa51e528\") " pod="openshift-service-ca/service-ca-55f6fc7b86-5hdsn"
Sep 17 21:42:08.255674 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:08.255650    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/82fa7360-125b-4ec6-b5ce-d76daa51e528-signing-cabundle\") pod \"service-ca-55f6fc7b86-5hdsn\" (UID: \"82fa7360-125b-4ec6-b5ce-d76daa51e528\") " pod="openshift-service-ca/service-ca-55f6fc7b86-5hdsn"
Sep 17 21:42:08.258069 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:08.258046    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/82fa7360-125b-4ec6-b5ce-d76daa51e528-signing-key\") pod \"service-ca-55f6fc7b86-5hdsn\" (UID: \"82fa7360-125b-4ec6-b5ce-d76daa51e528\") " pod="openshift-service-ca/service-ca-55f6fc7b86-5hdsn"
Sep 17 21:42:08.274761 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:08.274729    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xkws\" (UniqueName: \"kubernetes.io/projected/82fa7360-125b-4ec6-b5ce-d76daa51e528-kube-api-access-2xkws\") pod \"service-ca-55f6fc7b86-5hdsn\" (UID: \"82fa7360-125b-4ec6-b5ce-d76daa51e528\") " pod="openshift-service-ca/service-ca-55f6fc7b86-5hdsn"
Sep 17 21:42:08.319378 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:08.319342    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-5hdsn"
Sep 17 21:42:08.356390 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:08.356363    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ae24773-6451-4cfd-a768-8548a684e09f-metrics-tls\") pod \"dns-default-zht2d\" (UID: \"3ae24773-6451-4cfd-a768-8548a684e09f\") " pod="openshift-dns/dns-default-zht2d"
Sep 17 21:42:08.356531 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:08.356508    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:42:08.356616 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:08.356604    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae24773-6451-4cfd-a768-8548a684e09f-metrics-tls podName:3ae24773-6451-4cfd-a768-8548a684e09f nodeName:}" failed. No retries permitted until 2026-09-17 21:42:16.356581954 +0000 UTC m=+52.174908267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ae24773-6451-4cfd-a768-8548a684e09f-metrics-tls") pod "dns-default-zht2d" (UID: "3ae24773-6451-4cfd-a768-8548a684e09f") : secret "dns-default-metrics-tls" not found
Sep 17 21:42:08.615606 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:08.615582    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-5hdsn"]
Sep 17 21:42:08.617707 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:42:08.617679    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82fa7360_125b_4ec6_b5ce_d76daa51e528.slice/crio-913f6a5f6a7e4966b46cc8a9f9a5b0a79cbd610658dd6259e75b70eadba7321f WatchSource:0}: Error finding container 913f6a5f6a7e4966b46cc8a9f9a5b0a79cbd610658dd6259e75b70eadba7321f: Status 404 returned error can't find the container with id 913f6a5f6a7e4966b46cc8a9f9a5b0a79cbd610658dd6259e75b70eadba7321f
Sep 17 21:42:08.638741 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:08.638719    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hts6f_d38d2d6b-ead0-4052-ae97-d23439d4b9c9/serve-healthcheck-canary/0.log"
Sep 17 21:42:09.117090 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:09.117059    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-5hdsn" event={"ID":"82fa7360-125b-4ec6-b5ce-d76daa51e528","Type":"ContainerStarted","Data":"913f6a5f6a7e4966b46cc8a9f9a5b0a79cbd610658dd6259e75b70eadba7321f"}
Sep 17 21:42:09.363916 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:09.363885    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:09.364041 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:09.363943    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:09.364041 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:09.364016    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:42:09.364193 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:09.364082    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:42:09.364193 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:09.364124    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-kube-rbac-proxy-cm podName:87a36c1f-0c4f-4875-8d1e-f562ac1a19a2 nodeName:}" failed. No retries permitted until 2026-09-17 21:42:13.364102933 +0000 UTC m=+49.182429238 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sdnkh" (UID: "87a36c1f-0c4f-4875-8d1e-f562ac1a19a2") : configmap "kube-rbac-proxy" not found
Sep 17 21:42:09.364193 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:09.364145    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-insights-runtime-extractor-tls podName:87a36c1f-0c4f-4875-8d1e-f562ac1a19a2 nodeName:}" failed. No retries permitted until 2026-09-17 21:42:13.364134976 +0000 UTC m=+49.182461300 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sdnkh" (UID: "87a36c1f-0c4f-4875-8d1e-f562ac1a19a2") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:42:10.121981 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:10.121934    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-z2x7n" event={"ID":"5965f2cf-8d5f-4c57-bd7b-ba79842632d2","Type":"ContainerStarted","Data":"e70237b072d75205efd57ef2c8335fe84cde938895282e2806b746485573778b"}
Sep 17 21:42:10.122431 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:10.122019    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-z2x7n" event={"ID":"5965f2cf-8d5f-4c57-bd7b-ba79842632d2","Type":"ContainerStarted","Data":"399325858d92897d0357a0a9f50dfed5cc8dedbd36be8712eadbfd15aa6d75e2"}
Sep 17 21:42:10.140348 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:10.140291    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-z2x7n" podStartSLOduration=2.399268235 podStartE2EDuration="4.140272684s" totalImagesPullingTime="1.741004449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:42:10.139613649 +0000 UTC m=+45.957939962" watchObservedRunningTime="2026-09-17 21:42:10.140272684 +0000 UTC m=+45.958598999"
Sep 17 21:42:11.125637 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:11.125601    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-5hdsn" event={"ID":"82fa7360-125b-4ec6-b5ce-d76daa51e528","Type":"ContainerStarted","Data":"f2ba9b5446f5e30b4bec798d86fde739bed595882a9cfbfc366a624038dd908e"}
Sep 17 21:42:11.142867 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:11.142812    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-5hdsn" podStartSLOduration=1.82236633 podStartE2EDuration="4.142789959s" totalImagesPullingTime="2.320423629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:42:11.142619934 +0000 UTC m=+46.960946247" watchObservedRunningTime="2026-09-17 21:42:11.142789959 +0000 UTC m=+46.961116269"
Sep 17 21:42:13.395891 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:13.395850    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:13.396341 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:13.395906    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:13.396341 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:13.395974    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:42:13.396341 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:13.396022    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:42:13.396341 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:13.396038    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-kube-rbac-proxy-cm podName:87a36c1f-0c4f-4875-8d1e-f562ac1a19a2 nodeName:}" failed. No retries permitted until 2026-09-17 21:42:21.396024566 +0000 UTC m=+57.214350869 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sdnkh" (UID: "87a36c1f-0c4f-4875-8d1e-f562ac1a19a2") : configmap "kube-rbac-proxy" not found
Sep 17 21:42:13.396341 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:13.396076    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-insights-runtime-extractor-tls podName:87a36c1f-0c4f-4875-8d1e-f562ac1a19a2 nodeName:}" failed. No retries permitted until 2026-09-17 21:42:21.396061491 +0000 UTC m=+57.214387781 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sdnkh" (UID: "87a36c1f-0c4f-4875-8d1e-f562ac1a19a2") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:42:16.416623 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:16.416577    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ae24773-6451-4cfd-a768-8548a684e09f-metrics-tls\") pod \"dns-default-zht2d\" (UID: \"3ae24773-6451-4cfd-a768-8548a684e09f\") " pod="openshift-dns/dns-default-zht2d"
Sep 17 21:42:16.419062 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:16.419033    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ae24773-6451-4cfd-a768-8548a684e09f-metrics-tls\") pod \"dns-default-zht2d\" (UID: \"3ae24773-6451-4cfd-a768-8548a684e09f\") " pod="openshift-dns/dns-default-zht2d"
Sep 17 21:42:16.513982 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:16.513939    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zht2d"
Sep 17 21:42:16.799182 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:16.799092    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zht2d"]
Sep 17 21:42:16.801998 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:42:16.801971    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae24773_6451_4cfd_a768_8548a684e09f.slice/crio-7d1a70dad91a23378b39870431b591f0ca11bad4de3640f8babcc3c13c6d4d07 WatchSource:0}: Error finding container 7d1a70dad91a23378b39870431b591f0ca11bad4de3640f8babcc3c13c6d4d07: Status 404 returned error can't find the container with id 7d1a70dad91a23378b39870431b591f0ca11bad4de3640f8babcc3c13c6d4d07
Sep 17 21:42:17.142319 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:17.142235    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zht2d" event={"ID":"3ae24773-6451-4cfd-a768-8548a684e09f","Type":"ContainerStarted","Data":"7d1a70dad91a23378b39870431b591f0ca11bad4de3640f8babcc3c13c6d4d07"}
Sep 17 21:42:19.150489 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:19.150456    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zht2d" event={"ID":"3ae24773-6451-4cfd-a768-8548a684e09f","Type":"ContainerStarted","Data":"6e054a662ba931973a5fa895541fc73cfa45d2b511cc7e4b3d72feb8f6e007f4"}
Sep 17 21:42:19.150925 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:19.150514    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zht2d" event={"ID":"3ae24773-6451-4cfd-a768-8548a684e09f","Type":"ContainerStarted","Data":"dd2c94accbab670176a29c8d26b2dc2bf2e18dad01e5405db3b4ad7a27fd22ae"}
Sep 17 21:42:19.150925 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:19.150597    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zht2d"
Sep 17 21:42:19.168627 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:19.168577    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zht2d" podStartSLOduration=17.033146502 podStartE2EDuration="19.168563494s" totalImagesPullingTime="2.135416992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:42:19.16743386 +0000 UTC m=+54.985760174" watchObservedRunningTime="2026-09-17 21:42:19.168563494 +0000 UTC m=+54.986889806"
Sep 17 21:42:21.451792 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:21.451757    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:21.451792 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:21.451802    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:21.452334 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:21.451905    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:42:21.452334 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:21.451963    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-insights-runtime-extractor-tls podName:87a36c1f-0c4f-4875-8d1e-f562ac1a19a2 nodeName:}" failed. No retries permitted until 2026-09-17 21:42:37.451950532 +0000 UTC m=+73.270276828 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sdnkh" (UID: "87a36c1f-0c4f-4875-8d1e-f562ac1a19a2") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:42:21.452334 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:21.452297    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:24.073551 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:24.073523    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zdcnw"
Sep 17 21:42:29.157068 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:29.157034    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zht2d"
Sep 17 21:42:30.417104 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:30.417069    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e76c503-ec71-47a9-b3cd-63d3b12a439a-original-pull-secret\") pod \"global-pull-secret-syncer-hrbr6\" (UID: \"1e76c503-ec71-47a9-b3cd-63d3b12a439a\") " pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:42:30.417524 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:30.417146    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93119bca-cc51-4b33-964b-63bf4fe5c9e1-metrics-certs\") pod \"network-metrics-daemon-5gxhr\" (UID: \"93119bca-cc51-4b33-964b-63bf4fe5c9e1\") " pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:42:30.420024 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:30.420006    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 21:42:30.421050 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:30.421025    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 21:42:30.430310 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:30.430283    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e76c503-ec71-47a9-b3cd-63d3b12a439a-original-pull-secret\") pod \"global-pull-secret-syncer-hrbr6\" (UID: \"1e76c503-ec71-47a9-b3cd-63d3b12a439a\") " pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:42:30.430391 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:30.430341    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93119bca-cc51-4b33-964b-63bf4fe5c9e1-metrics-certs\") pod \"network-metrics-daemon-5gxhr\" (UID: \"93119bca-cc51-4b33-964b-63bf4fe5c9e1\") " pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:42:30.459121 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:30.459092    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-44pmn\""
Sep 17 21:42:30.467350 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:30.467330    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gxhr"
Sep 17 21:42:30.564075 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:30.561844    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrbr6"
Sep 17 21:42:30.618920 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:30.618886    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlhnr\" (UniqueName: \"kubernetes.io/projected/5fd48c9a-8cd8-4743-8dbf-eb20219c893c-kube-api-access-mlhnr\") pod \"network-check-target-hmscl\" (UID: \"5fd48c9a-8cd8-4743-8dbf-eb20219c893c\") " pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:42:30.622373 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:30.622349    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 21:42:30.632908 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:30.632681    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 21:42:30.642796 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:30.642761    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlhnr\" (UniqueName: \"kubernetes.io/projected/5fd48c9a-8cd8-4743-8dbf-eb20219c893c-kube-api-access-mlhnr\") pod \"network-check-target-hmscl\" (UID: \"5fd48c9a-8cd8-4743-8dbf-eb20219c893c\") " pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:42:30.763866 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:30.763842    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5gxhr"]
Sep 17 21:42:30.763994 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:30.763976    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7sc2s\""
Sep 17 21:42:30.766028 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:42:30.766000    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93119bca_cc51_4b33_964b_63bf4fe5c9e1.slice/crio-69dbacdfc75c0d07f47cccb80571f69a1db148a0fab4acc05b1a730c64060e29 WatchSource:0}: Error finding container 69dbacdfc75c0d07f47cccb80571f69a1db148a0fab4acc05b1a730c64060e29: Status 404 returned error can't find the container with id 69dbacdfc75c0d07f47cccb80571f69a1db148a0fab4acc05b1a730c64060e29
Sep 17 21:42:30.771427 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:30.771409    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:42:30.850650 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:30.850608    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hrbr6"]
Sep 17 21:42:30.854790 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:42:30.854749    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e76c503_ec71_47a9_b3cd_63d3b12a439a.slice/crio-60bfddbe863599e81b27155ee3fabfcb92663176d2e710370ab4a59892a74693 WatchSource:0}: Error finding container 60bfddbe863599e81b27155ee3fabfcb92663176d2e710370ab4a59892a74693: Status 404 returned error can't find the container with id 60bfddbe863599e81b27155ee3fabfcb92663176d2e710370ab4a59892a74693
Sep 17 21:42:31.072542 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.072450    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hmscl"]
Sep 17 21:42:31.075028 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:42:31.074995    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd48c9a_8cd8_4743_8dbf_eb20219c893c.slice/crio-4605058d515a0b4899694ae59948a8cfba778a4517c53a5b141c0ec279269af5 WatchSource:0}: Error finding container 4605058d515a0b4899694ae59948a8cfba778a4517c53a5b141c0ec279269af5: Status 404 returned error can't find the container with id 4605058d515a0b4899694ae59948a8cfba778a4517c53a5b141c0ec279269af5
Sep 17 21:42:31.188282 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.188242    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5gxhr" event={"ID":"93119bca-cc51-4b33-964b-63bf4fe5c9e1","Type":"ContainerStarted","Data":"69dbacdfc75c0d07f47cccb80571f69a1db148a0fab4acc05b1a730c64060e29"}
Sep 17 21:42:31.189399 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.189367    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hmscl" event={"ID":"5fd48c9a-8cd8-4743-8dbf-eb20219c893c","Type":"ContainerStarted","Data":"4605058d515a0b4899694ae59948a8cfba778a4517c53a5b141c0ec279269af5"}
Sep 17 21:42:31.190485 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.190466    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hrbr6" event={"ID":"1e76c503-ec71-47a9-b3cd-63d3b12a439a","Type":"ContainerStarted","Data":"60bfddbe863599e81b27155ee3fabfcb92663176d2e710370ab4a59892a74693"}
Sep 17 21:42:31.783369 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.783326    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b46d69d85-wq8dq"]
Sep 17 21:42:31.786405 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.786378    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f9bd94b-wbl44"]
Sep 17 21:42:31.786539 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.786467    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b46d69d85-wq8dq"
Sep 17 21:42:31.788875 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.788852    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr"]
Sep 17 21:42:31.790025 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.790005    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f9bd94b-wbl44"
Sep 17 21:42:31.791623 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.790861    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 21:42:31.791623 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.791273    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 21:42:31.791623 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.791364    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 21:42:31.791623 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.791490    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 21:42:31.792215 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.792198    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr"
Sep 17 21:42:31.793885 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.793866    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 21:42:31.794443 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.794412    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-6wk4z\""
Sep 17 21:42:31.795793 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.795647    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 21:42:31.795871 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.795861    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 21:42:31.796218 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.796200    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 21:42:31.796286 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.796275    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 21:42:31.803458 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.803432    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b46d69d85-wq8dq"]
Sep 17 21:42:31.803630 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.803616    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f9bd94b-wbl44"]
Sep 17 21:42:31.821789 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.821686    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr"]
Sep 17 21:42:31.823631 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.823360    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-77f5dbdf48-shvsz"]
Sep 17 21:42:31.826249 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.826193    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:31.828033 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.828011    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9bcc8a78-d00c-4280-a487-3d8b8e19d024-hub\") pod \"cluster-proxy-proxy-agent-6655ddf596-5cvgr\" (UID: \"9bcc8a78-d00c-4280-a487-3d8b8e19d024\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr"
Sep 17 21:42:31.828155 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.828047    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9bcc8a78-d00c-4280-a487-3d8b8e19d024-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6655ddf596-5cvgr\" (UID: \"9bcc8a78-d00c-4280-a487-3d8b8e19d024\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr"
Sep 17 21:42:31.828155 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.828101    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9bcc8a78-d00c-4280-a487-3d8b8e19d024-ca\") pod \"cluster-proxy-proxy-agent-6655ddf596-5cvgr\" (UID: \"9bcc8a78-d00c-4280-a487-3d8b8e19d024\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr"
Sep 17 21:42:31.828155 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.828129    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xns7k\" (UniqueName: \"kubernetes.io/projected/9bcc8a78-d00c-4280-a487-3d8b8e19d024-kube-api-access-xns7k\") pod \"cluster-proxy-proxy-agent-6655ddf596-5cvgr\" (UID: \"9bcc8a78-d00c-4280-a487-3d8b8e19d024\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr"
Sep 17 21:42:31.828155 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.828153    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84953328-efda-4b12-b7d3-d4ce123bd9e7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b5f9bd94b-wbl44\" (UID: \"84953328-efda-4b12-b7d3-d4ce123bd9e7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f9bd94b-wbl44"
Sep 17 21:42:31.828396 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.828204    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bcc8a78-d00c-4280-a487-3d8b8e19d024-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6655ddf596-5cvgr\" (UID: \"9bcc8a78-d00c-4280-a487-3d8b8e19d024\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr"
Sep 17 21:42:31.828396 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.828234    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9bcc8a78-d00c-4280-a487-3d8b8e19d024-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6655ddf596-5cvgr\" (UID: \"9bcc8a78-d00c-4280-a487-3d8b8e19d024\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr"
Sep 17 21:42:31.828396 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.828281    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prq6l\" (UniqueName: \"kubernetes.io/projected/84953328-efda-4b12-b7d3-d4ce123bd9e7-kube-api-access-prq6l\") pod \"managed-serviceaccount-addon-agent-6b5f9bd94b-wbl44\" (UID: \"84953328-efda-4b12-b7d3-d4ce123bd9e7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f9bd94b-wbl44"
Sep 17 21:42:31.828396 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.828335    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhhk6\" (UniqueName: \"kubernetes.io/projected/d6619f61-5f63-4e7f-865a-8f06faa70da2-kube-api-access-qhhk6\") pod \"klusterlet-addon-workmgr-6b46d69d85-wq8dq\" (UID: \"d6619f61-5f63-4e7f-865a-8f06faa70da2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b46d69d85-wq8dq"
Sep 17 21:42:31.828396 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.828394    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6619f61-5f63-4e7f-865a-8f06faa70da2-tmp\") pod \"klusterlet-addon-workmgr-6b46d69d85-wq8dq\" (UID: \"d6619f61-5f63-4e7f-865a-8f06faa70da2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b46d69d85-wq8dq"
Sep 17 21:42:31.828630 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.828418    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d6619f61-5f63-4e7f-865a-8f06faa70da2-klusterlet-config\") pod \"klusterlet-addon-workmgr-6b46d69d85-wq8dq\" (UID: \"d6619f61-5f63-4e7f-865a-8f06faa70da2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b46d69d85-wq8dq"
Sep 17 21:42:31.832824 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.832604    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 21:42:31.839056 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.839033    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fbg5w\""
Sep 17 21:42:31.840282 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.840260    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 21:42:31.841473 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.841455    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 21:42:31.860487 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.860452    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 21:42:31.898561 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.898528    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77f5dbdf48-shvsz"]
Sep 17 21:42:31.929371 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.929333    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xns7k\" (UniqueName: \"kubernetes.io/projected/9bcc8a78-d00c-4280-a487-3d8b8e19d024-kube-api-access-xns7k\") pod \"cluster-proxy-proxy-agent-6655ddf596-5cvgr\" (UID: \"9bcc8a78-d00c-4280-a487-3d8b8e19d024\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr"
Sep 17 21:42:31.929642 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.929407    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prq6l\" (UniqueName: \"kubernetes.io/projected/84953328-efda-4b12-b7d3-d4ce123bd9e7-kube-api-access-prq6l\") pod \"managed-serviceaccount-addon-agent-6b5f9bd94b-wbl44\" (UID: \"84953328-efda-4b12-b7d3-d4ce123bd9e7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f9bd94b-wbl44"
Sep 17 21:42:31.929642 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.929546    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhhk6\" (UniqueName: \"kubernetes.io/projected/d6619f61-5f63-4e7f-865a-8f06faa70da2-kube-api-access-qhhk6\") pod \"klusterlet-addon-workmgr-6b46d69d85-wq8dq\" (UID: \"d6619f61-5f63-4e7f-865a-8f06faa70da2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b46d69d85-wq8dq"
Sep 17 21:42:31.929642 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.929579    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d6619f61-5f63-4e7f-865a-8f06faa70da2-klusterlet-config\") pod \"klusterlet-addon-workmgr-6b46d69d85-wq8dq\" (UID: \"d6619f61-5f63-4e7f-865a-8f06faa70da2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b46d69d85-wq8dq"
Sep 17 21:42:31.929642 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.929615    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6278de03-2f77-4bea-86c2-d6d43306d76e-installation-pull-secrets\") pod \"image-registry-77f5dbdf48-shvsz\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") " pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:31.929878 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.929645    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6278de03-2f77-4bea-86c2-d6d43306d76e-registry-tls\") pod \"image-registry-77f5dbdf48-shvsz\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") " pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:31.929878 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.929680    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9bcc8a78-d00c-4280-a487-3d8b8e19d024-hub\") pod \"cluster-proxy-proxy-agent-6655ddf596-5cvgr\" (UID: \"9bcc8a78-d00c-4280-a487-3d8b8e19d024\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr"
Sep 17 21:42:31.929878 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.929704    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9bcc8a78-d00c-4280-a487-3d8b8e19d024-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6655ddf596-5cvgr\" (UID: \"9bcc8a78-d00c-4280-a487-3d8b8e19d024\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr"
Sep 17 21:42:31.929878 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.929725    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6278de03-2f77-4bea-86c2-d6d43306d76e-ca-trust-extracted\") pod \"image-registry-77f5dbdf48-shvsz\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") " pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:31.929878 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.929753    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9bcc8a78-d00c-4280-a487-3d8b8e19d024-ca\") pod \"cluster-proxy-proxy-agent-6655ddf596-5cvgr\" (UID: \"9bcc8a78-d00c-4280-a487-3d8b8e19d024\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr"
Sep 17 21:42:31.929878 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.929777    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6278de03-2f77-4bea-86c2-d6d43306d76e-image-registry-private-configuration\") pod \"image-registry-77f5dbdf48-shvsz\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") " pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:31.929878 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.929806    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84953328-efda-4b12-b7d3-d4ce123bd9e7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b5f9bd94b-wbl44\" (UID: \"84953328-efda-4b12-b7d3-d4ce123bd9e7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f9bd94b-wbl44"
Sep 17 21:42:31.929878 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.929837    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6619f61-5f63-4e7f-865a-8f06faa70da2-tmp\") pod \"klusterlet-addon-workmgr-6b46d69d85-wq8dq\" (UID: \"d6619f61-5f63-4e7f-865a-8f06faa70da2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b46d69d85-wq8dq"
Sep 17 21:42:31.929878 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.929871    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bcc8a78-d00c-4280-a487-3d8b8e19d024-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6655ddf596-5cvgr\" (UID: \"9bcc8a78-d00c-4280-a487-3d8b8e19d024\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr"
Sep 17 21:42:31.930355 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.929893    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6278de03-2f77-4bea-86c2-d6d43306d76e-bound-sa-token\") pod \"image-registry-77f5dbdf48-shvsz\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") " pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:31.930355 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.929925    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9bcc8a78-d00c-4280-a487-3d8b8e19d024-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6655ddf596-5cvgr\" (UID: \"9bcc8a78-d00c-4280-a487-3d8b8e19d024\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr"
Sep 17 21:42:31.930355 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.929953    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6278de03-2f77-4bea-86c2-d6d43306d76e-registry-certificates\") pod \"image-registry-77f5dbdf48-shvsz\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") " pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:31.930355 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.929974    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6278de03-2f77-4bea-86c2-d6d43306d76e-trusted-ca\") pod \"image-registry-77f5dbdf48-shvsz\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") " pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:31.930355 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.929995    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm7mq\" (UniqueName: \"kubernetes.io/projected/6278de03-2f77-4bea-86c2-d6d43306d76e-kube-api-access-fm7mq\") pod \"image-registry-77f5dbdf48-shvsz\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") " pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:31.931191 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.931151    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6619f61-5f63-4e7f-865a-8f06faa70da2-tmp\") pod \"klusterlet-addon-workmgr-6b46d69d85-wq8dq\" (UID: \"d6619f61-5f63-4e7f-865a-8f06faa70da2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b46d69d85-wq8dq"
Sep 17 21:42:31.931993 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.931947    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bcc8a78-d00c-4280-a487-3d8b8e19d024-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6655ddf596-5cvgr\" (UID: \"9bcc8a78-d00c-4280-a487-3d8b8e19d024\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr"
Sep 17 21:42:31.934251 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.934224    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d6619f61-5f63-4e7f-865a-8f06faa70da2-klusterlet-config\") pod \"klusterlet-addon-workmgr-6b46d69d85-wq8dq\" (UID: \"d6619f61-5f63-4e7f-865a-8f06faa70da2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b46d69d85-wq8dq"
Sep 17 21:42:31.935365 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.935247    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9bcc8a78-d00c-4280-a487-3d8b8e19d024-hub\") pod \"cluster-proxy-proxy-agent-6655ddf596-5cvgr\" (UID: \"9bcc8a78-d00c-4280-a487-3d8b8e19d024\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr"
Sep 17 21:42:31.937957 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.937823    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9bcc8a78-d00c-4280-a487-3d8b8e19d024-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6655ddf596-5cvgr\" (UID: \"9bcc8a78-d00c-4280-a487-3d8b8e19d024\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr"
Sep 17 21:42:31.937957 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.937922    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84953328-efda-4b12-b7d3-d4ce123bd9e7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b5f9bd94b-wbl44\" (UID: \"84953328-efda-4b12-b7d3-d4ce123bd9e7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f9bd94b-wbl44"
Sep 17 21:42:31.937957 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.937940    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9bcc8a78-d00c-4280-a487-3d8b8e19d024-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6655ddf596-5cvgr\" (UID: \"9bcc8a78-d00c-4280-a487-3d8b8e19d024\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr"
Sep 17 21:42:31.938341 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.938320    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9bcc8a78-d00c-4280-a487-3d8b8e19d024-ca\") pod \"cluster-proxy-proxy-agent-6655ddf596-5cvgr\" (UID: \"9bcc8a78-d00c-4280-a487-3d8b8e19d024\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr"
Sep 17 21:42:31.945113 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.945033    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhhk6\" (UniqueName: \"kubernetes.io/projected/d6619f61-5f63-4e7f-865a-8f06faa70da2-kube-api-access-qhhk6\") pod \"klusterlet-addon-workmgr-6b46d69d85-wq8dq\" (UID: \"d6619f61-5f63-4e7f-865a-8f06faa70da2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b46d69d85-wq8dq"
Sep 17 21:42:31.947323 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.947297    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xns7k\" (UniqueName: \"kubernetes.io/projected/9bcc8a78-d00c-4280-a487-3d8b8e19d024-kube-api-access-xns7k\") pod \"cluster-proxy-proxy-agent-6655ddf596-5cvgr\" (UID: \"9bcc8a78-d00c-4280-a487-3d8b8e19d024\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr"
Sep 17 21:42:31.950051 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.950026    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prq6l\" (UniqueName: \"kubernetes.io/projected/84953328-efda-4b12-b7d3-d4ce123bd9e7-kube-api-access-prq6l\") pod \"managed-serviceaccount-addon-agent-6b5f9bd94b-wbl44\" (UID: \"84953328-efda-4b12-b7d3-d4ce123bd9e7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f9bd94b-wbl44"
Sep 17 21:42:31.983014 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:31.982983    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77f5dbdf48-shvsz"]
Sep 17 21:42:31.983327 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:42:31.983298    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-fm7mq registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz" podUID="6278de03-2f77-4bea-86c2-d6d43306d76e"
Sep 17 21:42:32.020302 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.020270    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f88f88c4b-r97mr"]
Sep 17 21:42:32.025421 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.025395    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.031048 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.030352    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6278de03-2f77-4bea-86c2-d6d43306d76e-bound-sa-token\") pod \"image-registry-77f5dbdf48-shvsz\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") " pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:32.031048 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.030406    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6278de03-2f77-4bea-86c2-d6d43306d76e-registry-certificates\") pod \"image-registry-77f5dbdf48-shvsz\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") " pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:32.031048 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.030431    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6278de03-2f77-4bea-86c2-d6d43306d76e-trusted-ca\") pod \"image-registry-77f5dbdf48-shvsz\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") " pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:32.031048 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.030454    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm7mq\" (UniqueName: \"kubernetes.io/projected/6278de03-2f77-4bea-86c2-d6d43306d76e-kube-api-access-fm7mq\") pod \"image-registry-77f5dbdf48-shvsz\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") " pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:32.031048 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.030517    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6278de03-2f77-4bea-86c2-d6d43306d76e-installation-pull-secrets\") pod \"image-registry-77f5dbdf48-shvsz\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") " pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:32.031048 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.030544    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6278de03-2f77-4bea-86c2-d6d43306d76e-registry-tls\") pod \"image-registry-77f5dbdf48-shvsz\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") " pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:32.031048 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.030575    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6278de03-2f77-4bea-86c2-d6d43306d76e-ca-trust-extracted\") pod \"image-registry-77f5dbdf48-shvsz\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") " pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:32.031048 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.030609    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6278de03-2f77-4bea-86c2-d6d43306d76e-image-registry-private-configuration\") pod \"image-registry-77f5dbdf48-shvsz\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") " pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:32.031584 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.031237    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6278de03-2f77-4bea-86c2-d6d43306d76e-registry-certificates\") pod \"image-registry-77f5dbdf48-shvsz\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") " pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:32.034064 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.032362    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6278de03-2f77-4bea-86c2-d6d43306d76e-ca-trust-extracted\") pod \"image-registry-77f5dbdf48-shvsz\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") " pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:32.034788 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.034746    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6278de03-2f77-4bea-86c2-d6d43306d76e-installation-pull-secrets\") pod \"image-registry-77f5dbdf48-shvsz\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") " pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:32.034889 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.034860    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6278de03-2f77-4bea-86c2-d6d43306d76e-trusted-ca\") pod \"image-registry-77f5dbdf48-shvsz\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") " pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:32.034929 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.034902    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6278de03-2f77-4bea-86c2-d6d43306d76e-registry-tls\") pod \"image-registry-77f5dbdf48-shvsz\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") " pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:32.035437 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.035413    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6278de03-2f77-4bea-86c2-d6d43306d76e-image-registry-private-configuration\") pod \"image-registry-77f5dbdf48-shvsz\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") " pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:32.043974 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.043936    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f88f88c4b-r97mr"]
Sep 17 21:42:32.053321 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.053259    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm7mq\" (UniqueName: \"kubernetes.io/projected/6278de03-2f77-4bea-86c2-d6d43306d76e-kube-api-access-fm7mq\") pod \"image-registry-77f5dbdf48-shvsz\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") " pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:32.057464 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.057435    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6278de03-2f77-4bea-86c2-d6d43306d76e-bound-sa-token\") pod \"image-registry-77f5dbdf48-shvsz\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") " pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:32.102262 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.102110    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b46d69d85-wq8dq"
Sep 17 21:42:32.118696 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.118418    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f9bd94b-wbl44"
Sep 17 21:42:32.131578 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.131546    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c38cf73-dc73-4436-bbc7-0bf58648950a-bound-sa-token\") pod \"image-registry-5f88f88c4b-r97mr\" (UID: \"5c38cf73-dc73-4436-bbc7-0bf58648950a\") " pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.131709 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.131593    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c38cf73-dc73-4436-bbc7-0bf58648950a-registry-tls\") pod \"image-registry-5f88f88c4b-r97mr\" (UID: \"5c38cf73-dc73-4436-bbc7-0bf58648950a\") " pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.131709 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.131623    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kblt9\" (UniqueName: \"kubernetes.io/projected/5c38cf73-dc73-4436-bbc7-0bf58648950a-kube-api-access-kblt9\") pod \"image-registry-5f88f88c4b-r97mr\" (UID: \"5c38cf73-dc73-4436-bbc7-0bf58648950a\") " pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.131709 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.131671    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5c38cf73-dc73-4436-bbc7-0bf58648950a-registry-certificates\") pod \"image-registry-5f88f88c4b-r97mr\" (UID: \"5c38cf73-dc73-4436-bbc7-0bf58648950a\") " pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.131810 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.131710    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c38cf73-dc73-4436-bbc7-0bf58648950a-trusted-ca\") pod \"image-registry-5f88f88c4b-r97mr\" (UID: \"5c38cf73-dc73-4436-bbc7-0bf58648950a\") " pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.131810 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.131736    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5c38cf73-dc73-4436-bbc7-0bf58648950a-image-registry-private-configuration\") pod \"image-registry-5f88f88c4b-r97mr\" (UID: \"5c38cf73-dc73-4436-bbc7-0bf58648950a\") " pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.131810 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.131794    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5c38cf73-dc73-4436-bbc7-0bf58648950a-installation-pull-secrets\") pod \"image-registry-5f88f88c4b-r97mr\" (UID: \"5c38cf73-dc73-4436-bbc7-0bf58648950a\") " pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.131923 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.131826    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5c38cf73-dc73-4436-bbc7-0bf58648950a-ca-trust-extracted\") pod \"image-registry-5f88f88c4b-r97mr\" (UID: \"5c38cf73-dc73-4436-bbc7-0bf58648950a\") " pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.133545 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.133515    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr"
Sep 17 21:42:32.198421 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.197757    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:32.209346 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.209212    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:32.232738 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.232692    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6278de03-2f77-4bea-86c2-d6d43306d76e-ca-trust-extracted\") pod \"6278de03-2f77-4bea-86c2-d6d43306d76e\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") "
Sep 17 21:42:32.232852 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.232753    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6278de03-2f77-4bea-86c2-d6d43306d76e-registry-tls\") pod \"6278de03-2f77-4bea-86c2-d6d43306d76e\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") "
Sep 17 21:42:32.232852 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.232789    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6278de03-2f77-4bea-86c2-d6d43306d76e-image-registry-private-configuration\") pod \"6278de03-2f77-4bea-86c2-d6d43306d76e\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") "
Sep 17 21:42:32.232852 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.232839    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6278de03-2f77-4bea-86c2-d6d43306d76e-registry-certificates\") pod \"6278de03-2f77-4bea-86c2-d6d43306d76e\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") "
Sep 17 21:42:32.232998 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.232879    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6278de03-2f77-4bea-86c2-d6d43306d76e-installation-pull-secrets\") pod \"6278de03-2f77-4bea-86c2-d6d43306d76e\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") "
Sep 17 21:42:32.232998 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.232907    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6278de03-2f77-4bea-86c2-d6d43306d76e-trusted-ca\") pod \"6278de03-2f77-4bea-86c2-d6d43306d76e\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") "
Sep 17 21:42:32.232998 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.232937    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6278de03-2f77-4bea-86c2-d6d43306d76e-bound-sa-token\") pod \"6278de03-2f77-4bea-86c2-d6d43306d76e\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") "
Sep 17 21:42:32.232998 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.232961    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fm7mq\" (UniqueName: \"kubernetes.io/projected/6278de03-2f77-4bea-86c2-d6d43306d76e-kube-api-access-fm7mq\") pod \"6278de03-2f77-4bea-86c2-d6d43306d76e\" (UID: \"6278de03-2f77-4bea-86c2-d6d43306d76e\") "
Sep 17 21:42:32.233210 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.233061    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c38cf73-dc73-4436-bbc7-0bf58648950a-trusted-ca\") pod \"image-registry-5f88f88c4b-r97mr\" (UID: \"5c38cf73-dc73-4436-bbc7-0bf58648950a\") " pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.233210 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.233096    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5c38cf73-dc73-4436-bbc7-0bf58648950a-image-registry-private-configuration\") pod \"image-registry-5f88f88c4b-r97mr\" (UID: \"5c38cf73-dc73-4436-bbc7-0bf58648950a\") " pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.233210 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.233164    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5c38cf73-dc73-4436-bbc7-0bf58648950a-installation-pull-secrets\") pod \"image-registry-5f88f88c4b-r97mr\" (UID: \"5c38cf73-dc73-4436-bbc7-0bf58648950a\") " pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.233371 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.233218    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5c38cf73-dc73-4436-bbc7-0bf58648950a-ca-trust-extracted\") pod \"image-registry-5f88f88c4b-r97mr\" (UID: \"5c38cf73-dc73-4436-bbc7-0bf58648950a\") " pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.233371 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.233257    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c38cf73-dc73-4436-bbc7-0bf58648950a-bound-sa-token\") pod \"image-registry-5f88f88c4b-r97mr\" (UID: \"5c38cf73-dc73-4436-bbc7-0bf58648950a\") " pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.233371 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.233287    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c38cf73-dc73-4436-bbc7-0bf58648950a-registry-tls\") pod \"image-registry-5f88f88c4b-r97mr\" (UID: \"5c38cf73-dc73-4436-bbc7-0bf58648950a\") " pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.233371 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.233311    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kblt9\" (UniqueName: \"kubernetes.io/projected/5c38cf73-dc73-4436-bbc7-0bf58648950a-kube-api-access-kblt9\") pod \"image-registry-5f88f88c4b-r97mr\" (UID: \"5c38cf73-dc73-4436-bbc7-0bf58648950a\") " pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.233371 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.233360    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5c38cf73-dc73-4436-bbc7-0bf58648950a-registry-certificates\") pod \"image-registry-5f88f88c4b-r97mr\" (UID: \"5c38cf73-dc73-4436-bbc7-0bf58648950a\") " pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.234360 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.234214    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5c38cf73-dc73-4436-bbc7-0bf58648950a-registry-certificates\") pod \"image-registry-5f88f88c4b-r97mr\" (UID: \"5c38cf73-dc73-4436-bbc7-0bf58648950a\") " pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.235118 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.235096    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5c38cf73-dc73-4436-bbc7-0bf58648950a-ca-trust-extracted\") pod \"image-registry-5f88f88c4b-r97mr\" (UID: \"5c38cf73-dc73-4436-bbc7-0bf58648950a\") " pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.236713 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.236684    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6278de03-2f77-4bea-86c2-d6d43306d76e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6278de03-2f77-4bea-86c2-d6d43306d76e" (UID: "6278de03-2f77-4bea-86c2-d6d43306d76e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:42:32.237024 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.236915    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c38cf73-dc73-4436-bbc7-0bf58648950a-trusted-ca\") pod \"image-registry-5f88f88c4b-r97mr\" (UID: \"5c38cf73-dc73-4436-bbc7-0bf58648950a\") " pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.238157 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.237682    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6278de03-2f77-4bea-86c2-d6d43306d76e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6278de03-2f77-4bea-86c2-d6d43306d76e" (UID: "6278de03-2f77-4bea-86c2-d6d43306d76e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:42:32.238478 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.238436    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6278de03-2f77-4bea-86c2-d6d43306d76e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6278de03-2f77-4bea-86c2-d6d43306d76e" (UID: "6278de03-2f77-4bea-86c2-d6d43306d76e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:42:32.247412 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.247365    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6278de03-2f77-4bea-86c2-d6d43306d76e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6278de03-2f77-4bea-86c2-d6d43306d76e" (UID: "6278de03-2f77-4bea-86c2-d6d43306d76e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:42:32.249007 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.248980    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6278de03-2f77-4bea-86c2-d6d43306d76e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6278de03-2f77-4bea-86c2-d6d43306d76e" (UID: "6278de03-2f77-4bea-86c2-d6d43306d76e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:42:32.255312 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.255255    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6278de03-2f77-4bea-86c2-d6d43306d76e-kube-api-access-fm7mq" (OuterVolumeSpecName: "kube-api-access-fm7mq") pod "6278de03-2f77-4bea-86c2-d6d43306d76e" (UID: "6278de03-2f77-4bea-86c2-d6d43306d76e"). InnerVolumeSpecName "kube-api-access-fm7mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:42:32.255649 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.255601    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kblt9\" (UniqueName: \"kubernetes.io/projected/5c38cf73-dc73-4436-bbc7-0bf58648950a-kube-api-access-kblt9\") pod \"image-registry-5f88f88c4b-r97mr\" (UID: \"5c38cf73-dc73-4436-bbc7-0bf58648950a\") " pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.255649 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.255617    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5c38cf73-dc73-4436-bbc7-0bf58648950a-installation-pull-secrets\") pod \"image-registry-5f88f88c4b-r97mr\" (UID: \"5c38cf73-dc73-4436-bbc7-0bf58648950a\") " pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.255810 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.255715    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c38cf73-dc73-4436-bbc7-0bf58648950a-registry-tls\") pod \"image-registry-5f88f88c4b-r97mr\" (UID: \"5c38cf73-dc73-4436-bbc7-0bf58648950a\") " pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.257204 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.257157    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c38cf73-dc73-4436-bbc7-0bf58648950a-bound-sa-token\") pod \"image-registry-5f88f88c4b-r97mr\" (UID: \"5c38cf73-dc73-4436-bbc7-0bf58648950a\") " pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.257487 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.257446    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5c38cf73-dc73-4436-bbc7-0bf58648950a-image-registry-private-configuration\") pod \"image-registry-5f88f88c4b-r97mr\" (UID: \"5c38cf73-dc73-4436-bbc7-0bf58648950a\") " pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.260979 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.260955    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6278de03-2f77-4bea-86c2-d6d43306d76e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6278de03-2f77-4bea-86c2-d6d43306d76e" (UID: "6278de03-2f77-4bea-86c2-d6d43306d76e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:42:32.261072 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.261051    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6278de03-2f77-4bea-86c2-d6d43306d76e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6278de03-2f77-4bea-86c2-d6d43306d76e" (UID: "6278de03-2f77-4bea-86c2-d6d43306d76e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:42:32.334534 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.334379    2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6278de03-2f77-4bea-86c2-d6d43306d76e-bound-sa-token\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:42:32.334534 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.334410    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm7mq\" (UniqueName: \"kubernetes.io/projected/6278de03-2f77-4bea-86c2-d6d43306d76e-kube-api-access-fm7mq\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:42:32.334534 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.334429    2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6278de03-2f77-4bea-86c2-d6d43306d76e-ca-trust-extracted\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:42:32.334534 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.334446    2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6278de03-2f77-4bea-86c2-d6d43306d76e-registry-tls\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:42:32.334534 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.334463    2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6278de03-2f77-4bea-86c2-d6d43306d76e-image-registry-private-configuration\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:42:32.334534 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.334479    2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6278de03-2f77-4bea-86c2-d6d43306d76e-registry-certificates\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:42:32.334534 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.334494    2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6278de03-2f77-4bea-86c2-d6d43306d76e-installation-pull-secrets\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:42:32.334534 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.334510    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6278de03-2f77-4bea-86c2-d6d43306d76e-trusted-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:42:32.345515 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.345489    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:32.471221 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.470151    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f9bd94b-wbl44"]
Sep 17 21:42:32.477869 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:42:32.477837    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84953328_efda_4b12_b7d3_d4ce123bd9e7.slice/crio-f581fbd297ea6f457e22fd96bce8077405694617d26bdb6eacb4421e6c1a13b4 WatchSource:0}: Error finding container f581fbd297ea6f457e22fd96bce8077405694617d26bdb6eacb4421e6c1a13b4: Status 404 returned error can't find the container with id f581fbd297ea6f457e22fd96bce8077405694617d26bdb6eacb4421e6c1a13b4
Sep 17 21:42:32.485377 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.484590    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr"]
Sep 17 21:42:32.485377 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.484758    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b46d69d85-wq8dq"]
Sep 17 21:42:32.485945 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:42:32.485528    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bcc8a78_d00c_4280_a487_3d8b8e19d024.slice/crio-e238030c80e82e6391dcdbab8fd7c38d1ead4830de97ed404cf905b8b409ac84 WatchSource:0}: Error finding container e238030c80e82e6391dcdbab8fd7c38d1ead4830de97ed404cf905b8b409ac84: Status 404 returned error can't find the container with id e238030c80e82e6391dcdbab8fd7c38d1ead4830de97ed404cf905b8b409ac84
Sep 17 21:42:32.486104 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:42:32.485978    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6619f61_5f63_4e7f_865a_8f06faa70da2.slice/crio-c27b2c8f4544d83b1160e9b3b0501a0d79a4758a2e741ea7866438087e5c9345 WatchSource:0}: Error finding container c27b2c8f4544d83b1160e9b3b0501a0d79a4758a2e741ea7866438087e5c9345: Status 404 returned error can't find the container with id c27b2c8f4544d83b1160e9b3b0501a0d79a4758a2e741ea7866438087e5c9345
Sep 17 21:42:32.657488 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:32.657457    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f88f88c4b-r97mr"]
Sep 17 21:42:32.662328 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:42:32.662292    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c38cf73_dc73_4436_bbc7_0bf58648950a.slice/crio-8b1d5890de8cab54c5b60096def47da85b64f80e7b63423dc261d68a96b85d94 WatchSource:0}: Error finding container 8b1d5890de8cab54c5b60096def47da85b64f80e7b63423dc261d68a96b85d94: Status 404 returned error can't find the container with id 8b1d5890de8cab54c5b60096def47da85b64f80e7b63423dc261d68a96b85d94
Sep 17 21:42:33.205962 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:33.205918    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5gxhr" event={"ID":"93119bca-cc51-4b33-964b-63bf4fe5c9e1","Type":"ContainerStarted","Data":"46368a27a6ee17b0e94e79bd9992f39a8a711517a42efb3a4828d2cc67104f70"}
Sep 17 21:42:33.206450 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:33.206023    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5gxhr" event={"ID":"93119bca-cc51-4b33-964b-63bf4fe5c9e1","Type":"ContainerStarted","Data":"25ef8af3d2313bfbc6a7ece4b59255f69c557fdfd71560a355d8f07ef26fe92b"}
Sep 17 21:42:33.208230 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:33.208197    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b46d69d85-wq8dq" event={"ID":"d6619f61-5f63-4e7f-865a-8f06faa70da2","Type":"ContainerStarted","Data":"c27b2c8f4544d83b1160e9b3b0501a0d79a4758a2e741ea7866438087e5c9345"}
Sep 17 21:42:33.209873 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:33.209830    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f9bd94b-wbl44" event={"ID":"84953328-efda-4b12-b7d3-d4ce123bd9e7","Type":"ContainerStarted","Data":"f581fbd297ea6f457e22fd96bce8077405694617d26bdb6eacb4421e6c1a13b4"}
Sep 17 21:42:33.212060 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:33.212014    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr" event={"ID":"9bcc8a78-d00c-4280-a487-3d8b8e19d024","Type":"ContainerStarted","Data":"e238030c80e82e6391dcdbab8fd7c38d1ead4830de97ed404cf905b8b409ac84"}
Sep 17 21:42:33.216557 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:33.216002    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77f5dbdf48-shvsz"
Sep 17 21:42:33.216557 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:33.216479    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr" event={"ID":"5c38cf73-dc73-4436-bbc7-0bf58648950a","Type":"ContainerStarted","Data":"a6c34af9f1b023294bf04d389bab81d3c63a0ed2808f8bbea78bb93ecbaa737c"}
Sep 17 21:42:33.216557 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:33.216514    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr" event={"ID":"5c38cf73-dc73-4436-bbc7-0bf58648950a","Type":"ContainerStarted","Data":"8b1d5890de8cab54c5b60096def47da85b64f80e7b63423dc261d68a96b85d94"}
Sep 17 21:42:33.216794 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:33.216659    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:42:33.242323 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:33.242048    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5gxhr" podStartSLOduration=66.807713773 podStartE2EDuration="1m8.242027359s" totalImagesPullingTime="1.434313586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:42:33.240324668 +0000 UTC m=+69.058650993" watchObservedRunningTime="2026-09-17 21:42:33.242027359 +0000 UTC m=+69.060353672"
Sep 17 21:42:33.322623 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:33.322546    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr" podStartSLOduration=2.322523182 podStartE2EDuration="2.322523182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:42:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:42:33.284390679 +0000 UTC m=+69.102716991" watchObservedRunningTime="2026-09-17 21:42:33.322523182 +0000 UTC m=+69.140849496"
Sep 17 21:42:33.338955 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:33.338916    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77f5dbdf48-shvsz"]
Sep 17 21:42:33.342025 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:33.341998    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-77f5dbdf48-shvsz"]
Sep 17 21:42:34.753342 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:34.753064    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6278de03-2f77-4bea-86c2-d6d43306d76e" path="/var/lib/kubelet/pods/6278de03-2f77-4bea-86c2-d6d43306d76e/volumes"
Sep 17 21:42:37.484165 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:37.484127    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:37.486582 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:37.486552    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87a36c1f-0c4f-4875-8d1e-f562ac1a19a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdnkh\" (UID: \"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2\") " pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:37.664089 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:37.663878    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5wn52\""
Sep 17 21:42:37.671455 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:37.671423    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sdnkh"
Sep 17 21:42:39.245992 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:39.245960    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f9bd94b-wbl44" event={"ID":"84953328-efda-4b12-b7d3-d4ce123bd9e7","Type":"ContainerStarted","Data":"ae3fa3e00ecf7f81d0b5398179fba9b910594e49ff703d7735c084a713f1186b"}
Sep 17 21:42:39.248338 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:39.248272    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr" event={"ID":"9bcc8a78-d00c-4280-a487-3d8b8e19d024","Type":"ContainerStarted","Data":"2d0ce8a6146b3c4d57fb76689bee8eb8bfa205683e31378a2b977cca414f4442"}
Sep 17 21:42:39.248338 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:39.248315    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr" event={"ID":"9bcc8a78-d00c-4280-a487-3d8b8e19d024","Type":"ContainerStarted","Data":"56eb7a4f0865b6e14e379725cf47c63e3b47fe8f940c44998b357bf4ad4db442"}
Sep 17 21:42:39.249779 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:39.249755    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hmscl" event={"ID":"5fd48c9a-8cd8-4743-8dbf-eb20219c893c","Type":"ContainerStarted","Data":"81fef8e196726f81970587ae835c398bab438112caf4878f4a118c578f807a66"}
Sep 17 21:42:39.250043 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:39.250006    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:42:39.251487 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:39.251461    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hrbr6" event={"ID":"1e76c503-ec71-47a9-b3cd-63d3b12a439a","Type":"ContainerStarted","Data":"1670e747dcb75442755fd79299ec83a455ebf6a20ff08b01b1c62bbd48bc0d5e"}
Sep 17 21:42:39.274272 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:39.271291    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f9bd94b-wbl44" podStartSLOduration=1.693443461 podStartE2EDuration="8.271254242s" totalImagesPullingTime="6.577810781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:42:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:42:39.269794499 +0000 UTC m=+75.088120813" watchObservedRunningTime="2026-09-17 21:42:39.271254242 +0000 UTC m=+75.089580546"
Sep 17 21:42:39.291240 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:39.291000    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hrbr6" podStartSLOduration=66.093620611 podStartE2EDuration="1m14.290978974s" totalImagesPullingTime="8.197358363s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:42:39.290268161 +0000 UTC m=+75.108594471" watchObservedRunningTime="2026-09-17 21:42:39.290978974 +0000 UTC m=+75.109305290"
Sep 17 21:42:39.310824 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:39.310715    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hmscl" podStartSLOduration=66.333286685 podStartE2EDuration="1m14.310694012s" totalImagesPullingTime="7.977407327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:42:39.308650453 +0000 UTC m=+75.126976766" watchObservedRunningTime="2026-09-17 21:42:39.310694012 +0000 UTC m=+75.129020325"
Sep 17 21:42:39.343548 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:39.343512    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sdnkh"]
Sep 17 21:42:39.347876 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:42:39.347846    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a36c1f_0c4f_4875_8d1e_f562ac1a19a2.slice/crio-3591e1bd1a277fdabf1ac90733f93e711648b9e0853c5f22bb2712fdad56ba04 WatchSource:0}: Error finding container 3591e1bd1a277fdabf1ac90733f93e711648b9e0853c5f22bb2712fdad56ba04: Status 404 returned error can't find the container with id 3591e1bd1a277fdabf1ac90733f93e711648b9e0853c5f22bb2712fdad56ba04
Sep 17 21:42:40.257532 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:40.257494    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr" event={"ID":"9bcc8a78-d00c-4280-a487-3d8b8e19d024","Type":"ContainerStarted","Data":"3326fe60cf238dd9d0fd89cf667f03f6d503e5cb923d99b0a521af384c14ba89"}
Sep 17 21:42:40.259406 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:40.259383    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sdnkh" event={"ID":"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2","Type":"ContainerStarted","Data":"c7267ecdbc94e327f68332bd9c0aba7f9404b71f8d0ba9b9b513f20a55a81b01"}
Sep 17 21:42:40.259536 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:40.259425    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sdnkh" event={"ID":"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2","Type":"ContainerStarted","Data":"ada18edf89e9236f91a6fa5546771eaf69d220eae91cf209532d9d3af8b7b2d1"}
Sep 17 21:42:40.259536 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:40.259445    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sdnkh" event={"ID":"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2","Type":"ContainerStarted","Data":"3591e1bd1a277fdabf1ac90733f93e711648b9e0853c5f22bb2712fdad56ba04"}
Sep 17 21:42:40.279348 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:40.279299    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6655ddf596-5cvgr" podStartSLOduration=2.681206504 podStartE2EDuration="9.279284242s" totalImagesPullingTime="6.598077738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:42:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:42:40.277803077 +0000 UTC m=+76.096129419" watchObservedRunningTime="2026-09-17 21:42:40.279284242 +0000 UTC m=+76.097610554"
Sep 17 21:42:42.268261 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:42.268220    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sdnkh" event={"ID":"87a36c1f-0c4f-4875-8d1e-f562ac1a19a2","Type":"ContainerStarted","Data":"f629a1e5672af12d018eac78f3f51358f62bd5445e1689e015b931ce172e4271"}
Sep 17 21:42:42.289471 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:42.289417    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sdnkh" podStartSLOduration=35.48696514 podStartE2EDuration="37.289401404s" totalImagesPullingTime="1.802436264s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:42:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:42:42.288486536 +0000 UTC m=+78.106812848" watchObservedRunningTime="2026-09-17 21:42:42.289401404 +0000 UTC m=+78.107727716"
Sep 17 21:42:43.294977 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.294933    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zpttr"]
Sep 17 21:42:43.300782 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.300762    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.305652 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.305143    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 21:42:43.305652 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.305238    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 21:42:43.305652 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.305147    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 21:42:43.305652 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.305487    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 21:42:43.306048 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.306029    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 21:42:43.306232 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.306068    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 21:42:43.306522 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.306505    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tv9nv\""
Sep 17 21:42:43.434124 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.434088    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7aacc7a6-e805-4323-a14e-94514b3e1725-node-exporter-wtmp\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.434346 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.434143    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7aacc7a6-e805-4323-a14e-94514b3e1725-root\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.434346 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.434210    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7aacc7a6-e805-4323-a14e-94514b3e1725-node-exporter-accelerators-collector-config\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.434346 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.434257    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7aacc7a6-e805-4323-a14e-94514b3e1725-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.434346 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.434289    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7aacc7a6-e805-4323-a14e-94514b3e1725-metrics-client-ca\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.434346 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.434330    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7aacc7a6-e805-4323-a14e-94514b3e1725-sys\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.434621 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.434352    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7aacc7a6-e805-4323-a14e-94514b3e1725-node-exporter-textfile\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.434621 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.434380    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc8zg\" (UniqueName: \"kubernetes.io/projected/7aacc7a6-e805-4323-a14e-94514b3e1725-kube-api-access-xc8zg\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.434621 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.434408    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7aacc7a6-e805-4323-a14e-94514b3e1725-node-exporter-tls\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.535757 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.535714    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7aacc7a6-e805-4323-a14e-94514b3e1725-root\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.535757 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.535757    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7aacc7a6-e805-4323-a14e-94514b3e1725-node-exporter-accelerators-collector-config\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.536016 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.535783    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7aacc7a6-e805-4323-a14e-94514b3e1725-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.536016 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.535804    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7aacc7a6-e805-4323-a14e-94514b3e1725-metrics-client-ca\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.536016 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.535843    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7aacc7a6-e805-4323-a14e-94514b3e1725-sys\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.536016 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.535841    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7aacc7a6-e805-4323-a14e-94514b3e1725-root\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.536016 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.535861    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7aacc7a6-e805-4323-a14e-94514b3e1725-node-exporter-textfile\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.536016 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.535882    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc8zg\" (UniqueName: \"kubernetes.io/projected/7aacc7a6-e805-4323-a14e-94514b3e1725-kube-api-access-xc8zg\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.536016 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.535909    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7aacc7a6-e805-4323-a14e-94514b3e1725-node-exporter-tls\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.536016 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.535939    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7aacc7a6-e805-4323-a14e-94514b3e1725-sys\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.536016 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.535941    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7aacc7a6-e805-4323-a14e-94514b3e1725-node-exporter-wtmp\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.536471 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.536089    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7aacc7a6-e805-4323-a14e-94514b3e1725-node-exporter-wtmp\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.536471 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.536399    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7aacc7a6-e805-4323-a14e-94514b3e1725-node-exporter-textfile\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.536471 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.536457    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7aacc7a6-e805-4323-a14e-94514b3e1725-node-exporter-accelerators-collector-config\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.536615 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.536478    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7aacc7a6-e805-4323-a14e-94514b3e1725-metrics-client-ca\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.538666 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.538645    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7aacc7a6-e805-4323-a14e-94514b3e1725-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.538747 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.538666    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7aacc7a6-e805-4323-a14e-94514b3e1725-node-exporter-tls\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.544319 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.544275    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc8zg\" (UniqueName: \"kubernetes.io/projected/7aacc7a6-e805-4323-a14e-94514b3e1725-kube-api-access-xc8zg\") pod \"node-exporter-zpttr\" (UID: \"7aacc7a6-e805-4323-a14e-94514b3e1725\") " pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.612165 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:43.612086    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zpttr"
Sep 17 21:42:43.625995 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:42:43.625971    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aacc7a6_e805_4323_a14e_94514b3e1725.slice/crio-ef6ea64fde99768bb076c84e0424ace0a3be30aa7d69103418efe7e1da1e22c4 WatchSource:0}: Error finding container ef6ea64fde99768bb076c84e0424ace0a3be30aa7d69103418efe7e1da1e22c4: Status 404 returned error can't find the container with id ef6ea64fde99768bb076c84e0424ace0a3be30aa7d69103418efe7e1da1e22c4
Sep 17 21:42:44.277473 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:44.277442    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zpttr" event={"ID":"7aacc7a6-e805-4323-a14e-94514b3e1725","Type":"ContainerStarted","Data":"ef6ea64fde99768bb076c84e0424ace0a3be30aa7d69103418efe7e1da1e22c4"}
Sep 17 21:42:45.283091 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:45.283056    2564 generic.go:356] "Generic (PLEG): container finished" podID="7aacc7a6-e805-4323-a14e-94514b3e1725" containerID="f800408821847b4868c78a7dcbb923e95e758a6acfb04b259389268129e42d41" exitCode=0
Sep 17 21:42:45.283522 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:45.283150    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zpttr" event={"ID":"7aacc7a6-e805-4323-a14e-94514b3e1725","Type":"ContainerDied","Data":"f800408821847b4868c78a7dcbb923e95e758a6acfb04b259389268129e42d41"}
Sep 17 21:42:46.290892 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:46.290857    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zpttr" event={"ID":"7aacc7a6-e805-4323-a14e-94514b3e1725","Type":"ContainerStarted","Data":"9a57c6cc3a8e5839048a20dd0e32f2adabbfdc4ba80c819509ca95a08f1bdfea"}
Sep 17 21:42:46.291301 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:46.290922    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zpttr" event={"ID":"7aacc7a6-e805-4323-a14e-94514b3e1725","Type":"ContainerStarted","Data":"c5151878bb802a552d28a1d7e296c8ca194e49e3c1b8d073b03833f002d95d0a"}
Sep 17 21:42:46.353558 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:46.353504    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zpttr" podStartSLOduration=2.093448257 podStartE2EDuration="3.353487329s" totalImagesPullingTime="1.190317279s" totalInitContainerRuntime="69.721793ms" isStatefulPod=true podCreationTimestamp="2026-09-17 21:42:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:42:46.343972765 +0000 UTC m=+82.162299077" watchObservedRunningTime="2026-09-17 21:42:46.353487329 +0000 UTC m=+82.171813638"
Sep 17 21:42:49.305420 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:49.305382    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b46d69d85-wq8dq" event={"ID":"d6619f61-5f63-4e7f-865a-8f06faa70da2","Type":"ContainerStarted","Data":"e5c0fe6529d2be52b20b44cfbabaec5b8e575f9b2fb9f9fac657aec52d078273"}
Sep 17 21:42:49.305932 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:49.305654    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b46d69d85-wq8dq"
Sep 17 21:42:49.307279 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:49.307258    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b46d69d85-wq8dq"
Sep 17 21:42:49.326040 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:49.325992    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b46d69d85-wq8dq" podStartSLOduration=2.191855877 podStartE2EDuration="18.325976737s" totalImagesPullingTime="16.13412086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:42:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:42:49.325572383 +0000 UTC m=+85.143898695" watchObservedRunningTime="2026-09-17 21:42:49.325976737 +0000 UTC m=+85.144303225"
Sep 17 21:42:55.232038 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:42:55.232005    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f88f88c4b-r97mr"
Sep 17 21:43:10.262572 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:43:10.262542    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hmscl"
Sep 17 21:46:24.611041 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:46:24.611014    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdcnw_6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2/ovn-acl-logging/0.log"
Sep 17 21:46:24.613675 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:46:24.613654    2564 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 21:46:24.621601 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:46:24.621577    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdcnw_6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2/ovn-acl-logging/0.log"
Sep 17 21:47:03.437723 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:03.437684    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79"]
Sep 17 21:47:03.440713 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:03.440695    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79"
Sep 17 21:47:03.443669 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:03.443641    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 17 21:47:03.443779 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:03.443690    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 17 21:47:03.444126 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:03.444110    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 17 21:47:03.444359 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:03.444337    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-n8qrl\""
Sep 17 21:47:03.444474 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:03.444359    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 17 21:47:03.465313 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:03.465283    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79"]
Sep 17 21:47:03.598474 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:03.598427    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9da019c5-6b53-4b87-ae3a-1fad457500d5-tmp\") pod \"opendatahub-operator-controller-manager-555cdfd7b4-g9d79\" (UID: \"9da019c5-6b53-4b87-ae3a-1fad457500d5\") " pod="opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79"
Sep 17 21:47:03.598474 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:03.598480    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9da019c5-6b53-4b87-ae3a-1fad457500d5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-555cdfd7b4-g9d79\" (UID: \"9da019c5-6b53-4b87-ae3a-1fad457500d5\") " pod="opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79"
Sep 17 21:47:03.598734 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:03.598556    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9da019c5-6b53-4b87-ae3a-1fad457500d5-webhook-cert\") pod \"opendatahub-operator-controller-manager-555cdfd7b4-g9d79\" (UID: \"9da019c5-6b53-4b87-ae3a-1fad457500d5\") " pod="opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79"
Sep 17 21:47:03.598734 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:03.598601    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9da019c5-6b53-4b87-ae3a-1fad457500d5-manifests\") pod \"opendatahub-operator-controller-manager-555cdfd7b4-g9d79\" (UID: \"9da019c5-6b53-4b87-ae3a-1fad457500d5\") " pod="opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79"
Sep 17 21:47:03.598734 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:03.598659    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pq5m\" (UniqueName: \"kubernetes.io/projected/9da019c5-6b53-4b87-ae3a-1fad457500d5-kube-api-access-8pq5m\") pod \"opendatahub-operator-controller-manager-555cdfd7b4-g9d79\" (UID: \"9da019c5-6b53-4b87-ae3a-1fad457500d5\") " pod="opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79"
Sep 17 21:47:03.699994 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:03.699907    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9da019c5-6b53-4b87-ae3a-1fad457500d5-webhook-cert\") pod \"opendatahub-operator-controller-manager-555cdfd7b4-g9d79\" (UID: \"9da019c5-6b53-4b87-ae3a-1fad457500d5\") " pod="opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79"
Sep 17 21:47:03.699994 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:03.699956    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9da019c5-6b53-4b87-ae3a-1fad457500d5-manifests\") pod \"opendatahub-operator-controller-manager-555cdfd7b4-g9d79\" (UID: \"9da019c5-6b53-4b87-ae3a-1fad457500d5\") " pod="opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79"
Sep 17 21:47:03.699994 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:03.699978    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pq5m\" (UniqueName: \"kubernetes.io/projected/9da019c5-6b53-4b87-ae3a-1fad457500d5-kube-api-access-8pq5m\") pod \"opendatahub-operator-controller-manager-555cdfd7b4-g9d79\" (UID: \"9da019c5-6b53-4b87-ae3a-1fad457500d5\") " pod="opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79"
Sep 17 21:47:03.700294 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:03.700012    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9da019c5-6b53-4b87-ae3a-1fad457500d5-tmp\") pod \"opendatahub-operator-controller-manager-555cdfd7b4-g9d79\" (UID: \"9da019c5-6b53-4b87-ae3a-1fad457500d5\") " pod="opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79"
Sep 17 21:47:03.700294 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:03.700043    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9da019c5-6b53-4b87-ae3a-1fad457500d5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-555cdfd7b4-g9d79\" (UID: \"9da019c5-6b53-4b87-ae3a-1fad457500d5\") " pod="opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79"
Sep 17 21:47:03.700437 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:03.700414    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9da019c5-6b53-4b87-ae3a-1fad457500d5-manifests\") pod \"opendatahub-operator-controller-manager-555cdfd7b4-g9d79\" (UID: \"9da019c5-6b53-4b87-ae3a-1fad457500d5\") " pod="opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79"
Sep 17 21:47:03.702543 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:03.702516    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9da019c5-6b53-4b87-ae3a-1fad457500d5-tmp\") pod \"opendatahub-operator-controller-manager-555cdfd7b4-g9d79\" (UID: \"9da019c5-6b53-4b87-ae3a-1fad457500d5\") " pod="opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79"
Sep 17 21:47:03.702640 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:03.702589    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9da019c5-6b53-4b87-ae3a-1fad457500d5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-555cdfd7b4-g9d79\" (UID: \"9da019c5-6b53-4b87-ae3a-1fad457500d5\") " pod="opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79"
Sep 17 21:47:03.702700 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:03.702685    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9da019c5-6b53-4b87-ae3a-1fad457500d5-webhook-cert\") pod \"opendatahub-operator-controller-manager-555cdfd7b4-g9d79\" (UID: \"9da019c5-6b53-4b87-ae3a-1fad457500d5\") " pod="opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79"
Sep 17 21:47:03.713284 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:03.713258    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pq5m\" (UniqueName: \"kubernetes.io/projected/9da019c5-6b53-4b87-ae3a-1fad457500d5-kube-api-access-8pq5m\") pod \"opendatahub-operator-controller-manager-555cdfd7b4-g9d79\" (UID: \"9da019c5-6b53-4b87-ae3a-1fad457500d5\") " pod="opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79"
Sep 17 21:47:03.751302 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:03.751254    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79"
Sep 17 21:47:04.034836 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:04.034812    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79"]
Sep 17 21:47:04.037321 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:47:04.037298    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da019c5_6b53_4b87_ae3a_1fad457500d5.slice/crio-3b03c27084931d0cb7d4b40b472d0f46f15c9a766cbd1ffa8318becaa8ba66d6 WatchSource:0}: Error finding container 3b03c27084931d0cb7d4b40b472d0f46f15c9a766cbd1ffa8318becaa8ba66d6: Status 404 returned error can't find the container with id 3b03c27084931d0cb7d4b40b472d0f46f15c9a766cbd1ffa8318becaa8ba66d6
Sep 17 21:47:04.039039 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:04.039023    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:47:04.168367 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:04.168330    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79" event={"ID":"9da019c5-6b53-4b87-ae3a-1fad457500d5","Type":"ContainerStarted","Data":"3b03c27084931d0cb7d4b40b472d0f46f15c9a766cbd1ffa8318becaa8ba66d6"}
Sep 17 21:47:07.186534 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:07.186495    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79" event={"ID":"9da019c5-6b53-4b87-ae3a-1fad457500d5","Type":"ContainerStarted","Data":"31cbca07d5243070fd3ccb8c4b389144276b40b19210575d420d1a4a1c893649"}
Sep 17 21:47:09.195125 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:09.195091    2564 generic.go:356] "Generic (PLEG): container finished" podID="9da019c5-6b53-4b87-ae3a-1fad457500d5" containerID="31cbca07d5243070fd3ccb8c4b389144276b40b19210575d420d1a4a1c893649" exitCode=0
Sep 17 21:47:09.195520 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:09.195183    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79" event={"ID":"9da019c5-6b53-4b87-ae3a-1fad457500d5","Type":"ContainerDied","Data":"31cbca07d5243070fd3ccb8c4b389144276b40b19210575d420d1a4a1c893649"}
Sep 17 21:47:10.201287 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:10.201244    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79" event={"ID":"9da019c5-6b53-4b87-ae3a-1fad457500d5","Type":"ContainerStarted","Data":"ed9ebcf1dece4ed04431d58c376db1276afe90ac37da0f20b0d4ba3c43ba3fc7"}
Sep 17 21:47:10.201773 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:10.201480    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79"
Sep 17 21:47:10.225999 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:10.225948    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79" podStartSLOduration=2.820268477 podStartE2EDuration="7.225932562s" totalImagesPullingTime="3.369130523s" totalInitContainerRuntime="1.036533562s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:47:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:47:10.224575772 +0000 UTC m=+346.042902077" watchObservedRunningTime="2026-09-17 21:47:10.225932562 +0000 UTC m=+346.044258873"
Sep 17 21:47:21.207506 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:21.207428    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-555cdfd7b4-g9d79"
Sep 17 21:47:24.731569 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:24.731537    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-f7577f47d-jfkrw"]
Sep 17 21:47:24.739372 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:24.739348    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-f7577f47d-jfkrw"
Sep 17 21:47:24.744911 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:24.744881    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 17 21:47:24.744911 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:24.744895    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 21:47:24.744911 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:24.744910    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 21:47:24.745194 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:24.744889    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 17 21:47:24.758888 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:24.758866    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpvrj\" (UniqueName: \"kubernetes.io/projected/5169fe15-2965-44fe-888a-ff71d9fb1e1c-kube-api-access-qpvrj\") pod \"kube-auth-proxy-f7577f47d-jfkrw\" (UID: \"5169fe15-2965-44fe-888a-ff71d9fb1e1c\") " pod="openshift-ingress/kube-auth-proxy-f7577f47d-jfkrw"
Sep 17 21:47:24.759007 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:24.758905    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5169fe15-2965-44fe-888a-ff71d9fb1e1c-tmp\") pod \"kube-auth-proxy-f7577f47d-jfkrw\" (UID: \"5169fe15-2965-44fe-888a-ff71d9fb1e1c\") " pod="openshift-ingress/kube-auth-proxy-f7577f47d-jfkrw"
Sep 17 21:47:24.759007 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:24.758977    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5169fe15-2965-44fe-888a-ff71d9fb1e1c-tls-certs\") pod \"kube-auth-proxy-f7577f47d-jfkrw\" (UID: \"5169fe15-2965-44fe-888a-ff71d9fb1e1c\") " pod="openshift-ingress/kube-auth-proxy-f7577f47d-jfkrw"
Sep 17 21:47:24.859724 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:24.859681    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5169fe15-2965-44fe-888a-ff71d9fb1e1c-tls-certs\") pod \"kube-auth-proxy-f7577f47d-jfkrw\" (UID: \"5169fe15-2965-44fe-888a-ff71d9fb1e1c\") " pod="openshift-ingress/kube-auth-proxy-f7577f47d-jfkrw"
Sep 17 21:47:24.859724 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:24.859722    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpvrj\" (UniqueName: \"kubernetes.io/projected/5169fe15-2965-44fe-888a-ff71d9fb1e1c-kube-api-access-qpvrj\") pod \"kube-auth-proxy-f7577f47d-jfkrw\" (UID: \"5169fe15-2965-44fe-888a-ff71d9fb1e1c\") " pod="openshift-ingress/kube-auth-proxy-f7577f47d-jfkrw"
Sep 17 21:47:24.859943 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:24.859752    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5169fe15-2965-44fe-888a-ff71d9fb1e1c-tmp\") pod \"kube-auth-proxy-f7577f47d-jfkrw\" (UID: \"5169fe15-2965-44fe-888a-ff71d9fb1e1c\") " pod="openshift-ingress/kube-auth-proxy-f7577f47d-jfkrw"
Sep 17 21:47:24.862112 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:24.862091    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5169fe15-2965-44fe-888a-ff71d9fb1e1c-tmp\") pod \"kube-auth-proxy-f7577f47d-jfkrw\" (UID: \"5169fe15-2965-44fe-888a-ff71d9fb1e1c\") " pod="openshift-ingress/kube-auth-proxy-f7577f47d-jfkrw"
Sep 17 21:47:24.865194 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:24.862650    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5169fe15-2965-44fe-888a-ff71d9fb1e1c-tls-certs\") pod \"kube-auth-proxy-f7577f47d-jfkrw\" (UID: \"5169fe15-2965-44fe-888a-ff71d9fb1e1c\") " pod="openshift-ingress/kube-auth-proxy-f7577f47d-jfkrw"
Sep 17 21:47:24.878868 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:24.878839    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-f7577f47d-jfkrw"]
Sep 17 21:47:24.889882 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:24.889853    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpvrj\" (UniqueName: \"kubernetes.io/projected/5169fe15-2965-44fe-888a-ff71d9fb1e1c-kube-api-access-qpvrj\") pod \"kube-auth-proxy-f7577f47d-jfkrw\" (UID: \"5169fe15-2965-44fe-888a-ff71d9fb1e1c\") " pod="openshift-ingress/kube-auth-proxy-f7577f47d-jfkrw"
Sep 17 21:47:25.048661 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:25.048549    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-f7577f47d-jfkrw"
Sep 17 21:47:25.338709 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:25.338627    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-f7577f47d-jfkrw"]
Sep 17 21:47:25.344395 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:47:25.344371    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5169fe15_2965_44fe_888a_ff71d9fb1e1c.slice/crio-a1a76b45aae9f12c98e0e740afe54a7d9908247084347838efdde05cdb27b28e WatchSource:0}: Error finding container a1a76b45aae9f12c98e0e740afe54a7d9908247084347838efdde05cdb27b28e: Status 404 returned error can't find the container with id a1a76b45aae9f12c98e0e740afe54a7d9908247084347838efdde05cdb27b28e
Sep 17 21:47:26.264444 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:26.264396    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-f7577f47d-jfkrw" event={"ID":"5169fe15-2965-44fe-888a-ff71d9fb1e1c","Type":"ContainerStarted","Data":"a1a76b45aae9f12c98e0e740afe54a7d9908247084347838efdde05cdb27b28e"}
Sep 17 21:47:27.816656 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.816623    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"]
Sep 17 21:47:27.820368 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.820345    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:27.826006 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.825984    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 17 21:47:27.826281 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.826264    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-44x5w\""
Sep 17 21:47:27.830563 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.830540    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 21:47:27.830667 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.830544    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 17 21:47:27.830667 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.830593    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 17 21:47:27.843944 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.843914    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"]
Sep 17 21:47:27.885977 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.885936    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adf9ba56-93db-44e0-a506-55993c89430b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vnqfd\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:27.885977 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.885982    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/adf9ba56-93db-44e0-a506-55993c89430b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vnqfd\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:27.886272 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.886047    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/adf9ba56-93db-44e0-a506-55993c89430b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vnqfd\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:27.886272 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.886119    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/adf9ba56-93db-44e0-a506-55993c89430b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vnqfd\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:27.886272 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.886163    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/adf9ba56-93db-44e0-a506-55993c89430b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vnqfd\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:27.886272 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.886221    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g4pc\" (UniqueName: \"kubernetes.io/projected/adf9ba56-93db-44e0-a506-55993c89430b-kube-api-access-4g4pc\") pod \"istiod-openshift-gateway-7b7995d5cd-vnqfd\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:27.886272 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.886253    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/adf9ba56-93db-44e0-a506-55993c89430b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vnqfd\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:27.987672 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.987636    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/adf9ba56-93db-44e0-a506-55993c89430b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vnqfd\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:27.987672 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.987685    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/adf9ba56-93db-44e0-a506-55993c89430b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vnqfd\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:27.987932 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.987731    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g4pc\" (UniqueName: \"kubernetes.io/projected/adf9ba56-93db-44e0-a506-55993c89430b-kube-api-access-4g4pc\") pod \"istiod-openshift-gateway-7b7995d5cd-vnqfd\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:27.987932 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.987762    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/adf9ba56-93db-44e0-a506-55993c89430b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vnqfd\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:27.987932 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.987821    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adf9ba56-93db-44e0-a506-55993c89430b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vnqfd\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:27.987932 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.987836    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/adf9ba56-93db-44e0-a506-55993c89430b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vnqfd\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:27.988126 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.987971    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/adf9ba56-93db-44e0-a506-55993c89430b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vnqfd\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:27.988444 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.988408    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/adf9ba56-93db-44e0-a506-55993c89430b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vnqfd\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:27.990933 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.990911    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/adf9ba56-93db-44e0-a506-55993c89430b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vnqfd\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:27.991030 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.990957    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/adf9ba56-93db-44e0-a506-55993c89430b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vnqfd\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:27.991139 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.991109    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/adf9ba56-93db-44e0-a506-55993c89430b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vnqfd\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:27.991139 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:27.991130    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/adf9ba56-93db-44e0-a506-55993c89430b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vnqfd\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:28.004823 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:28.004797    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adf9ba56-93db-44e0-a506-55993c89430b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vnqfd\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:28.005485 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:28.005462    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g4pc\" (UniqueName: \"kubernetes.io/projected/adf9ba56-93db-44e0-a506-55993c89430b-kube-api-access-4g4pc\") pod \"istiod-openshift-gateway-7b7995d5cd-vnqfd\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:28.130894 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:28.130823    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:28.274410 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:28.274376    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-f7577f47d-jfkrw" event={"ID":"5169fe15-2965-44fe-888a-ff71d9fb1e1c","Type":"ContainerStarted","Data":"d16fab6f7d882d636dbf424a0e6443ffebde5d2db5a4216e8895103feb032c5b"}
Sep 17 21:47:28.412832 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:28.412791    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-f7577f47d-jfkrw" podStartSLOduration=1.576139055 podStartE2EDuration="4.412772344s" totalImagesPullingTime="2.836633289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:47:28.30533432 +0000 UTC m=+364.123660632" watchObservedRunningTime="2026-09-17 21:47:28.412772344 +0000 UTC m=+364.231098657"
Sep 17 21:47:28.412929 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:28.412915    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"]
Sep 17 21:47:28.415188 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:47:28.415147    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf9ba56_93db_44e0_a506_55993c89430b.slice/crio-ceef7d274d0e8e59b4a755ffac85540e772fcb2ff914c3d87c5d4cf3bad0a686 WatchSource:0}: Error finding container ceef7d274d0e8e59b4a755ffac85540e772fcb2ff914c3d87c5d4cf3bad0a686: Status 404 returned error can't find the container with id ceef7d274d0e8e59b4a755ffac85540e772fcb2ff914c3d87c5d4cf3bad0a686
Sep 17 21:47:29.279085 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:29.279049    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd" event={"ID":"adf9ba56-93db-44e0-a506-55993c89430b","Type":"ContainerStarted","Data":"ceef7d274d0e8e59b4a755ffac85540e772fcb2ff914c3d87c5d4cf3bad0a686"}
Sep 17 21:47:42.525239 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:42.525194    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:47:42.525567 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:42.525275    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:47:43.331628 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:43.331591    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd" event={"ID":"adf9ba56-93db-44e0-a506-55993c89430b","Type":"ContainerStarted","Data":"e5f50d125d5f1fa8348cd4adf1c6c0f7feefe3f63de861ec1b14b7a7e7ede8d2"}
Sep 17 21:47:43.331810 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:43.331758    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:43.333519 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:43.333495    2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-vnqfd container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 21:47:43.333632 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:43.333548    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd" podUID="adf9ba56-93db-44e0-a506-55993c89430b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:47:43.364137 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:43.364078    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd" podStartSLOduration=2.256390354 podStartE2EDuration="16.364063322s" totalImagesPullingTime="14.107672968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:47:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:47:43.361289456 +0000 UTC m=+379.179615767" watchObservedRunningTime="2026-09-17 21:47:43.364063322 +0000 UTC m=+379.182389636"
Sep 17 21:47:44.196388 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.196352    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pxwmt"]
Sep 17 21:47:44.199574 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.199555    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxwmt"
Sep 17 21:47:44.205359 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:47:44.205323    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kserve-controller-manager-d6f4b65b8-pxwmt\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="63db50f1-321e-4505-b616-65e69af22944" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxwmt"
Sep 17 21:47:44.205480 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:47:44.205366    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kserve-webhook-server-cert\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" type="*v1.Secret"
Sep 17 21:47:44.205480 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:47:44.205368    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kserve-controller-manager-dockercfg-mwkfm\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-mwkfm\"" type="*v1.Secret"
Sep 17 21:47:44.236008 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.235974    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-jjkm7"]
Sep 17 21:47:44.239588 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.239567    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jjkm7"
Sep 17 21:47:44.242750 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.242725    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 17 21:47:44.243207 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.243185    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-cs8qg\""
Sep 17 21:47:44.245007 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.244963    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pxwmt"]
Sep 17 21:47:44.325184 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.325134    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63db50f1-321e-4505-b616-65e69af22944-cert\") pod \"kserve-controller-manager-d6f4b65b8-pxwmt\" (UID: \"63db50f1-321e-4505-b616-65e69af22944\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxwmt"
Sep 17 21:47:44.325184 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.325184    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnhjn\" (UniqueName: \"kubernetes.io/projected/63db50f1-321e-4505-b616-65e69af22944-kube-api-access-lnhjn\") pod \"kserve-controller-manager-d6f4b65b8-pxwmt\" (UID: \"63db50f1-321e-4505-b616-65e69af22944\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxwmt"
Sep 17 21:47:44.325393 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.325212    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc00aec1-f082-4d1f-aef8-78b30b8672e8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jjkm7\" (UID: \"bc00aec1-f082-4d1f-aef8-78b30b8672e8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jjkm7"
Sep 17 21:47:44.325393 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.325331    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5xnb\" (UniqueName: \"kubernetes.io/projected/bc00aec1-f082-4d1f-aef8-78b30b8672e8-kube-api-access-w5xnb\") pod \"llmisvc-controller-manager-777c6bd7d5-jjkm7\" (UID: \"bc00aec1-f082-4d1f-aef8-78b30b8672e8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jjkm7"
Sep 17 21:47:44.336796 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.336764    2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-vnqfd container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 21:47:44.336932 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.336833    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd" podUID="adf9ba56-93db-44e0-a506-55993c89430b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:47:44.373668 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.373637    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-jjkm7"]
Sep 17 21:47:44.426112 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.426077    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63db50f1-321e-4505-b616-65e69af22944-cert\") pod \"kserve-controller-manager-d6f4b65b8-pxwmt\" (UID: \"63db50f1-321e-4505-b616-65e69af22944\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxwmt"
Sep 17 21:47:44.426307 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.426120    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnhjn\" (UniqueName: \"kubernetes.io/projected/63db50f1-321e-4505-b616-65e69af22944-kube-api-access-lnhjn\") pod \"kserve-controller-manager-d6f4b65b8-pxwmt\" (UID: \"63db50f1-321e-4505-b616-65e69af22944\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxwmt"
Sep 17 21:47:44.426307 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.426207    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc00aec1-f082-4d1f-aef8-78b30b8672e8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jjkm7\" (UID: \"bc00aec1-f082-4d1f-aef8-78b30b8672e8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jjkm7"
Sep 17 21:47:44.426430 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:47:44.426312    2564 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 17 21:47:44.426430 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:47:44.426380    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc00aec1-f082-4d1f-aef8-78b30b8672e8-cert podName:bc00aec1-f082-4d1f-aef8-78b30b8672e8 nodeName:}" failed. No retries permitted until 2026-09-17 21:47:44.926358999 +0000 UTC m=+380.744685306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/bc00aec1-f082-4d1f-aef8-78b30b8672e8-cert") pod "llmisvc-controller-manager-777c6bd7d5-jjkm7" (UID: "bc00aec1-f082-4d1f-aef8-78b30b8672e8") : secret "llmisvc-webhook-server-cert" not found
Sep 17 21:47:44.426546 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.426442    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5xnb\" (UniqueName: \"kubernetes.io/projected/bc00aec1-f082-4d1f-aef8-78b30b8672e8-kube-api-access-w5xnb\") pod \"llmisvc-controller-manager-777c6bd7d5-jjkm7\" (UID: \"bc00aec1-f082-4d1f-aef8-78b30b8672e8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jjkm7"
Sep 17 21:47:44.440818 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.440788    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnhjn\" (UniqueName: \"kubernetes.io/projected/63db50f1-321e-4505-b616-65e69af22944-kube-api-access-lnhjn\") pod \"kserve-controller-manager-d6f4b65b8-pxwmt\" (UID: \"63db50f1-321e-4505-b616-65e69af22944\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxwmt"
Sep 17 21:47:44.447430 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.447364    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5xnb\" (UniqueName: \"kubernetes.io/projected/bc00aec1-f082-4d1f-aef8-78b30b8672e8-kube-api-access-w5xnb\") pod \"llmisvc-controller-manager-777c6bd7d5-jjkm7\" (UID: \"bc00aec1-f082-4d1f-aef8-78b30b8672e8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jjkm7"
Sep 17 21:47:44.574416 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.574384    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"]
Sep 17 21:47:44.577762 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.577740    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.582440 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.582421    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-8hgm5\""
Sep 17 21:47:44.603805 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.603774    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"]
Sep 17 21:47:44.728719 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.728633    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70308246-88ae-41dd-ad39-6b2bd81f69c8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.728719 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.728679    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70308246-88ae-41dd-ad39-6b2bd81f69c8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.728719 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.728700    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70308246-88ae-41dd-ad39-6b2bd81f69c8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.728939 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.728800    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70308246-88ae-41dd-ad39-6b2bd81f69c8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.728939 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.728849    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70308246-88ae-41dd-ad39-6b2bd81f69c8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.728939 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.728881    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70308246-88ae-41dd-ad39-6b2bd81f69c8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.728939 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.728898    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz7v9\" (UniqueName: \"kubernetes.io/projected/70308246-88ae-41dd-ad39-6b2bd81f69c8-kube-api-access-jz7v9\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.728939 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.728914    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70308246-88ae-41dd-ad39-6b2bd81f69c8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.729087 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.728952    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70308246-88ae-41dd-ad39-6b2bd81f69c8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.829651 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.829602    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70308246-88ae-41dd-ad39-6b2bd81f69c8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.829842 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.829661    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70308246-88ae-41dd-ad39-6b2bd81f69c8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.829842 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.829691    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70308246-88ae-41dd-ad39-6b2bd81f69c8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.829842 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.829749    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70308246-88ae-41dd-ad39-6b2bd81f69c8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.829842 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.829788    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70308246-88ae-41dd-ad39-6b2bd81f69c8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.829842 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.829814    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70308246-88ae-41dd-ad39-6b2bd81f69c8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.829842 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.829840    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz7v9\" (UniqueName: \"kubernetes.io/projected/70308246-88ae-41dd-ad39-6b2bd81f69c8-kube-api-access-jz7v9\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.830155 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.830130    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70308246-88ae-41dd-ad39-6b2bd81f69c8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.830155 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.830147    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70308246-88ae-41dd-ad39-6b2bd81f69c8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.830297 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.830238    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70308246-88ae-41dd-ad39-6b2bd81f69c8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.830551 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.830297    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70308246-88ae-41dd-ad39-6b2bd81f69c8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.830669 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.830585    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70308246-88ae-41dd-ad39-6b2bd81f69c8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.830768 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.830505    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70308246-88ae-41dd-ad39-6b2bd81f69c8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.831218 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.831153    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70308246-88ae-41dd-ad39-6b2bd81f69c8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.832218 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.832195    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70308246-88ae-41dd-ad39-6b2bd81f69c8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.833123 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.833101    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70308246-88ae-41dd-ad39-6b2bd81f69c8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.849529 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.849495    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz7v9\" (UniqueName: \"kubernetes.io/projected/70308246-88ae-41dd-ad39-6b2bd81f69c8-kube-api-access-jz7v9\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.857817 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.857789    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70308246-88ae-41dd-ad39-6b2bd81f69c8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj\" (UID: \"70308246-88ae-41dd-ad39-6b2bd81f69c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.888222 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.888191    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:44.931201 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.931146    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc00aec1-f082-4d1f-aef8-78b30b8672e8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jjkm7\" (UID: \"bc00aec1-f082-4d1f-aef8-78b30b8672e8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jjkm7"
Sep 17 21:47:44.933799 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:44.933773    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc00aec1-f082-4d1f-aef8-78b30b8672e8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jjkm7\" (UID: \"bc00aec1-f082-4d1f-aef8-78b30b8672e8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jjkm7"
Sep 17 21:47:45.049527 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:45.049446    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-mwkfm\""
Sep 17 21:47:45.152493 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:45.152461    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jjkm7"
Sep 17 21:47:45.201018 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:45.200992    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"]
Sep 17 21:47:45.202685 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:47:45.202654    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70308246_88ae_41dd_ad39_6b2bd81f69c8.slice/crio-27f411d0a0c28e64047172da656f5bcefdca6524bebb96a58195b2f08825f735 WatchSource:0}: Error finding container 27f411d0a0c28e64047172da656f5bcefdca6524bebb96a58195b2f08825f735: Status 404 returned error can't find the container with id 27f411d0a0c28e64047172da656f5bcefdca6524bebb96a58195b2f08825f735
Sep 17 21:47:45.347499 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:45.347410    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj" event={"ID":"70308246-88ae-41dd-ad39-6b2bd81f69c8","Type":"ContainerStarted","Data":"27f411d0a0c28e64047172da656f5bcefdca6524bebb96a58195b2f08825f735"}
Sep 17 21:47:45.427083 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:47:45.427049    2564 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: failed to sync secret cache: timed out waiting for the condition
Sep 17 21:47:45.427273 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:47:45.427140    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63db50f1-321e-4505-b616-65e69af22944-cert podName:63db50f1-321e-4505-b616-65e69af22944 nodeName:}" failed. No retries permitted until 2026-09-17 21:47:45.927121075 +0000 UTC m=+381.745447369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/63db50f1-321e-4505-b616-65e69af22944-cert") pod "kserve-controller-manager-d6f4b65b8-pxwmt" (UID: "63db50f1-321e-4505-b616-65e69af22944") : failed to sync secret cache: timed out waiting for the condition
Sep 17 21:47:45.439768 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:45.439741    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-jjkm7"]
Sep 17 21:47:45.440785 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:47:45.440763    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbc00aec1_f082_4d1f_aef8_78b30b8672e8.slice/crio-1de4ba7b7892461d0394a86ea0bdbee5d532d5cfaa446a16fe74ad8570af5056 WatchSource:0}: Error finding container 1de4ba7b7892461d0394a86ea0bdbee5d532d5cfaa446a16fe74ad8570af5056: Status 404 returned error can't find the container with id 1de4ba7b7892461d0394a86ea0bdbee5d532d5cfaa446a16fe74ad8570af5056
Sep 17 21:47:45.705696 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:45.705662    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 17 21:47:45.940760 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:45.940713    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63db50f1-321e-4505-b616-65e69af22944-cert\") pod \"kserve-controller-manager-d6f4b65b8-pxwmt\" (UID: \"63db50f1-321e-4505-b616-65e69af22944\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxwmt"
Sep 17 21:47:45.944001 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:45.943968    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63db50f1-321e-4505-b616-65e69af22944-cert\") pod \"kserve-controller-manager-d6f4b65b8-pxwmt\" (UID: \"63db50f1-321e-4505-b616-65e69af22944\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxwmt"
Sep 17 21:47:45.998593 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:45.998510    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"]
Sep 17 21:47:45.998745 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:45.998725    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd" podUID="adf9ba56-93db-44e0-a506-55993c89430b" containerName="discovery" containerID="cri-o://e5f50d125d5f1fa8348cd4adf1c6c0f7feefe3f63de861ec1b14b7a7e7ede8d2" gracePeriod=30
Sep 17 21:47:45.999722 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:45.999518    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:46.010042 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.010010    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxwmt"
Sep 17 21:47:46.310474 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.310445    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pxwmt"]
Sep 17 21:47:46.320951 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.320927    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:46.354294 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.354160    2564 generic.go:356] "Generic (PLEG): container finished" podID="adf9ba56-93db-44e0-a506-55993c89430b" containerID="e5f50d125d5f1fa8348cd4adf1c6c0f7feefe3f63de861ec1b14b7a7e7ede8d2" exitCode=0
Sep 17 21:47:46.354294 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.354282    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"
Sep 17 21:47:46.354542 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.354315    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd" event={"ID":"adf9ba56-93db-44e0-a506-55993c89430b","Type":"ContainerDied","Data":"e5f50d125d5f1fa8348cd4adf1c6c0f7feefe3f63de861ec1b14b7a7e7ede8d2"}
Sep 17 21:47:46.354542 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.354359    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd" event={"ID":"adf9ba56-93db-44e0-a506-55993c89430b","Type":"ContainerDied","Data":"ceef7d274d0e8e59b4a755ffac85540e772fcb2ff914c3d87c5d4cf3bad0a686"}
Sep 17 21:47:46.354542 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.354387    2564 scope.go:117] "RemoveContainer" containerID="e5f50d125d5f1fa8348cd4adf1c6c0f7feefe3f63de861ec1b14b7a7e7ede8d2"
Sep 17 21:47:46.357667 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.357588    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jjkm7" event={"ID":"bc00aec1-f082-4d1f-aef8-78b30b8672e8","Type":"ContainerStarted","Data":"1de4ba7b7892461d0394a86ea0bdbee5d532d5cfaa446a16fe74ad8570af5056"}
Sep 17 21:47:46.405575 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:47:46.405542    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63db50f1_321e_4505_b616_65e69af22944.slice/crio-40d0d2439f0d479e95ba591624abe5b6250f1fb1a77782c8db54e1bac02632fb WatchSource:0}: Error finding container 40d0d2439f0d479e95ba591624abe5b6250f1fb1a77782c8db54e1bac02632fb: Status 404 returned error can't find the container with id 40d0d2439f0d479e95ba591624abe5b6250f1fb1a77782c8db54e1bac02632fb
Sep 17 21:47:46.408703 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.408677    2564 scope.go:117] "RemoveContainer" containerID="e5f50d125d5f1fa8348cd4adf1c6c0f7feefe3f63de861ec1b14b7a7e7ede8d2"
Sep 17 21:47:46.409034 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:47:46.409001    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5f50d125d5f1fa8348cd4adf1c6c0f7feefe3f63de861ec1b14b7a7e7ede8d2\": container with ID starting with e5f50d125d5f1fa8348cd4adf1c6c0f7feefe3f63de861ec1b14b7a7e7ede8d2 not found: ID does not exist" containerID="e5f50d125d5f1fa8348cd4adf1c6c0f7feefe3f63de861ec1b14b7a7e7ede8d2"
Sep 17 21:47:46.409108 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.409039    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f50d125d5f1fa8348cd4adf1c6c0f7feefe3f63de861ec1b14b7a7e7ede8d2"} err="failed to get container status \"e5f50d125d5f1fa8348cd4adf1c6c0f7feefe3f63de861ec1b14b7a7e7ede8d2\": rpc error: code = NotFound desc = could not find container \"e5f50d125d5f1fa8348cd4adf1c6c0f7feefe3f63de861ec1b14b7a7e7ede8d2\": container with ID starting with e5f50d125d5f1fa8348cd4adf1c6c0f7feefe3f63de861ec1b14b7a7e7ede8d2 not found: ID does not exist"
Sep 17 21:47:46.445147 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.445116    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/adf9ba56-93db-44e0-a506-55993c89430b-istio-csr-ca-configmap\") pod \"adf9ba56-93db-44e0-a506-55993c89430b\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") "
Sep 17 21:47:46.445147 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.445154    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4g4pc\" (UniqueName: \"kubernetes.io/projected/adf9ba56-93db-44e0-a506-55993c89430b-kube-api-access-4g4pc\") pod \"adf9ba56-93db-44e0-a506-55993c89430b\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") "
Sep 17 21:47:46.445395 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.445214    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/adf9ba56-93db-44e0-a506-55993c89430b-cacerts\") pod \"adf9ba56-93db-44e0-a506-55993c89430b\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") "
Sep 17 21:47:46.445395 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.445251    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/adf9ba56-93db-44e0-a506-55993c89430b-istio-kubeconfig\") pod \"adf9ba56-93db-44e0-a506-55993c89430b\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") "
Sep 17 21:47:46.445395 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.445307    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/adf9ba56-93db-44e0-a506-55993c89430b-istio-csr-dns-cert\") pod \"adf9ba56-93db-44e0-a506-55993c89430b\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") "
Sep 17 21:47:46.445395 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.445332    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/adf9ba56-93db-44e0-a506-55993c89430b-local-certs\") pod \"adf9ba56-93db-44e0-a506-55993c89430b\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") "
Sep 17 21:47:46.445395 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.445382    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adf9ba56-93db-44e0-a506-55993c89430b-istio-token\") pod \"adf9ba56-93db-44e0-a506-55993c89430b\" (UID: \"adf9ba56-93db-44e0-a506-55993c89430b\") "
Sep 17 21:47:46.445622 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.445557    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adf9ba56-93db-44e0-a506-55993c89430b-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "adf9ba56-93db-44e0-a506-55993c89430b" (UID: "adf9ba56-93db-44e0-a506-55993c89430b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:47:46.445677 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.445645    2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/adf9ba56-93db-44e0-a506-55993c89430b-istio-csr-ca-configmap\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:47:46.448670 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.448636    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adf9ba56-93db-44e0-a506-55993c89430b-local-certs" (OuterVolumeSpecName: "local-certs") pod "adf9ba56-93db-44e0-a506-55993c89430b" (UID: "adf9ba56-93db-44e0-a506-55993c89430b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:47:46.448995 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.448968    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adf9ba56-93db-44e0-a506-55993c89430b-cacerts" (OuterVolumeSpecName: "cacerts") pod "adf9ba56-93db-44e0-a506-55993c89430b" (UID: "adf9ba56-93db-44e0-a506-55993c89430b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:47:46.449776 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.449737    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adf9ba56-93db-44e0-a506-55993c89430b-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "adf9ba56-93db-44e0-a506-55993c89430b" (UID: "adf9ba56-93db-44e0-a506-55993c89430b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:47:46.450278 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.450235    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adf9ba56-93db-44e0-a506-55993c89430b-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "adf9ba56-93db-44e0-a506-55993c89430b" (UID: "adf9ba56-93db-44e0-a506-55993c89430b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:47:46.450366 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.450294    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf9ba56-93db-44e0-a506-55993c89430b-istio-token" (OuterVolumeSpecName: "istio-token") pod "adf9ba56-93db-44e0-a506-55993c89430b" (UID: "adf9ba56-93db-44e0-a506-55993c89430b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:47:46.450911 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.450883    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf9ba56-93db-44e0-a506-55993c89430b-kube-api-access-4g4pc" (OuterVolumeSpecName: "kube-api-access-4g4pc") pod "adf9ba56-93db-44e0-a506-55993c89430b" (UID: "adf9ba56-93db-44e0-a506-55993c89430b"). InnerVolumeSpecName "kube-api-access-4g4pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:47:46.546915 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.546800    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g4pc\" (UniqueName: \"kubernetes.io/projected/adf9ba56-93db-44e0-a506-55993c89430b-kube-api-access-4g4pc\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:47:46.546915 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.546837    2564 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/adf9ba56-93db-44e0-a506-55993c89430b-cacerts\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:47:46.546915 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.546852    2564 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/adf9ba56-93db-44e0-a506-55993c89430b-istio-kubeconfig\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:47:46.546915 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.546864    2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/adf9ba56-93db-44e0-a506-55993c89430b-istio-csr-dns-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:47:46.546915 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.546873    2564 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/adf9ba56-93db-44e0-a506-55993c89430b-local-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:47:46.546915 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.546881    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adf9ba56-93db-44e0-a506-55993c89430b-istio-token\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:47:46.708691 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.705637    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"]
Sep 17 21:47:46.720912 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.720873    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vnqfd"]
Sep 17 21:47:46.755117 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:46.755072    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adf9ba56-93db-44e0-a506-55993c89430b" path="/var/lib/kubelet/pods/adf9ba56-93db-44e0-a506-55993c89430b/volumes"
Sep 17 21:47:47.366318 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:47.366281    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxwmt" event={"ID":"63db50f1-321e-4505-b616-65e69af22944","Type":"ContainerStarted","Data":"40d0d2439f0d479e95ba591624abe5b6250f1fb1a77782c8db54e1bac02632fb"}
Sep 17 21:47:49.300854 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:49.300717    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:47:49.300854 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:49.300795    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:47:49.300854 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:49.300835    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:47:50.385348 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:50.385309    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jjkm7" event={"ID":"bc00aec1-f082-4d1f-aef8-78b30b8672e8","Type":"ContainerStarted","Data":"88621cd20e4a4637a5cf5e25f4ecf7bd323fe09599382e48a035edfca6fc91af"}
Sep 17 21:47:50.386158 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:50.385409    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jjkm7"
Sep 17 21:47:50.387258 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:50.387222    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxwmt" event={"ID":"63db50f1-321e-4505-b616-65e69af22944","Type":"ContainerStarted","Data":"36d44c0029185a5c7fbecbbcae11aa6f11455eec28e58b174eae2126b92c91ad"}
Sep 17 21:47:50.387360 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:50.387303    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxwmt"
Sep 17 21:47:50.388962 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:50.388940    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj" event={"ID":"70308246-88ae-41dd-ad39-6b2bd81f69c8","Type":"ContainerStarted","Data":"73e77cd47d142dfd429b0deff2c876838580f45f48d18ec54d41bfd11ab23729"}
Sep 17 21:47:50.410598 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:50.410545    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jjkm7" podStartSLOduration=2.545678424 podStartE2EDuration="6.410532487s" totalImagesPullingTime="3.864854063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:47:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:47:50.408789957 +0000 UTC m=+386.227116268" watchObservedRunningTime="2026-09-17 21:47:50.410532487 +0000 UTC m=+386.228858798"
Sep 17 21:47:50.450812 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:50.450765    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj" podStartSLOduration=2.355156726 podStartE2EDuration="6.450750223s" totalImagesPullingTime="4.095593497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:47:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:47:50.449994267 +0000 UTC m=+386.268320579" watchObservedRunningTime="2026-09-17 21:47:50.450750223 +0000 UTC m=+386.269076535"
Sep 17 21:47:50.474646 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:50.474591    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxwmt" podStartSLOduration=3.579913487 podStartE2EDuration="6.47457412s" totalImagesPullingTime="2.894660633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:47:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:47:50.472672295 +0000 UTC m=+386.290998608" watchObservedRunningTime="2026-09-17 21:47:50.47457412 +0000 UTC m=+386.292900468"
Sep 17 21:47:50.889051 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:50.889013    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:50.893788 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:50.893766    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:51.393079 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:51.393045    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:47:51.393864 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:47:51.393847    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj"
Sep 17 21:48:21.395220 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:48:21.395160    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jjkm7"
Sep 17 21:48:21.398237 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:48:21.398217    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxwmt"
Sep 17 21:49:00.954730 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:00.954647    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sfj9k"]
Sep 17 21:49:00.955249 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:00.954937    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adf9ba56-93db-44e0-a506-55993c89430b" containerName="discovery"
Sep 17 21:49:00.955249 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:00.954948    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="adf9ba56-93db-44e0-a506-55993c89430b" containerName="discovery"
Sep 17 21:49:00.955249 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:00.954997    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="adf9ba56-93db-44e0-a506-55993c89430b" containerName="discovery"
Sep 17 21:49:00.957983 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:00.957965    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sfj9k"
Sep 17 21:49:00.972274 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:00.972247    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-m8x5s\""
Sep 17 21:49:00.972428 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:00.972411    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 21:49:00.972869 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:00.972852    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 21:49:00.973094 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:00.973074    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 17 21:49:00.994484 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:00.994458    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sfj9k"]
Sep 17 21:49:01.031413 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:01.031381    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j85c5\" (UniqueName: \"kubernetes.io/projected/307933ba-b117-4062-971b-765cc92e9a1c-kube-api-access-j85c5\") pod \"dns-operator-controller-manager-7b84cd7c9c-sfj9k\" (UID: \"307933ba-b117-4062-971b-765cc92e9a1c\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sfj9k"
Sep 17 21:49:01.131964 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:01.131930    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j85c5\" (UniqueName: \"kubernetes.io/projected/307933ba-b117-4062-971b-765cc92e9a1c-kube-api-access-j85c5\") pod \"dns-operator-controller-manager-7b84cd7c9c-sfj9k\" (UID: \"307933ba-b117-4062-971b-765cc92e9a1c\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sfj9k"
Sep 17 21:49:01.143754 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:01.143724    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j85c5\" (UniqueName: \"kubernetes.io/projected/307933ba-b117-4062-971b-765cc92e9a1c-kube-api-access-j85c5\") pod \"dns-operator-controller-manager-7b84cd7c9c-sfj9k\" (UID: \"307933ba-b117-4062-971b-765cc92e9a1c\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sfj9k"
Sep 17 21:49:01.267914 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:01.267819    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sfj9k"
Sep 17 21:49:01.552581 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:01.552555    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sfj9k"]
Sep 17 21:49:01.554651 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:49:01.554626    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod307933ba_b117_4062_971b_765cc92e9a1c.slice/crio-a5e289b8d924e9f2af206e94dc47d9c2dbf5b6722dd2644294ee9f059304fc84 WatchSource:0}: Error finding container a5e289b8d924e9f2af206e94dc47d9c2dbf5b6722dd2644294ee9f059304fc84: Status 404 returned error can't find the container with id a5e289b8d924e9f2af206e94dc47d9c2dbf5b6722dd2644294ee9f059304fc84
Sep 17 21:49:01.664406 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:01.664365    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sfj9k" event={"ID":"307933ba-b117-4062-971b-765cc92e9a1c","Type":"ContainerStarted","Data":"a5e289b8d924e9f2af206e94dc47d9c2dbf5b6722dd2644294ee9f059304fc84"}
Sep 17 21:49:04.680681 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:04.680639    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sfj9k" event={"ID":"307933ba-b117-4062-971b-765cc92e9a1c","Type":"ContainerStarted","Data":"77d44d919442bafb2545ccc860e54e72d6661a615c0633507bbfefa6485ea4be"}
Sep 17 21:49:04.681103 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:04.680745    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sfj9k"
Sep 17 21:49:04.723425 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:04.723373    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sfj9k" podStartSLOduration=2.067839843 podStartE2EDuration="4.723357361s" totalImagesPullingTime="2.655517518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:49:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:49:04.721852992 +0000 UTC m=+460.540179305" watchObservedRunningTime="2026-09-17 21:49:04.723357361 +0000 UTC m=+460.541683672"
Sep 17 21:49:05.726872 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:05.726842    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"]
Sep 17 21:49:05.730029 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:05.730012    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"
Sep 17 21:49:05.733500 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:05.733484    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-z6995\""
Sep 17 21:49:05.758054 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:05.758024    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"]
Sep 17 21:49:05.871451 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:05.871410    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlk7c\" (UniqueName: \"kubernetes.io/projected/c1886cfc-6d0d-422a-94cd-81ff713e1b9e-kube-api-access-qlk7c\") pod \"kuadrant-operator-controller-manager-76f8f686bf-79rq6\" (UID: \"c1886cfc-6d0d-422a-94cd-81ff713e1b9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"
Sep 17 21:49:05.871451 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:05.871456    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1886cfc-6d0d-422a-94cd-81ff713e1b9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-79rq6\" (UID: \"c1886cfc-6d0d-422a-94cd-81ff713e1b9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"
Sep 17 21:49:05.972528 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:05.972491    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlk7c\" (UniqueName: \"kubernetes.io/projected/c1886cfc-6d0d-422a-94cd-81ff713e1b9e-kube-api-access-qlk7c\") pod \"kuadrant-operator-controller-manager-76f8f686bf-79rq6\" (UID: \"c1886cfc-6d0d-422a-94cd-81ff713e1b9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"
Sep 17 21:49:05.972688 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:05.972534    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1886cfc-6d0d-422a-94cd-81ff713e1b9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-79rq6\" (UID: \"c1886cfc-6d0d-422a-94cd-81ff713e1b9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"
Sep 17 21:49:05.972996 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:05.972976    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1886cfc-6d0d-422a-94cd-81ff713e1b9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-79rq6\" (UID: \"c1886cfc-6d0d-422a-94cd-81ff713e1b9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"
Sep 17 21:49:06.001437 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:06.001371    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlk7c\" (UniqueName: \"kubernetes.io/projected/c1886cfc-6d0d-422a-94cd-81ff713e1b9e-kube-api-access-qlk7c\") pod \"kuadrant-operator-controller-manager-76f8f686bf-79rq6\" (UID: \"c1886cfc-6d0d-422a-94cd-81ff713e1b9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"
Sep 17 21:49:06.039880 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:06.039850    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"
Sep 17 21:49:06.366522 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:06.366490    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"]
Sep 17 21:49:06.367895 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:49:06.367859    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1886cfc_6d0d_422a_94cd_81ff713e1b9e.slice/crio-2d8c842f893d3f90b7f8c17f09f672456e32a538784a4b83ce73a91293694f64 WatchSource:0}: Error finding container 2d8c842f893d3f90b7f8c17f09f672456e32a538784a4b83ce73a91293694f64: Status 404 returned error can't find the container with id 2d8c842f893d3f90b7f8c17f09f672456e32a538784a4b83ce73a91293694f64
Sep 17 21:49:06.690266 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:06.690230    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6" event={"ID":"c1886cfc-6d0d-422a-94cd-81ff713e1b9e","Type":"ContainerStarted","Data":"2d8c842f893d3f90b7f8c17f09f672456e32a538784a4b83ce73a91293694f64"}
Sep 17 21:49:08.235342 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:08.235310    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-5k9n9"]
Sep 17 21:49:08.238528 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:08.238509    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-5k9n9"
Sep 17 21:49:08.251466 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:49:08.251437    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"authorino-operator-76b478bd78-5k9n9\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="62702f8b-12b0-4adb-9f3f-5a02cb1f31e1" pod="kuadrant-system/authorino-operator-76b478bd78-5k9n9"
Sep 17 21:49:08.251699 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:49:08.251677    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"authorino-operator-dockercfg-zk8tl\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-zk8tl\"" type="*v1.Secret"
Sep 17 21:49:08.286001 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:08.285971    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-5k9n9"]
Sep 17 21:49:08.393031 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:08.392999    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgrch\" (UniqueName: \"kubernetes.io/projected/62702f8b-12b0-4adb-9f3f-5a02cb1f31e1-kube-api-access-fgrch\") pod \"authorino-operator-76b478bd78-5k9n9\" (UID: \"62702f8b-12b0-4adb-9f3f-5a02cb1f31e1\") " pod="kuadrant-system/authorino-operator-76b478bd78-5k9n9"
Sep 17 21:49:08.494283 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:08.494113    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgrch\" (UniqueName: \"kubernetes.io/projected/62702f8b-12b0-4adb-9f3f-5a02cb1f31e1-kube-api-access-fgrch\") pod \"authorino-operator-76b478bd78-5k9n9\" (UID: \"62702f8b-12b0-4adb-9f3f-5a02cb1f31e1\") " pod="kuadrant-system/authorino-operator-76b478bd78-5k9n9"
Sep 17 21:49:08.511990 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:08.511961    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgrch\" (UniqueName: \"kubernetes.io/projected/62702f8b-12b0-4adb-9f3f-5a02cb1f31e1-kube-api-access-fgrch\") pod \"authorino-operator-76b478bd78-5k9n9\" (UID: \"62702f8b-12b0-4adb-9f3f-5a02cb1f31e1\") " pod="kuadrant-system/authorino-operator-76b478bd78-5k9n9"
Sep 17 21:49:09.549533 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:09.549493    2564 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="kuadrant-system/authorino-operator-76b478bd78-5k9n9" secret="" err="failed to sync secret cache: timed out waiting for the condition"
Sep 17 21:49:09.549989 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:09.549581    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-5k9n9"
Sep 17 21:49:09.723129 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:09.723089    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-zk8tl\""
Sep 17 21:49:09.839890 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:09.839862    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-5k9n9"]
Sep 17 21:49:10.546896 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:10.546849    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs"]
Sep 17 21:49:10.551515 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:10.551493    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs"
Sep 17 21:49:10.554563 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:10.554539    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-mvw4j\""
Sep 17 21:49:10.576228 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:10.576195    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs"]
Sep 17 21:49:10.714087 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:10.714057    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlk6r\" (UniqueName: \"kubernetes.io/projected/db6a016d-e7ac-4295-b4bd-ee2c8169f24e-kube-api-access-mlk6r\") pod \"limitador-operator-controller-manager-74495c69f8-rgvgs\" (UID: \"db6a016d-e7ac-4295-b4bd-ee2c8169f24e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs"
Sep 17 21:49:10.815123 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:10.815025    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlk6r\" (UniqueName: \"kubernetes.io/projected/db6a016d-e7ac-4295-b4bd-ee2c8169f24e-kube-api-access-mlk6r\") pod \"limitador-operator-controller-manager-74495c69f8-rgvgs\" (UID: \"db6a016d-e7ac-4295-b4bd-ee2c8169f24e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs"
Sep 17 21:49:10.838721 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:10.838689    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlk6r\" (UniqueName: \"kubernetes.io/projected/db6a016d-e7ac-4295-b4bd-ee2c8169f24e-kube-api-access-mlk6r\") pod \"limitador-operator-controller-manager-74495c69f8-rgvgs\" (UID: \"db6a016d-e7ac-4295-b4bd-ee2c8169f24e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs"
Sep 17 21:49:10.863495 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:10.863461    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs"
Sep 17 21:49:10.881151 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:49:10.881123    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62702f8b_12b0_4adb_9f3f_5a02cb1f31e1.slice/crio-454c8410dcc80b8148b7bf51d2740e0d72cd1fd4de13ee65dcfc1b8939ab7960 WatchSource:0}: Error finding container 454c8410dcc80b8148b7bf51d2740e0d72cd1fd4de13ee65dcfc1b8939ab7960: Status 404 returned error can't find the container with id 454c8410dcc80b8148b7bf51d2740e0d72cd1fd4de13ee65dcfc1b8939ab7960
Sep 17 21:49:11.149568 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:11.149542    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs"]
Sep 17 21:49:11.720431 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:11.720386    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6" event={"ID":"c1886cfc-6d0d-422a-94cd-81ff713e1b9e","Type":"ContainerStarted","Data":"901717f181d25cbecba7809739c50397308519e171fb44dd934a0cddf1ac4a70"}
Sep 17 21:49:11.720910 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:11.720574    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"
Sep 17 21:49:11.721953 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:11.721914    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-5k9n9" event={"ID":"62702f8b-12b0-4adb-9f3f-5a02cb1f31e1","Type":"ContainerStarted","Data":"454c8410dcc80b8148b7bf51d2740e0d72cd1fd4de13ee65dcfc1b8939ab7960"}
Sep 17 21:49:11.723554 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:11.723528    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs" event={"ID":"db6a016d-e7ac-4295-b4bd-ee2c8169f24e","Type":"ContainerStarted","Data":"94f9db6cbf6dbe7add24670dc62b8059e89bea5c6dd4303e6f2dc29285378054"}
Sep 17 21:49:11.755146 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:11.755090    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6" podStartSLOduration=2.165465959 podStartE2EDuration="6.755067772s" totalImagesPullingTime="4.589601813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:49:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:49:11.753288935 +0000 UTC m=+467.571615281" watchObservedRunningTime="2026-09-17 21:49:11.755067772 +0000 UTC m=+467.573394090"
Sep 17 21:49:12.731521 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:12.731429    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-5k9n9" event={"ID":"62702f8b-12b0-4adb-9f3f-5a02cb1f31e1","Type":"ContainerStarted","Data":"f162bffd46e9f47dd32e03434d4cc85446080eab281c87e3aad9b184dc828369"}
Sep 17 21:49:12.754134 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:12.754075    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-5k9n9" podStartSLOduration=3.237177379 podStartE2EDuration="4.754052953s" totalImagesPullingTime="1.516875574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:49:12.752418433 +0000 UTC m=+468.570744763" watchObservedRunningTime="2026-09-17 21:49:12.754052953 +0000 UTC m=+468.572379274"
Sep 17 21:49:13.737740 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:13.737706    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs" event={"ID":"db6a016d-e7ac-4295-b4bd-ee2c8169f24e","Type":"ContainerStarted","Data":"0f0637777d5898ddf9dc58a099b44482f80706fc9bfdac27250731ed8ed70b00"}
Sep 17 21:49:13.738243 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:13.737818    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-5k9n9"
Sep 17 21:49:13.738243 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:13.738059    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs"
Sep 17 21:49:15.686558 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:15.686524    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sfj9k"
Sep 17 21:49:15.707390 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:15.707339    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs" podStartSLOduration=3.938535855 podStartE2EDuration="5.707322632s" totalImagesPullingTime="1.768786777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:49:13.767766048 +0000 UTC m=+469.586092350" watchObservedRunningTime="2026-09-17 21:49:15.707322632 +0000 UTC m=+471.525648944"
Sep 17 21:49:22.733689 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:22.733654    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"
Sep 17 21:49:24.758657 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:24.758626    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-5k9n9"
Sep 17 21:49:24.759079 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:24.758693    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs"
Sep 17 21:49:25.267605 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.267570    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"]
Sep 17 21:49:25.267783 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.267758    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6" podUID="c1886cfc-6d0d-422a-94cd-81ff713e1b9e" containerName="manager" containerID="cri-o://901717f181d25cbecba7809739c50397308519e171fb44dd934a0cddf1ac4a70" gracePeriod=2
Sep 17 21:49:25.281590 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.281564    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"]
Sep 17 21:49:25.313960 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.313925    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs"]
Sep 17 21:49:25.314149 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.314128    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs" podUID="db6a016d-e7ac-4295-b4bd-ee2c8169f24e" containerName="manager" containerID="cri-o://0f0637777d5898ddf9dc58a099b44482f80706fc9bfdac27250731ed8ed70b00" gracePeriod=2
Sep 17 21:49:25.316796 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:49:25.316764    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-79rq6\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="c1886cfc-6d0d-422a-94cd-81ff713e1b9e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"
Sep 17 21:49:25.335709 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.335676    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs"]
Sep 17 21:49:25.364420 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.364385    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdx6"]
Sep 17 21:49:25.364741 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.364723    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1886cfc-6d0d-422a-94cd-81ff713e1b9e" containerName="manager"
Sep 17 21:49:25.364741 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.364741    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1886cfc-6d0d-422a-94cd-81ff713e1b9e" containerName="manager"
Sep 17 21:49:25.364900 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.364758    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db6a016d-e7ac-4295-b4bd-ee2c8169f24e" containerName="manager"
Sep 17 21:49:25.364900 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.364766    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="db6a016d-e7ac-4295-b4bd-ee2c8169f24e" containerName="manager"
Sep 17 21:49:25.364900 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.364847    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1886cfc-6d0d-422a-94cd-81ff713e1b9e" containerName="manager"
Sep 17 21:49:25.364900 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.364860    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="db6a016d-e7ac-4295-b4bd-ee2c8169f24e" containerName="manager"
Sep 17 21:49:25.367982 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.367959    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdx6"
Sep 17 21:49:25.372461 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:49:25.372431    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-79rq6\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="c1886cfc-6d0d-422a-94cd-81ff713e1b9e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"
Sep 17 21:49:25.378031 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:49:25.378004    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rgvgs\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="db6a016d-e7ac-4295-b4bd-ee2c8169f24e" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs"
Sep 17 21:49:25.420667 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.420454    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-f6xcz"]
Sep 17 21:49:25.423952 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.423920    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdx6"]
Sep 17 21:49:25.424077 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.424044    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f6xcz"
Sep 17 21:49:25.433343 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.432952    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vz78\" (UniqueName: \"kubernetes.io/projected/c1015b08-0358-412e-b5a1-0910a4de69f5-kube-api-access-7vz78\") pod \"limitador-operator-controller-manager-74495c69f8-f6xcz\" (UID: \"c1015b08-0358-412e-b5a1-0910a4de69f5\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f6xcz"
Sep 17 21:49:25.433343 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.433017    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/834734bd-21b7-46bf-887d-33e00a91a7f7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ngdx6\" (UID: \"834734bd-21b7-46bf-887d-33e00a91a7f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdx6"
Sep 17 21:49:25.433343 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.433094    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flgvb\" (UniqueName: \"kubernetes.io/projected/834734bd-21b7-46bf-887d-33e00a91a7f7-kube-api-access-flgvb\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ngdx6\" (UID: \"834734bd-21b7-46bf-887d-33e00a91a7f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdx6"
Sep 17 21:49:25.453781 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.453688    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-f6xcz"]
Sep 17 21:49:25.457589 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:49:25.456618    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-79rq6\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="c1886cfc-6d0d-422a-94cd-81ff713e1b9e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"
Sep 17 21:49:25.459292 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:49:25.459254    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rgvgs\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="db6a016d-e7ac-4295-b4bd-ee2c8169f24e" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs"
Sep 17 21:49:25.533484 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.533453    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vz78\" (UniqueName: \"kubernetes.io/projected/c1015b08-0358-412e-b5a1-0910a4de69f5-kube-api-access-7vz78\") pod \"limitador-operator-controller-manager-74495c69f8-f6xcz\" (UID: \"c1015b08-0358-412e-b5a1-0910a4de69f5\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f6xcz"
Sep 17 21:49:25.533618 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.533507    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/834734bd-21b7-46bf-887d-33e00a91a7f7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ngdx6\" (UID: \"834734bd-21b7-46bf-887d-33e00a91a7f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdx6"
Sep 17 21:49:25.533618 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.533565    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flgvb\" (UniqueName: \"kubernetes.io/projected/834734bd-21b7-46bf-887d-33e00a91a7f7-kube-api-access-flgvb\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ngdx6\" (UID: \"834734bd-21b7-46bf-887d-33e00a91a7f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdx6"
Sep 17 21:49:25.533936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.533916    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/834734bd-21b7-46bf-887d-33e00a91a7f7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ngdx6\" (UID: \"834734bd-21b7-46bf-887d-33e00a91a7f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdx6"
Sep 17 21:49:25.552018 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.551995    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"
Sep 17 21:49:25.579278 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.579257    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs"
Sep 17 21:49:25.583544 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.583524    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flgvb\" (UniqueName: \"kubernetes.io/projected/834734bd-21b7-46bf-887d-33e00a91a7f7-kube-api-access-flgvb\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ngdx6\" (UID: \"834734bd-21b7-46bf-887d-33e00a91a7f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdx6"
Sep 17 21:49:25.588320 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.588304    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vz78\" (UniqueName: \"kubernetes.io/projected/c1015b08-0358-412e-b5a1-0910a4de69f5-kube-api-access-7vz78\") pod \"limitador-operator-controller-manager-74495c69f8-f6xcz\" (UID: \"c1015b08-0358-412e-b5a1-0910a4de69f5\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f6xcz"
Sep 17 21:49:25.593764 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:49:25.593740    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-79rq6\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="c1886cfc-6d0d-422a-94cd-81ff713e1b9e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"
Sep 17 21:49:25.608403 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:49:25.608373    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rgvgs\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="db6a016d-e7ac-4295-b4bd-ee2c8169f24e" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs"
Sep 17 21:49:25.614521 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:49:25.614494    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-79rq6\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="c1886cfc-6d0d-422a-94cd-81ff713e1b9e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"
Sep 17 21:49:25.618921 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:49:25.618897    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rgvgs\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="db6a016d-e7ac-4295-b4bd-ee2c8169f24e" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs"
Sep 17 21:49:25.634266 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.634247    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlk6r\" (UniqueName: \"kubernetes.io/projected/db6a016d-e7ac-4295-b4bd-ee2c8169f24e-kube-api-access-mlk6r\") pod \"db6a016d-e7ac-4295-b4bd-ee2c8169f24e\" (UID: \"db6a016d-e7ac-4295-b4bd-ee2c8169f24e\") "
Sep 17 21:49:25.634379 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.634292    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1886cfc-6d0d-422a-94cd-81ff713e1b9e-extensions-socket-volume\") pod \"c1886cfc-6d0d-422a-94cd-81ff713e1b9e\" (UID: \"c1886cfc-6d0d-422a-94cd-81ff713e1b9e\") "
Sep 17 21:49:25.634379 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.634320    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlk7c\" (UniqueName: \"kubernetes.io/projected/c1886cfc-6d0d-422a-94cd-81ff713e1b9e-kube-api-access-qlk7c\") pod \"c1886cfc-6d0d-422a-94cd-81ff713e1b9e\" (UID: \"c1886cfc-6d0d-422a-94cd-81ff713e1b9e\") "
Sep 17 21:49:25.634724 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.634698    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1886cfc-6d0d-422a-94cd-81ff713e1b9e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c1886cfc-6d0d-422a-94cd-81ff713e1b9e" (UID: "c1886cfc-6d0d-422a-94cd-81ff713e1b9e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:49:25.636662 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.636642    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db6a016d-e7ac-4295-b4bd-ee2c8169f24e-kube-api-access-mlk6r" (OuterVolumeSpecName: "kube-api-access-mlk6r") pod "db6a016d-e7ac-4295-b4bd-ee2c8169f24e" (UID: "db6a016d-e7ac-4295-b4bd-ee2c8169f24e"). InnerVolumeSpecName "kube-api-access-mlk6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:49:25.636730 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.636657    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1886cfc-6d0d-422a-94cd-81ff713e1b9e-kube-api-access-qlk7c" (OuterVolumeSpecName: "kube-api-access-qlk7c") pod "c1886cfc-6d0d-422a-94cd-81ff713e1b9e" (UID: "c1886cfc-6d0d-422a-94cd-81ff713e1b9e"). InnerVolumeSpecName "kube-api-access-qlk7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:49:25.713284 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.713259    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdx6"
Sep 17 21:49:25.735690 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.735663    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1886cfc-6d0d-422a-94cd-81ff713e1b9e-extensions-socket-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:49:25.735690 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.735690    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlk7c\" (UniqueName: \"kubernetes.io/projected/c1886cfc-6d0d-422a-94cd-81ff713e1b9e-kube-api-access-qlk7c\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:49:25.735829 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.735704    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlk6r\" (UniqueName: \"kubernetes.io/projected/db6a016d-e7ac-4295-b4bd-ee2c8169f24e-kube-api-access-mlk6r\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:49:25.737478 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.737454    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f6xcz"
Sep 17 21:49:25.799587 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.799555    2564 generic.go:356] "Generic (PLEG): container finished" podID="c1886cfc-6d0d-422a-94cd-81ff713e1b9e" containerID="901717f181d25cbecba7809739c50397308519e171fb44dd934a0cddf1ac4a70" exitCode=0
Sep 17 21:49:25.799952 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.799716    2564 scope.go:117] "RemoveContainer" containerID="901717f181d25cbecba7809739c50397308519e171fb44dd934a0cddf1ac4a70"
Sep 17 21:49:25.799952 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.799760    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"
Sep 17 21:49:25.802657 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.802628    2564 generic.go:356] "Generic (PLEG): container finished" podID="db6a016d-e7ac-4295-b4bd-ee2c8169f24e" containerID="0f0637777d5898ddf9dc58a099b44482f80706fc9bfdac27250731ed8ed70b00" exitCode=0
Sep 17 21:49:25.802753 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.802728    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs"
Sep 17 21:49:25.803551 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:49:25.803518    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-79rq6\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="c1886cfc-6d0d-422a-94cd-81ff713e1b9e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"
Sep 17 21:49:25.808885 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:49:25.808707    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rgvgs\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="db6a016d-e7ac-4295-b4bd-ee2c8169f24e" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs"
Sep 17 21:49:25.815188 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.815141    2564 scope.go:117] "RemoveContainer" containerID="901717f181d25cbecba7809739c50397308519e171fb44dd934a0cddf1ac4a70"
Sep 17 21:49:25.815594 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:49:25.815487    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"901717f181d25cbecba7809739c50397308519e171fb44dd934a0cddf1ac4a70\": container with ID starting with 901717f181d25cbecba7809739c50397308519e171fb44dd934a0cddf1ac4a70 not found: ID does not exist" containerID="901717f181d25cbecba7809739c50397308519e171fb44dd934a0cddf1ac4a70"
Sep 17 21:49:25.815594 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.815513    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"901717f181d25cbecba7809739c50397308519e171fb44dd934a0cddf1ac4a70"} err="failed to get container status \"901717f181d25cbecba7809739c50397308519e171fb44dd934a0cddf1ac4a70\": rpc error: code = NotFound desc = could not find container \"901717f181d25cbecba7809739c50397308519e171fb44dd934a0cddf1ac4a70\": container with ID starting with 901717f181d25cbecba7809739c50397308519e171fb44dd934a0cddf1ac4a70 not found: ID does not exist"
Sep 17 21:49:25.815594 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.815530    2564 scope.go:117] "RemoveContainer" containerID="0f0637777d5898ddf9dc58a099b44482f80706fc9bfdac27250731ed8ed70b00"
Sep 17 21:49:25.819889 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:49:25.819810    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-79rq6\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="c1886cfc-6d0d-422a-94cd-81ff713e1b9e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"
Sep 17 21:49:25.823859 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.823831    2564 scope.go:117] "RemoveContainer" containerID="0f0637777d5898ddf9dc58a099b44482f80706fc9bfdac27250731ed8ed70b00"
Sep 17 21:49:25.824121 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:49:25.824104    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f0637777d5898ddf9dc58a099b44482f80706fc9bfdac27250731ed8ed70b00\": container with ID starting with 0f0637777d5898ddf9dc58a099b44482f80706fc9bfdac27250731ed8ed70b00 not found: ID does not exist" containerID="0f0637777d5898ddf9dc58a099b44482f80706fc9bfdac27250731ed8ed70b00"
Sep 17 21:49:25.824162 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:25.824130    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f0637777d5898ddf9dc58a099b44482f80706fc9bfdac27250731ed8ed70b00"} err="failed to get container status \"0f0637777d5898ddf9dc58a099b44482f80706fc9bfdac27250731ed8ed70b00\": rpc error: code = NotFound desc = could not find container \"0f0637777d5898ddf9dc58a099b44482f80706fc9bfdac27250731ed8ed70b00\": container with ID starting with 0f0637777d5898ddf9dc58a099b44482f80706fc9bfdac27250731ed8ed70b00 not found: ID does not exist"
Sep 17 21:49:25.836079 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:49:25.836054    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rgvgs\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="db6a016d-e7ac-4295-b4bd-ee2c8169f24e" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs"
Sep 17 21:49:25.842903 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:49:25.842871    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-79rq6\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="c1886cfc-6d0d-422a-94cd-81ff713e1b9e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79rq6"
Sep 17 21:49:25.846907 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:49:25.846870    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rgvgs\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="db6a016d-e7ac-4295-b4bd-ee2c8169f24e" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rgvgs"
Sep 17 21:49:26.037537 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:26.037506    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdx6"]
Sep 17 21:49:26.046238 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:49:26.046160    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod834734bd_21b7_46bf_887d_33e00a91a7f7.slice/crio-88d1bf88a7ad473f08e99518bf99a9f23e7ee7c0b0f90069cbc586d64757535f WatchSource:0}: Error finding container 88d1bf88a7ad473f08e99518bf99a9f23e7ee7c0b0f90069cbc586d64757535f: Status 404 returned error can't find the container with id 88d1bf88a7ad473f08e99518bf99a9f23e7ee7c0b0f90069cbc586d64757535f
Sep 17 21:49:26.092082 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:26.092057    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-f6xcz"]
Sep 17 21:49:26.096834 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:49:26.096806    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1015b08_0358_412e_b5a1_0910a4de69f5.slice/crio-5b3606920416cd96abf1c7fa48cdff9952aa2473def4be844974d83cfb1018df WatchSource:0}: Error finding container 5b3606920416cd96abf1c7fa48cdff9952aa2473def4be844974d83cfb1018df: Status 404 returned error can't find the container with id 5b3606920416cd96abf1c7fa48cdff9952aa2473def4be844974d83cfb1018df
Sep 17 21:49:26.758070 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:26.753776    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1886cfc-6d0d-422a-94cd-81ff713e1b9e" path="/var/lib/kubelet/pods/c1886cfc-6d0d-422a-94cd-81ff713e1b9e/volumes"
Sep 17 21:49:26.758070 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:26.754318    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db6a016d-e7ac-4295-b4bd-ee2c8169f24e" path="/var/lib/kubelet/pods/db6a016d-e7ac-4295-b4bd-ee2c8169f24e/volumes"
Sep 17 21:49:26.808955 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:26.808911    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdx6" event={"ID":"834734bd-21b7-46bf-887d-33e00a91a7f7","Type":"ContainerStarted","Data":"e0c3c1d072c68a08599f8263208abe53ea0c613880dfa52ce4d1483483658bfd"}
Sep 17 21:49:26.809392 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:26.808974    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdx6" event={"ID":"834734bd-21b7-46bf-887d-33e00a91a7f7","Type":"ContainerStarted","Data":"88d1bf88a7ad473f08e99518bf99a9f23e7ee7c0b0f90069cbc586d64757535f"}
Sep 17 21:49:26.809392 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:26.809003    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdx6"
Sep 17 21:49:26.811571 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:26.811543    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f6xcz" event={"ID":"c1015b08-0358-412e-b5a1-0910a4de69f5","Type":"ContainerStarted","Data":"d77b0027bcda032bc5ec1bdd27ed0b87b31f87214c7a58720d08c2bd7789614c"}
Sep 17 21:49:26.811748 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:26.811589    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f6xcz" event={"ID":"c1015b08-0358-412e-b5a1-0910a4de69f5","Type":"ContainerStarted","Data":"5b3606920416cd96abf1c7fa48cdff9952aa2473def4be844974d83cfb1018df"}
Sep 17 21:49:26.811748 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:26.811611    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f6xcz"
Sep 17 21:49:26.899279 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:26.899228    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdx6" podStartSLOduration=1.899204706 podStartE2EDuration="1.899204706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:49:26.889182441 +0000 UTC m=+482.707508744" watchObservedRunningTime="2026-09-17 21:49:26.899204706 +0000 UTC m=+482.717531018"
Sep 17 21:49:26.949516 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:26.949461    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f6xcz" podStartSLOduration=1.949443891 podStartE2EDuration="1.949443891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:49:26.949414456 +0000 UTC m=+482.767740767" watchObservedRunningTime="2026-09-17 21:49:26.949443891 +0000 UTC m=+482.767770203"
Sep 17 21:49:28.061127 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:28.061094    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf"]
Sep 17 21:49:28.064592 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:28.064574    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf"
Sep 17 21:49:28.101314 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:28.101280    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf"]
Sep 17 21:49:28.150680 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:28.150651    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b923825-e691-4554-aef8-c604b8fa3749-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dxkvf\" (UID: \"6b923825-e691-4554-aef8-c604b8fa3749\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf"
Sep 17 21:49:28.150680 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:28.150683    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7wlx\" (UniqueName: \"kubernetes.io/projected/6b923825-e691-4554-aef8-c604b8fa3749-kube-api-access-h7wlx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dxkvf\" (UID: \"6b923825-e691-4554-aef8-c604b8fa3749\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf"
Sep 17 21:49:28.251158 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:28.251122    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b923825-e691-4554-aef8-c604b8fa3749-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dxkvf\" (UID: \"6b923825-e691-4554-aef8-c604b8fa3749\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf"
Sep 17 21:49:28.251158 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:28.251157    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7wlx\" (UniqueName: \"kubernetes.io/projected/6b923825-e691-4554-aef8-c604b8fa3749-kube-api-access-h7wlx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dxkvf\" (UID: \"6b923825-e691-4554-aef8-c604b8fa3749\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf"
Sep 17 21:49:28.251509 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:28.251486    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b923825-e691-4554-aef8-c604b8fa3749-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dxkvf\" (UID: \"6b923825-e691-4554-aef8-c604b8fa3749\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf"
Sep 17 21:49:28.262241 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:28.262203    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7wlx\" (UniqueName: \"kubernetes.io/projected/6b923825-e691-4554-aef8-c604b8fa3749-kube-api-access-h7wlx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dxkvf\" (UID: \"6b923825-e691-4554-aef8-c604b8fa3749\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf"
Sep 17 21:49:28.374157 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:28.374075    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf"
Sep 17 21:49:28.666253 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:28.666227    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf"]
Sep 17 21:49:28.667881 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:49:28.667845    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b923825_e691_4554_aef8_c604b8fa3749.slice/crio-146beaaf8e13f2dbad98466f0ab6d92e538f802fdb36d59cf42de3e637407b31 WatchSource:0}: Error finding container 146beaaf8e13f2dbad98466f0ab6d92e538f802fdb36d59cf42de3e637407b31: Status 404 returned error can't find the container with id 146beaaf8e13f2dbad98466f0ab6d92e538f802fdb36d59cf42de3e637407b31
Sep 17 21:49:28.823746 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:28.823699    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf" event={"ID":"6b923825-e691-4554-aef8-c604b8fa3749","Type":"ContainerStarted","Data":"03f5a5e0832b6f9e684484d9726470a3da140cc1b40b9dfb043693d60b996305"}
Sep 17 21:49:28.823916 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:28.823763    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf" event={"ID":"6b923825-e691-4554-aef8-c604b8fa3749","Type":"ContainerStarted","Data":"146beaaf8e13f2dbad98466f0ab6d92e538f802fdb36d59cf42de3e637407b31"}
Sep 17 21:49:28.823916 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:28.823843    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf"
Sep 17 21:49:28.884665 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:28.884610    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf" podStartSLOduration=0.88458802 podStartE2EDuration="884.58802ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:49:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:49:28.88223586 +0000 UTC m=+484.700562172" watchObservedRunningTime="2026-09-17 21:49:28.88458802 +0000 UTC m=+484.702914334"
Sep 17 21:49:37.820285 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:37.820248    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f6xcz"
Sep 17 21:49:37.820792 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:37.820342    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdx6"
Sep 17 21:49:39.830975 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:39.830946    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf"
Sep 17 21:49:39.962935 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:39.962899    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdx6"]
Sep 17 21:49:39.963139 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:39.963115    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdx6" podUID="834734bd-21b7-46bf-887d-33e00a91a7f7" containerName="manager" containerID="cri-o://e0c3c1d072c68a08599f8263208abe53ea0c613880dfa52ce4d1483483658bfd" gracePeriod=10
Sep 17 21:49:40.253576 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:40.253543    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdx6"
Sep 17 21:49:40.348878 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:40.348845    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/834734bd-21b7-46bf-887d-33e00a91a7f7-extensions-socket-volume\") pod \"834734bd-21b7-46bf-887d-33e00a91a7f7\" (UID: \"834734bd-21b7-46bf-887d-33e00a91a7f7\") "
Sep 17 21:49:40.348878 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:40.348887    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flgvb\" (UniqueName: \"kubernetes.io/projected/834734bd-21b7-46bf-887d-33e00a91a7f7-kube-api-access-flgvb\") pod \"834734bd-21b7-46bf-887d-33e00a91a7f7\" (UID: \"834734bd-21b7-46bf-887d-33e00a91a7f7\") "
Sep 17 21:49:40.349298 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:40.349270    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834734bd-21b7-46bf-887d-33e00a91a7f7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "834734bd-21b7-46bf-887d-33e00a91a7f7" (UID: "834734bd-21b7-46bf-887d-33e00a91a7f7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:49:40.351089 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:40.351070    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/834734bd-21b7-46bf-887d-33e00a91a7f7-kube-api-access-flgvb" (OuterVolumeSpecName: "kube-api-access-flgvb") pod "834734bd-21b7-46bf-887d-33e00a91a7f7" (UID: "834734bd-21b7-46bf-887d-33e00a91a7f7"). InnerVolumeSpecName "kube-api-access-flgvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:49:40.449637 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:40.449598    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flgvb\" (UniqueName: \"kubernetes.io/projected/834734bd-21b7-46bf-887d-33e00a91a7f7-kube-api-access-flgvb\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:49:40.449637 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:40.449630    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/834734bd-21b7-46bf-887d-33e00a91a7f7-extensions-socket-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:49:40.878954 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:40.878921    2564 generic.go:356] "Generic (PLEG): container finished" podID="834734bd-21b7-46bf-887d-33e00a91a7f7" containerID="e0c3c1d072c68a08599f8263208abe53ea0c613880dfa52ce4d1483483658bfd" exitCode=0
Sep 17 21:49:40.879475 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:40.879003    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdx6" event={"ID":"834734bd-21b7-46bf-887d-33e00a91a7f7","Type":"ContainerDied","Data":"e0c3c1d072c68a08599f8263208abe53ea0c613880dfa52ce4d1483483658bfd"}
Sep 17 21:49:40.879475 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:40.879031    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdx6"
Sep 17 21:49:40.879475 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:40.879046    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdx6" event={"ID":"834734bd-21b7-46bf-887d-33e00a91a7f7","Type":"ContainerDied","Data":"88d1bf88a7ad473f08e99518bf99a9f23e7ee7c0b0f90069cbc586d64757535f"}
Sep 17 21:49:40.879475 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:40.879067    2564 scope.go:117] "RemoveContainer" containerID="e0c3c1d072c68a08599f8263208abe53ea0c613880dfa52ce4d1483483658bfd"
Sep 17 21:49:40.889439 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:40.889418    2564 scope.go:117] "RemoveContainer" containerID="e0c3c1d072c68a08599f8263208abe53ea0c613880dfa52ce4d1483483658bfd"
Sep 17 21:49:40.889778 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:49:40.889760    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0c3c1d072c68a08599f8263208abe53ea0c613880dfa52ce4d1483483658bfd\": container with ID starting with e0c3c1d072c68a08599f8263208abe53ea0c613880dfa52ce4d1483483658bfd not found: ID does not exist" containerID="e0c3c1d072c68a08599f8263208abe53ea0c613880dfa52ce4d1483483658bfd"
Sep 17 21:49:40.889837 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:40.889789    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c3c1d072c68a08599f8263208abe53ea0c613880dfa52ce4d1483483658bfd"} err="failed to get container status \"e0c3c1d072c68a08599f8263208abe53ea0c613880dfa52ce4d1483483658bfd\": rpc error: code = NotFound desc = could not find container \"e0c3c1d072c68a08599f8263208abe53ea0c613880dfa52ce4d1483483658bfd\": container with ID starting with e0c3c1d072c68a08599f8263208abe53ea0c613880dfa52ce4d1483483658bfd not found: ID does not exist"
Sep 17 21:49:40.956429 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:40.956391    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdx6"]
Sep 17 21:49:40.970379 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:40.970346    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdx6"]
Sep 17 21:49:42.753741 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:42.753706    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="834734bd-21b7-46bf-887d-33e00a91a7f7" path="/var/lib/kubelet/pods/834734bd-21b7-46bf-887d-33e00a91a7f7/volumes"
Sep 17 21:49:56.507298 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.507262    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"]
Sep 17 21:49:56.507745 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.507630    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="834734bd-21b7-46bf-887d-33e00a91a7f7" containerName="manager"
Sep 17 21:49:56.507745 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.507645    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="834734bd-21b7-46bf-887d-33e00a91a7f7" containerName="manager"
Sep 17 21:49:56.507745 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.507713    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="834734bd-21b7-46bf-887d-33e00a91a7f7" containerName="manager"
Sep 17 21:49:56.519408 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.519388    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.522568 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.522550    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rntf6\""
Sep 17 21:49:56.525885 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.525864    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"]
Sep 17 21:49:56.579365 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.579332    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f47358c-f7a1-4166-9a6c-b723820bc434-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.579536 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.579378    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f47358c-f7a1-4166-9a6c-b723820bc434-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.579536 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.579399    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f47358c-f7a1-4166-9a6c-b723820bc434-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.579536 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.579470    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f47358c-f7a1-4166-9a6c-b723820bc434-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.579536 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.579511    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f47358c-f7a1-4166-9a6c-b723820bc434-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.579536 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.579533    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwd2f\" (UniqueName: \"kubernetes.io/projected/7f47358c-f7a1-4166-9a6c-b723820bc434-kube-api-access-fwd2f\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.579723 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.579558    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f47358c-f7a1-4166-9a6c-b723820bc434-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.579723 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.579579    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f47358c-f7a1-4166-9a6c-b723820bc434-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.579723 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.579614    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f47358c-f7a1-4166-9a6c-b723820bc434-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.680725 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.680695    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f47358c-f7a1-4166-9a6c-b723820bc434-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.680725 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.680729    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f47358c-f7a1-4166-9a6c-b723820bc434-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.680972 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.680761    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f47358c-f7a1-4166-9a6c-b723820bc434-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.680972 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.680779    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f47358c-f7a1-4166-9a6c-b723820bc434-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.680972 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.680816    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f47358c-f7a1-4166-9a6c-b723820bc434-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.680972 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.680836    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f47358c-f7a1-4166-9a6c-b723820bc434-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.680972 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.680855    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwd2f\" (UniqueName: \"kubernetes.io/projected/7f47358c-f7a1-4166-9a6c-b723820bc434-kube-api-access-fwd2f\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.680972 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.680890    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f47358c-f7a1-4166-9a6c-b723820bc434-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.680972 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.680913    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f47358c-f7a1-4166-9a6c-b723820bc434-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.681624 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.681563    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f47358c-f7a1-4166-9a6c-b723820bc434-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.681624 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.681568    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f47358c-f7a1-4166-9a6c-b723820bc434-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.681770 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.681624    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f47358c-f7a1-4166-9a6c-b723820bc434-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.681770 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.681645    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f47358c-f7a1-4166-9a6c-b723820bc434-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.681922 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.681900    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f47358c-f7a1-4166-9a6c-b723820bc434-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.683483 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.683460    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f47358c-f7a1-4166-9a6c-b723820bc434-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.683630 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.683616    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f47358c-f7a1-4166-9a6c-b723820bc434-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.690395 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.690363    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f47358c-f7a1-4166-9a6c-b723820bc434-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.690653 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.690635    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwd2f\" (UniqueName: \"kubernetes.io/projected/7f47358c-f7a1-4166-9a6c-b723820bc434-kube-api-access-fwd2f\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2h86h\" (UID: \"7f47358c-f7a1-4166-9a6c-b723820bc434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:56.832905 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:56.832824    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:57.124460 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:57.124434    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"]
Sep 17 21:49:57.125556 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:49:57.125532    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f47358c_f7a1_4166_9a6c_b723820bc434.slice/crio-e7c09f04ee089664542ee46b4560a1840cf35986f78fb8ac77bd4adf15e86225 WatchSource:0}: Error finding container e7c09f04ee089664542ee46b4560a1840cf35986f78fb8ac77bd4adf15e86225: Status 404 returned error can't find the container with id e7c09f04ee089664542ee46b4560a1840cf35986f78fb8ac77bd4adf15e86225
Sep 17 21:49:57.128146 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:57.128108    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:49:57.128407 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:57.128383    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:49:57.128475 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:57.128439    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:49:57.964699 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:57.964667    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h" event={"ID":"7f47358c-f7a1-4166-9a6c-b723820bc434","Type":"ContainerStarted","Data":"8051553c006bc20b9a53e4036be794065c084b86edbad9403eb7eeefe9f35dbd"}
Sep 17 21:49:57.965072 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:57.964744    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h" event={"ID":"7f47358c-f7a1-4166-9a6c-b723820bc434","Type":"ContainerStarted","Data":"e7c09f04ee089664542ee46b4560a1840cf35986f78fb8ac77bd4adf15e86225"}
Sep 17 21:49:57.989281 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:57.989230    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h" podStartSLOduration=1.989214958 podStartE2EDuration="1.989214958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:49:57.988219806 +0000 UTC m=+513.806546122" watchObservedRunningTime="2026-09-17 21:49:57.989214958 +0000 UTC m=+513.807541266"
Sep 17 21:49:58.833888 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:58.833853    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:58.838431 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:58.838406    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:58.969960 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:58.969929    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:49:58.971014 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:49:58.970996    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:50:00.402689 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:00.402653    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-6j9kv"]
Sep 17 21:50:00.405973 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:00.405952    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-6j9kv"
Sep 17 21:50:00.408810 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:00.408777    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bdgv9\""
Sep 17 21:50:00.418695 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:00.418672    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-6j9kv"]
Sep 17 21:50:00.512834 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:00.512801    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpsnx\" (UniqueName: \"kubernetes.io/projected/aa3aee0d-ae5d-4a33-a08f-0e8d5071815c-kube-api-access-jpsnx\") pod \"authorino-7bcfd8f9cd-6j9kv\" (UID: \"aa3aee0d-ae5d-4a33-a08f-0e8d5071815c\") " pod="kuadrant-system/authorino-7bcfd8f9cd-6j9kv"
Sep 17 21:50:00.613569 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:00.613529    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpsnx\" (UniqueName: \"kubernetes.io/projected/aa3aee0d-ae5d-4a33-a08f-0e8d5071815c-kube-api-access-jpsnx\") pod \"authorino-7bcfd8f9cd-6j9kv\" (UID: \"aa3aee0d-ae5d-4a33-a08f-0e8d5071815c\") " pod="kuadrant-system/authorino-7bcfd8f9cd-6j9kv"
Sep 17 21:50:00.627953 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:00.627916    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpsnx\" (UniqueName: \"kubernetes.io/projected/aa3aee0d-ae5d-4a33-a08f-0e8d5071815c-kube-api-access-jpsnx\") pod \"authorino-7bcfd8f9cd-6j9kv\" (UID: \"aa3aee0d-ae5d-4a33-a08f-0e8d5071815c\") " pod="kuadrant-system/authorino-7bcfd8f9cd-6j9kv"
Sep 17 21:50:00.715002 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:00.714922    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-6j9kv"
Sep 17 21:50:00.986593 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:00.986510    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-6j9kv"]
Sep 17 21:50:00.990351 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:50:00.990324    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa3aee0d_ae5d_4a33_a08f_0e8d5071815c.slice/crio-8a82941a75c2c9ea39c4661bf1115046f9adb77ab1c852e47ccb85ea4c62a269 WatchSource:0}: Error finding container 8a82941a75c2c9ea39c4661bf1115046f9adb77ab1c852e47ccb85ea4c62a269: Status 404 returned error can't find the container with id 8a82941a75c2c9ea39c4661bf1115046f9adb77ab1c852e47ccb85ea4c62a269
Sep 17 21:50:01.985945 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:01.985913    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-6j9kv" event={"ID":"aa3aee0d-ae5d-4a33-a08f-0e8d5071815c","Type":"ContainerStarted","Data":"8a82941a75c2c9ea39c4661bf1115046f9adb77ab1c852e47ccb85ea4c62a269"}
Sep 17 21:50:02.991809 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:02.991773    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-6j9kv" event={"ID":"aa3aee0d-ae5d-4a33-a08f-0e8d5071815c","Type":"ContainerStarted","Data":"f0f1fd81f4f4ea109d80fce9fec353e30bf7eac4fe6312fae7e7e1c202c72d58"}
Sep 17 21:50:03.010896 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:03.010842    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-6j9kv" podStartSLOduration=1.189195561 podStartE2EDuration="3.010825862s" totalImagesPullingTime="1.821630301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:50:03.009339208 +0000 UTC m=+518.827665520" watchObservedRunningTime="2026-09-17 21:50:03.010825862 +0000 UTC m=+518.829152174"
Sep 17 21:50:04.276447 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:04.276414    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-6j9kv"]
Sep 17 21:50:05.000766 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:05.000728    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-6j9kv" podUID="aa3aee0d-ae5d-4a33-a08f-0e8d5071815c" containerName="authorino" containerID="cri-o://f0f1fd81f4f4ea109d80fce9fec353e30bf7eac4fe6312fae7e7e1c202c72d58" gracePeriod=30
Sep 17 21:50:05.242867 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:05.242838    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-6j9kv"
Sep 17 21:50:05.353408 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:05.353318    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpsnx\" (UniqueName: \"kubernetes.io/projected/aa3aee0d-ae5d-4a33-a08f-0e8d5071815c-kube-api-access-jpsnx\") pod \"aa3aee0d-ae5d-4a33-a08f-0e8d5071815c\" (UID: \"aa3aee0d-ae5d-4a33-a08f-0e8d5071815c\") "
Sep 17 21:50:05.355541 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:05.355510    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3aee0d-ae5d-4a33-a08f-0e8d5071815c-kube-api-access-jpsnx" (OuterVolumeSpecName: "kube-api-access-jpsnx") pod "aa3aee0d-ae5d-4a33-a08f-0e8d5071815c" (UID: "aa3aee0d-ae5d-4a33-a08f-0e8d5071815c"). InnerVolumeSpecName "kube-api-access-jpsnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:50:05.453858 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:05.453821    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpsnx\" (UniqueName: \"kubernetes.io/projected/aa3aee0d-ae5d-4a33-a08f-0e8d5071815c-kube-api-access-jpsnx\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:50:06.006568 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:06.006530    2564 generic.go:356] "Generic (PLEG): container finished" podID="aa3aee0d-ae5d-4a33-a08f-0e8d5071815c" containerID="f0f1fd81f4f4ea109d80fce9fec353e30bf7eac4fe6312fae7e7e1c202c72d58" exitCode=0
Sep 17 21:50:06.006753 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:06.006573    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-6j9kv"
Sep 17 21:50:06.006753 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:06.006615    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-6j9kv" event={"ID":"aa3aee0d-ae5d-4a33-a08f-0e8d5071815c","Type":"ContainerDied","Data":"f0f1fd81f4f4ea109d80fce9fec353e30bf7eac4fe6312fae7e7e1c202c72d58"}
Sep 17 21:50:06.006753 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:06.006650    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-6j9kv" event={"ID":"aa3aee0d-ae5d-4a33-a08f-0e8d5071815c","Type":"ContainerDied","Data":"8a82941a75c2c9ea39c4661bf1115046f9adb77ab1c852e47ccb85ea4c62a269"}
Sep 17 21:50:06.006753 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:06.006671    2564 scope.go:117] "RemoveContainer" containerID="f0f1fd81f4f4ea109d80fce9fec353e30bf7eac4fe6312fae7e7e1c202c72d58"
Sep 17 21:50:06.016103 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:06.016084    2564 scope.go:117] "RemoveContainer" containerID="f0f1fd81f4f4ea109d80fce9fec353e30bf7eac4fe6312fae7e7e1c202c72d58"
Sep 17 21:50:06.016433 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:50:06.016410    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f1fd81f4f4ea109d80fce9fec353e30bf7eac4fe6312fae7e7e1c202c72d58\": container with ID starting with f0f1fd81f4f4ea109d80fce9fec353e30bf7eac4fe6312fae7e7e1c202c72d58 not found: ID does not exist" containerID="f0f1fd81f4f4ea109d80fce9fec353e30bf7eac4fe6312fae7e7e1c202c72d58"
Sep 17 21:50:06.016539 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:06.016440    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f1fd81f4f4ea109d80fce9fec353e30bf7eac4fe6312fae7e7e1c202c72d58"} err="failed to get container status \"f0f1fd81f4f4ea109d80fce9fec353e30bf7eac4fe6312fae7e7e1c202c72d58\": rpc error: code = NotFound desc = could not find container \"f0f1fd81f4f4ea109d80fce9fec353e30bf7eac4fe6312fae7e7e1c202c72d58\": container with ID starting with f0f1fd81f4f4ea109d80fce9fec353e30bf7eac4fe6312fae7e7e1c202c72d58 not found: ID does not exist"
Sep 17 21:50:06.028006 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:06.027978    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-6j9kv"]
Sep 17 21:50:06.032187 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:06.032151    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-6j9kv"]
Sep 17 21:50:06.753489 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:06.753456    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa3aee0d-ae5d-4a33-a08f-0e8d5071815c" path="/var/lib/kubelet/pods/aa3aee0d-ae5d-4a33-a08f-0e8d5071815c/volumes"
Sep 17 21:50:38.931263 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:38.931231    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-v4s6h"]
Sep 17 21:50:38.931669 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:38.931539    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa3aee0d-ae5d-4a33-a08f-0e8d5071815c" containerName="authorino"
Sep 17 21:50:38.931669 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:38.931550    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa3aee0d-ae5d-4a33-a08f-0e8d5071815c" containerName="authorino"
Sep 17 21:50:38.931669 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:38.931611    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa3aee0d-ae5d-4a33-a08f-0e8d5071815c" containerName="authorino"
Sep 17 21:50:38.936939 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:38.936918    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-v4s6h"
Sep 17 21:50:38.939797 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:38.939780    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bdgv9\""
Sep 17 21:50:38.944439 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:38.944418    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-v4s6h"]
Sep 17 21:50:39.036414 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:39.036380    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zls4v\" (UniqueName: \"kubernetes.io/projected/75873a27-6c3d-4f31-863e-1b31bbd7d92a-kube-api-access-zls4v\") pod \"authorino-557f876f69-v4s6h\" (UID: \"75873a27-6c3d-4f31-863e-1b31bbd7d92a\") " pod="kuadrant-system/authorino-557f876f69-v4s6h"
Sep 17 21:50:39.081944 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:39.081908    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-v4s6h"]
Sep 17 21:50:39.082166 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:50:39.082147    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-zls4v], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-v4s6h" podUID="75873a27-6c3d-4f31-863e-1b31bbd7d92a"
Sep 17 21:50:39.136803 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:39.136756    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zls4v\" (UniqueName: \"kubernetes.io/projected/75873a27-6c3d-4f31-863e-1b31bbd7d92a-kube-api-access-zls4v\") pod \"authorino-557f876f69-v4s6h\" (UID: \"75873a27-6c3d-4f31-863e-1b31bbd7d92a\") " pod="kuadrant-system/authorino-557f876f69-v4s6h"
Sep 17 21:50:39.155962 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:39.155926    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-v4s6h"
Sep 17 21:50:39.155962 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:39.155949    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zls4v\" (UniqueName: \"kubernetes.io/projected/75873a27-6c3d-4f31-863e-1b31bbd7d92a-kube-api-access-zls4v\") pod \"authorino-557f876f69-v4s6h\" (UID: \"75873a27-6c3d-4f31-863e-1b31bbd7d92a\") " pod="kuadrant-system/authorino-557f876f69-v4s6h"
Sep 17 21:50:39.169546 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:39.169524    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-v4s6h"
Sep 17 21:50:39.237693 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:39.237603    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zls4v\" (UniqueName: \"kubernetes.io/projected/75873a27-6c3d-4f31-863e-1b31bbd7d92a-kube-api-access-zls4v\") pod \"75873a27-6c3d-4f31-863e-1b31bbd7d92a\" (UID: \"75873a27-6c3d-4f31-863e-1b31bbd7d92a\") "
Sep 17 21:50:39.239850 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:39.239821    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75873a27-6c3d-4f31-863e-1b31bbd7d92a-kube-api-access-zls4v" (OuterVolumeSpecName: "kube-api-access-zls4v") pod "75873a27-6c3d-4f31-863e-1b31bbd7d92a" (UID: "75873a27-6c3d-4f31-863e-1b31bbd7d92a"). InnerVolumeSpecName "kube-api-access-zls4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:50:39.272128 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:39.272097    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-646fdff977-nr5dj"]
Sep 17 21:50:39.276266 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:39.276248    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-646fdff977-nr5dj"
Sep 17 21:50:39.281424 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:39.281387    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 17 21:50:39.306760 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:39.302803    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-646fdff977-nr5dj"]
Sep 17 21:50:39.338674 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:39.338646    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/69af91e2-a1d2-478a-abfd-9240cb3b6f89-tls-cert\") pod \"authorino-646fdff977-nr5dj\" (UID: \"69af91e2-a1d2-478a-abfd-9240cb3b6f89\") " pod="kuadrant-system/authorino-646fdff977-nr5dj"
Sep 17 21:50:39.338836 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:39.338724    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9lgm\" (UniqueName: \"kubernetes.io/projected/69af91e2-a1d2-478a-abfd-9240cb3b6f89-kube-api-access-f9lgm\") pod \"authorino-646fdff977-nr5dj\" (UID: \"69af91e2-a1d2-478a-abfd-9240cb3b6f89\") " pod="kuadrant-system/authorino-646fdff977-nr5dj"
Sep 17 21:50:39.338836 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:39.338774    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zls4v\" (UniqueName: \"kubernetes.io/projected/75873a27-6c3d-4f31-863e-1b31bbd7d92a-kube-api-access-zls4v\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:50:39.439386 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:39.439348    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/69af91e2-a1d2-478a-abfd-9240cb3b6f89-tls-cert\") pod \"authorino-646fdff977-nr5dj\" (UID: \"69af91e2-a1d2-478a-abfd-9240cb3b6f89\") " pod="kuadrant-system/authorino-646fdff977-nr5dj"
Sep 17 21:50:39.439543 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:39.439426    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9lgm\" (UniqueName: \"kubernetes.io/projected/69af91e2-a1d2-478a-abfd-9240cb3b6f89-kube-api-access-f9lgm\") pod \"authorino-646fdff977-nr5dj\" (UID: \"69af91e2-a1d2-478a-abfd-9240cb3b6f89\") " pod="kuadrant-system/authorino-646fdff977-nr5dj"
Sep 17 21:50:39.442108 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:39.442079    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/69af91e2-a1d2-478a-abfd-9240cb3b6f89-tls-cert\") pod \"authorino-646fdff977-nr5dj\" (UID: \"69af91e2-a1d2-478a-abfd-9240cb3b6f89\") " pod="kuadrant-system/authorino-646fdff977-nr5dj"
Sep 17 21:50:39.459631 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:39.459587    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9lgm\" (UniqueName: \"kubernetes.io/projected/69af91e2-a1d2-478a-abfd-9240cb3b6f89-kube-api-access-f9lgm\") pod \"authorino-646fdff977-nr5dj\" (UID: \"69af91e2-a1d2-478a-abfd-9240cb3b6f89\") " pod="kuadrant-system/authorino-646fdff977-nr5dj"
Sep 17 21:50:39.584917 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:39.584826    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-646fdff977-nr5dj"
Sep 17 21:50:39.859911 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:39.859823    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-646fdff977-nr5dj"]
Sep 17 21:50:39.862023 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:50:39.861995    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69af91e2_a1d2_478a_abfd_9240cb3b6f89.slice/crio-7f5f27b92c68815e3944baf652ede92faddbbeceb2468da8a36a420622c7b41e WatchSource:0}: Error finding container 7f5f27b92c68815e3944baf652ede92faddbbeceb2468da8a36a420622c7b41e: Status 404 returned error can't find the container with id 7f5f27b92c68815e3944baf652ede92faddbbeceb2468da8a36a420622c7b41e
Sep 17 21:50:40.162220 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:40.162189    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-646fdff977-nr5dj" event={"ID":"69af91e2-a1d2-478a-abfd-9240cb3b6f89","Type":"ContainerStarted","Data":"7f5f27b92c68815e3944baf652ede92faddbbeceb2468da8a36a420622c7b41e"}
Sep 17 21:50:40.162670 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:40.162222    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-v4s6h"
Sep 17 21:50:40.193644 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:40.193611    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-v4s6h"]
Sep 17 21:50:40.197602 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:40.197579    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-v4s6h"]
Sep 17 21:50:40.754491 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:40.754458    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75873a27-6c3d-4f31-863e-1b31bbd7d92a" path="/var/lib/kubelet/pods/75873a27-6c3d-4f31-863e-1b31bbd7d92a/volumes"
Sep 17 21:50:41.168452 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:50:41.168413    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-646fdff977-nr5dj" event={"ID":"69af91e2-a1d2-478a-abfd-9240cb3b6f89","Type":"ContainerStarted","Data":"f2eae77553dd958161f266a7812b5f160f520306382b041a3b28550235029778"}
Sep 17 21:51:24.638776 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:24.638738    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdcnw_6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2/ovn-acl-logging/0.log"
Sep 17 21:51:24.646122 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:24.646101    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdcnw_6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2/ovn-acl-logging/0.log"
Sep 17 21:51:27.369475 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:27.369418    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-646fdff977-nr5dj" podStartSLOduration=47.698871438 podStartE2EDuration="48.36940201s" totalImagesPullingTime="670.530572ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:50:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:50:41.201887849 +0000 UTC m=+557.020214161" watchObservedRunningTime="2026-09-17 21:51:27.36940201 +0000 UTC m=+603.187728323"
Sep 17 21:51:27.369864 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:27.369582    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-sf4lg"]
Sep 17 21:51:27.372778 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:27.372762    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-sf4lg"
Sep 17 21:51:27.375784 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:27.375758    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-88ctk\""
Sep 17 21:51:27.376228 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:27.376212    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 17 21:51:27.399654 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:27.399612    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-sf4lg"]
Sep 17 21:51:27.446855 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:27.446819    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2b4s\" (UniqueName: \"kubernetes.io/projected/1eadf925-d59d-45cf-8d67-e7538c47abd1-kube-api-access-f2b4s\") pod \"payload-pre-processing-59fbc47bf8-sf4lg\" (UID: \"1eadf925-d59d-45cf-8d67-e7538c47abd1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-sf4lg"
Sep 17 21:51:27.446855 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:27.446867    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eadf925-d59d-45cf-8d67-e7538c47abd1-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-sf4lg\" (UID: \"1eadf925-d59d-45cf-8d67-e7538c47abd1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-sf4lg"
Sep 17 21:51:27.547664 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:27.547627    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2b4s\" (UniqueName: \"kubernetes.io/projected/1eadf925-d59d-45cf-8d67-e7538c47abd1-kube-api-access-f2b4s\") pod \"payload-pre-processing-59fbc47bf8-sf4lg\" (UID: \"1eadf925-d59d-45cf-8d67-e7538c47abd1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-sf4lg"
Sep 17 21:51:27.547860 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:27.547678    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eadf925-d59d-45cf-8d67-e7538c47abd1-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-sf4lg\" (UID: \"1eadf925-d59d-45cf-8d67-e7538c47abd1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-sf4lg"
Sep 17 21:51:27.548383 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:27.548363    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eadf925-d59d-45cf-8d67-e7538c47abd1-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-sf4lg\" (UID: \"1eadf925-d59d-45cf-8d67-e7538c47abd1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-sf4lg"
Sep 17 21:51:27.585675 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:27.585643    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2b4s\" (UniqueName: \"kubernetes.io/projected/1eadf925-d59d-45cf-8d67-e7538c47abd1-kube-api-access-f2b4s\") pod \"payload-pre-processing-59fbc47bf8-sf4lg\" (UID: \"1eadf925-d59d-45cf-8d67-e7538c47abd1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-sf4lg"
Sep 17 21:51:27.683656 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:27.683620    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-sf4lg"
Sep 17 21:51:27.982851 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:27.982768    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-sf4lg"]
Sep 17 21:51:27.991242 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:51:27.991214    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eadf925_d59d_45cf_8d67_e7538c47abd1.slice/crio-8bc5fc893c9fb44a7b41b8cfe3670236f2aa60ba6d35a7b6beaf1b222990acff WatchSource:0}: Error finding container 8bc5fc893c9fb44a7b41b8cfe3670236f2aa60ba6d35a7b6beaf1b222990acff: Status 404 returned error can't find the container with id 8bc5fc893c9fb44a7b41b8cfe3670236f2aa60ba6d35a7b6beaf1b222990acff
Sep 17 21:51:28.383985 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:28.383903    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-sf4lg" event={"ID":"1eadf925-d59d-45cf-8d67-e7538c47abd1","Type":"ContainerStarted","Data":"8bc5fc893c9fb44a7b41b8cfe3670236f2aa60ba6d35a7b6beaf1b222990acff"}
Sep 17 21:51:32.426084 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:32.426040    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-sf4lg" event={"ID":"1eadf925-d59d-45cf-8d67-e7538c47abd1","Type":"ContainerStarted","Data":"f4bf14d80d8e8d04a719a3298c27d0633382b919d5d8d202697f73dcff07dd89"}
Sep 17 21:51:32.426529 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:32.426215    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-sf4lg"
Sep 17 21:51:37.450923 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:37.450825    2564 generic.go:356] "Generic (PLEG): container finished" podID="7f47358c-f7a1-4166-9a6c-b723820bc434" containerID="8051553c006bc20b9a53e4036be794065c084b86edbad9403eb7eeefe9f35dbd" exitCode=0
Sep 17 21:51:37.450923 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:37.450904    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h" event={"ID":"7f47358c-f7a1-4166-9a6c-b723820bc434","Type":"ContainerDied","Data":"8051553c006bc20b9a53e4036be794065c084b86edbad9403eb7eeefe9f35dbd"}
Sep 17 21:51:37.451513 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:37.451432    2564 scope.go:117] "RemoveContainer" containerID="8051553c006bc20b9a53e4036be794065c084b86edbad9403eb7eeefe9f35dbd"
Sep 17 21:51:37.452458 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:37.452430    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:51:37.452519 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:37.452500    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:51:37.452557 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:37.452532    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:51:37.485316 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:37.485262    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-sf4lg" podStartSLOduration=7.120302994 podStartE2EDuration="10.48524521s" totalImagesPullingTime="3.364942216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:51:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:51:32.462999443 +0000 UTC m=+608.281325754" watchObservedRunningTime="2026-09-17 21:51:37.48524521 +0000 UTC m=+613.303571522"
Sep 17 21:51:38.457694 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:38.457661    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h" event={"ID":"7f47358c-f7a1-4166-9a6c-b723820bc434","Type":"ContainerStarted","Data":"713806acb877243120ac42fbab6b9e1796fa22ea4b25a9bc96eed639f4637a91"}
Sep 17 21:51:38.833310 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:38.833216    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:51:38.833478 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:38.833441    2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2h86h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 17 21:51:38.833526 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:38.833500    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h" podUID="7f47358c-f7a1-4166-9a6c-b723820bc434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 17 21:51:39.833041 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:39.833010    2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2h86h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 17 21:51:39.833465 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:39.833062    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h" podUID="7f47358c-f7a1-4166-9a6c-b723820bc434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 17 21:51:40.833622 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:40.833583    2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2h86h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 17 21:51:40.834032 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:40.833642    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h" podUID="7f47358c-f7a1-4166-9a6c-b723820bc434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 17 21:51:41.833296 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:41.833253    2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2h86h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 17 21:51:41.833476 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:41.833323    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h" podUID="7f47358c-f7a1-4166-9a6c-b723820bc434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 17 21:51:42.833965 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:42.833929    2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2h86h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 17 21:51:42.834385 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:42.833986    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h" podUID="7f47358c-f7a1-4166-9a6c-b723820bc434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 17 21:51:43.433950 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:43.433921    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-sf4lg"
Sep 17 21:51:43.837396 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:43.837316    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:51:43.837803 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:43.837689    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:51:43.838377 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:43.838361    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2h86h"
Sep 17 21:51:47.104895 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.104820    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"]
Sep 17 21:51:47.114001 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.113973    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"
Sep 17 21:51:47.117686 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.117637    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-v95hl\""
Sep 17 21:51:47.118809 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.118793    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 17 21:51:47.118874 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.118795    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 17 21:51:47.118874 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.118797    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 17 21:51:47.127580 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.127557    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"]
Sep 17 21:51:47.203522 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.203490    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f178fb0f-cdee-4a0a-b436-1e0cf009c55d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs\" (UID: \"f178fb0f-cdee-4a0a-b436-1e0cf009c55d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"
Sep 17 21:51:47.203522 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.203523    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f178fb0f-cdee-4a0a-b436-1e0cf009c55d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs\" (UID: \"f178fb0f-cdee-4a0a-b436-1e0cf009c55d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"
Sep 17 21:51:47.203737 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.203560    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4tp2\" (UniqueName: \"kubernetes.io/projected/f178fb0f-cdee-4a0a-b436-1e0cf009c55d-kube-api-access-r4tp2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs\" (UID: \"f178fb0f-cdee-4a0a-b436-1e0cf009c55d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"
Sep 17 21:51:47.203737 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.203631    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f178fb0f-cdee-4a0a-b436-1e0cf009c55d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs\" (UID: \"f178fb0f-cdee-4a0a-b436-1e0cf009c55d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"
Sep 17 21:51:47.203737 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.203675    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f178fb0f-cdee-4a0a-b436-1e0cf009c55d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs\" (UID: \"f178fb0f-cdee-4a0a-b436-1e0cf009c55d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"
Sep 17 21:51:47.203737 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.203712    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f178fb0f-cdee-4a0a-b436-1e0cf009c55d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs\" (UID: \"f178fb0f-cdee-4a0a-b436-1e0cf009c55d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"
Sep 17 21:51:47.304465 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.304426    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4tp2\" (UniqueName: \"kubernetes.io/projected/f178fb0f-cdee-4a0a-b436-1e0cf009c55d-kube-api-access-r4tp2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs\" (UID: \"f178fb0f-cdee-4a0a-b436-1e0cf009c55d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"
Sep 17 21:51:47.304465 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.304468    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f178fb0f-cdee-4a0a-b436-1e0cf009c55d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs\" (UID: \"f178fb0f-cdee-4a0a-b436-1e0cf009c55d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"
Sep 17 21:51:47.304683 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.304590    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f178fb0f-cdee-4a0a-b436-1e0cf009c55d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs\" (UID: \"f178fb0f-cdee-4a0a-b436-1e0cf009c55d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"
Sep 17 21:51:47.304683 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.304647    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f178fb0f-cdee-4a0a-b436-1e0cf009c55d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs\" (UID: \"f178fb0f-cdee-4a0a-b436-1e0cf009c55d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"
Sep 17 21:51:47.304774 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.304684    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f178fb0f-cdee-4a0a-b436-1e0cf009c55d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs\" (UID: \"f178fb0f-cdee-4a0a-b436-1e0cf009c55d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"
Sep 17 21:51:47.304774 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.304726    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f178fb0f-cdee-4a0a-b436-1e0cf009c55d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs\" (UID: \"f178fb0f-cdee-4a0a-b436-1e0cf009c55d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"
Sep 17 21:51:47.304876 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.304857    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f178fb0f-cdee-4a0a-b436-1e0cf009c55d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs\" (UID: \"f178fb0f-cdee-4a0a-b436-1e0cf009c55d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"
Sep 17 21:51:47.304914 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.304884    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f178fb0f-cdee-4a0a-b436-1e0cf009c55d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs\" (UID: \"f178fb0f-cdee-4a0a-b436-1e0cf009c55d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"
Sep 17 21:51:47.304990 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.304974    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f178fb0f-cdee-4a0a-b436-1e0cf009c55d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs\" (UID: \"f178fb0f-cdee-4a0a-b436-1e0cf009c55d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"
Sep 17 21:51:47.307211 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.307192    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f178fb0f-cdee-4a0a-b436-1e0cf009c55d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs\" (UID: \"f178fb0f-cdee-4a0a-b436-1e0cf009c55d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"
Sep 17 21:51:47.307324 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.307310    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f178fb0f-cdee-4a0a-b436-1e0cf009c55d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs\" (UID: \"f178fb0f-cdee-4a0a-b436-1e0cf009c55d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"
Sep 17 21:51:47.323635 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.323607    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4tp2\" (UniqueName: \"kubernetes.io/projected/f178fb0f-cdee-4a0a-b436-1e0cf009c55d-kube-api-access-r4tp2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs\" (UID: \"f178fb0f-cdee-4a0a-b436-1e0cf009c55d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"
Sep 17 21:51:47.423877 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.423848    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"
Sep 17 21:51:47.726713 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:47.726498    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"]
Sep 17 21:51:47.729675 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:51:47.729648    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf178fb0f_cdee_4a0a_b436_1e0cf009c55d.slice/crio-9abea350df858f3a430395806a54ab90e80c6bae01930c0d1a378b1104f15628 WatchSource:0}: Error finding container 9abea350df858f3a430395806a54ab90e80c6bae01930c0d1a378b1104f15628: Status 404 returned error can't find the container with id 9abea350df858f3a430395806a54ab90e80c6bae01930c0d1a378b1104f15628
Sep 17 21:51:48.509353 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:48.509320    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs" event={"ID":"f178fb0f-cdee-4a0a-b436-1e0cf009c55d","Type":"ContainerStarted","Data":"9abea350df858f3a430395806a54ab90e80c6bae01930c0d1a378b1104f15628"}
Sep 17 21:51:49.516315 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:49.516275    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs" event={"ID":"f178fb0f-cdee-4a0a-b436-1e0cf009c55d","Type":"ContainerStarted","Data":"1824c6f278b31ab63564bf4675155c65797c52d87be6b54a0495d43a45abab2e"}
Sep 17 21:51:49.566667 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:49.566611    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs" podStartSLOduration=1.350324829 podStartE2EDuration="2.566592861s" totalImagesPullingTime="1.216268032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:51:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:51:49.565676543 +0000 UTC m=+625.384002856" watchObservedRunningTime="2026-09-17 21:51:49.566592861 +0000 UTC m=+625.384919173"
Sep 17 21:51:57.424743 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:57.424700    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"
Sep 17 21:51:57.425245 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:57.424790    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"
Sep 17 21:51:57.437370 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:57.437349    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"
Sep 17 21:51:57.563006 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:51:57.562979    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs"
Sep 17 21:52:58.596405 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:58.596374    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"]
Sep 17 21:52:58.599550 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:58.599533    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"
Sep 17 21:52:58.608958 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:58.608938    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 17 21:52:58.618447 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:58.618423    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"]
Sep 17 21:52:58.732337 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:58.732303    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f46e3e76-c17b-44ee-a908-4c096eb7996f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfctf\" (UID: \"f46e3e76-c17b-44ee-a908-4c096eb7996f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"
Sep 17 21:52:58.732522 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:58.732381    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f46e3e76-c17b-44ee-a908-4c096eb7996f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfctf\" (UID: \"f46e3e76-c17b-44ee-a908-4c096eb7996f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"
Sep 17 21:52:58.732522 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:58.732468    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwp78\" (UniqueName: \"kubernetes.io/projected/f46e3e76-c17b-44ee-a908-4c096eb7996f-kube-api-access-rwp78\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfctf\" (UID: \"f46e3e76-c17b-44ee-a908-4c096eb7996f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"
Sep 17 21:52:58.732522 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:58.732506    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f46e3e76-c17b-44ee-a908-4c096eb7996f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfctf\" (UID: \"f46e3e76-c17b-44ee-a908-4c096eb7996f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"
Sep 17 21:52:58.732663 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:58.732543    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f46e3e76-c17b-44ee-a908-4c096eb7996f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfctf\" (UID: \"f46e3e76-c17b-44ee-a908-4c096eb7996f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"
Sep 17 21:52:58.732663 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:58.732580    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f46e3e76-c17b-44ee-a908-4c096eb7996f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfctf\" (UID: \"f46e3e76-c17b-44ee-a908-4c096eb7996f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"
Sep 17 21:52:58.833096 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:58.833063    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwp78\" (UniqueName: \"kubernetes.io/projected/f46e3e76-c17b-44ee-a908-4c096eb7996f-kube-api-access-rwp78\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfctf\" (UID: \"f46e3e76-c17b-44ee-a908-4c096eb7996f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"
Sep 17 21:52:58.833328 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:58.833111    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f46e3e76-c17b-44ee-a908-4c096eb7996f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfctf\" (UID: \"f46e3e76-c17b-44ee-a908-4c096eb7996f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"
Sep 17 21:52:58.833328 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:58.833140    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f46e3e76-c17b-44ee-a908-4c096eb7996f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfctf\" (UID: \"f46e3e76-c17b-44ee-a908-4c096eb7996f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"
Sep 17 21:52:58.833328 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:58.833192    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f46e3e76-c17b-44ee-a908-4c096eb7996f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfctf\" (UID: \"f46e3e76-c17b-44ee-a908-4c096eb7996f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"
Sep 17 21:52:58.833328 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:58.833221    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f46e3e76-c17b-44ee-a908-4c096eb7996f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfctf\" (UID: \"f46e3e76-c17b-44ee-a908-4c096eb7996f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"
Sep 17 21:52:58.833328 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:58.833290    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f46e3e76-c17b-44ee-a908-4c096eb7996f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfctf\" (UID: \"f46e3e76-c17b-44ee-a908-4c096eb7996f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"
Sep 17 21:52:58.833700 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:58.833660    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f46e3e76-c17b-44ee-a908-4c096eb7996f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfctf\" (UID: \"f46e3e76-c17b-44ee-a908-4c096eb7996f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"
Sep 17 21:52:58.833829 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:58.833713    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f46e3e76-c17b-44ee-a908-4c096eb7996f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfctf\" (UID: \"f46e3e76-c17b-44ee-a908-4c096eb7996f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"
Sep 17 21:52:58.833829 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:58.833747    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f46e3e76-c17b-44ee-a908-4c096eb7996f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfctf\" (UID: \"f46e3e76-c17b-44ee-a908-4c096eb7996f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"
Sep 17 21:52:58.835680 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:58.835654    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f46e3e76-c17b-44ee-a908-4c096eb7996f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfctf\" (UID: \"f46e3e76-c17b-44ee-a908-4c096eb7996f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"
Sep 17 21:52:58.835820 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:58.835802    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f46e3e76-c17b-44ee-a908-4c096eb7996f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfctf\" (UID: \"f46e3e76-c17b-44ee-a908-4c096eb7996f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"
Sep 17 21:52:58.842618 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:58.842582    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwp78\" (UniqueName: \"kubernetes.io/projected/f46e3e76-c17b-44ee-a908-4c096eb7996f-kube-api-access-rwp78\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfctf\" (UID: \"f46e3e76-c17b-44ee-a908-4c096eb7996f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"
Sep 17 21:52:58.910162 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:58.910129    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"
Sep 17 21:52:59.189478 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:59.189444    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"]
Sep 17 21:52:59.191629 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:52:59.191594    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46e3e76_c17b_44ee_a908_4c096eb7996f.slice/crio-79af91114ab4d2a8eb94d702b324d6128b166a33f925d06fe65312810b0c4b80 WatchSource:0}: Error finding container 79af91114ab4d2a8eb94d702b324d6128b166a33f925d06fe65312810b0c4b80: Status 404 returned error can't find the container with id 79af91114ab4d2a8eb94d702b324d6128b166a33f925d06fe65312810b0c4b80
Sep 17 21:52:59.193442 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:59.193419    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:52:59.836389 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:59.836349    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf" event={"ID":"f46e3e76-c17b-44ee-a908-4c096eb7996f","Type":"ContainerStarted","Data":"85da20ac65a0211414491e934f3df7057816271826b3282bbb3723d50d717172"}
Sep 17 21:52:59.836765 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:59.836416    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf" event={"ID":"f46e3e76-c17b-44ee-a908-4c096eb7996f","Type":"ContainerStarted","Data":"79af91114ab4d2a8eb94d702b324d6128b166a33f925d06fe65312810b0c4b80"}
Sep 17 21:52:59.858648 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:52:59.858599    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf" podStartSLOduration=1.682714561 podStartE2EDuration="1.858582914s" totalImagesPullingTime="175.868353ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:52:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:52:59.857555872 +0000 UTC m=+695.675882185" watchObservedRunningTime="2026-09-17 21:52:59.858582914 +0000 UTC m=+695.676909251"
Sep 17 21:53:08.910717 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:08.910677    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"
Sep 17 21:53:08.911248 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:08.910976    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"
Sep 17 21:53:08.923756 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:08.923731    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"
Sep 17 21:53:09.901834 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:09.901797    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfctf"
Sep 17 21:53:24.863225 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:24.863195    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"]
Sep 17 21:53:24.866869 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:24.866849    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"
Sep 17 21:53:24.869623 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:24.869598    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 17 21:53:24.879916 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:24.879891    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"]
Sep 17 21:53:24.959056 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:24.959028    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af2e15df-b573-4f4c-8d00-bd04763918d6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sll6k\" (UID: \"af2e15df-b573-4f4c-8d00-bd04763918d6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"
Sep 17 21:53:24.959302 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:24.959097    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af2e15df-b573-4f4c-8d00-bd04763918d6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sll6k\" (UID: \"af2e15df-b573-4f4c-8d00-bd04763918d6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"
Sep 17 21:53:24.959302 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:24.959127    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xw6s\" (UniqueName: \"kubernetes.io/projected/af2e15df-b573-4f4c-8d00-bd04763918d6-kube-api-access-9xw6s\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sll6k\" (UID: \"af2e15df-b573-4f4c-8d00-bd04763918d6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"
Sep 17 21:53:24.959302 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:24.959164    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af2e15df-b573-4f4c-8d00-bd04763918d6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sll6k\" (UID: \"af2e15df-b573-4f4c-8d00-bd04763918d6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"
Sep 17 21:53:24.959302 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:24.959210    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af2e15df-b573-4f4c-8d00-bd04763918d6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sll6k\" (UID: \"af2e15df-b573-4f4c-8d00-bd04763918d6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"
Sep 17 21:53:24.959302 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:24.959252    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af2e15df-b573-4f4c-8d00-bd04763918d6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sll6k\" (UID: \"af2e15df-b573-4f4c-8d00-bd04763918d6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"
Sep 17 21:53:25.060540 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:25.060507    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af2e15df-b573-4f4c-8d00-bd04763918d6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sll6k\" (UID: \"af2e15df-b573-4f4c-8d00-bd04763918d6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"
Sep 17 21:53:25.060713 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:25.060578    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af2e15df-b573-4f4c-8d00-bd04763918d6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sll6k\" (UID: \"af2e15df-b573-4f4c-8d00-bd04763918d6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"
Sep 17 21:53:25.060713 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:25.060609    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xw6s\" (UniqueName: \"kubernetes.io/projected/af2e15df-b573-4f4c-8d00-bd04763918d6-kube-api-access-9xw6s\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sll6k\" (UID: \"af2e15df-b573-4f4c-8d00-bd04763918d6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"
Sep 17 21:53:25.060713 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:25.060648    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af2e15df-b573-4f4c-8d00-bd04763918d6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sll6k\" (UID: \"af2e15df-b573-4f4c-8d00-bd04763918d6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"
Sep 17 21:53:25.060713 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:25.060673    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af2e15df-b573-4f4c-8d00-bd04763918d6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sll6k\" (UID: \"af2e15df-b573-4f4c-8d00-bd04763918d6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"
Sep 17 21:53:25.060902 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:25.060717    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af2e15df-b573-4f4c-8d00-bd04763918d6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sll6k\" (UID: \"af2e15df-b573-4f4c-8d00-bd04763918d6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"
Sep 17 21:53:25.061086 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:25.061052    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af2e15df-b573-4f4c-8d00-bd04763918d6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sll6k\" (UID: \"af2e15df-b573-4f4c-8d00-bd04763918d6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"
Sep 17 21:53:25.061086 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:25.061063    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af2e15df-b573-4f4c-8d00-bd04763918d6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sll6k\" (UID: \"af2e15df-b573-4f4c-8d00-bd04763918d6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"
Sep 17 21:53:25.061262 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:25.061160    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af2e15df-b573-4f4c-8d00-bd04763918d6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sll6k\" (UID: \"af2e15df-b573-4f4c-8d00-bd04763918d6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"
Sep 17 21:53:25.063182 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:25.063137    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af2e15df-b573-4f4c-8d00-bd04763918d6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sll6k\" (UID: \"af2e15df-b573-4f4c-8d00-bd04763918d6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"
Sep 17 21:53:25.063328 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:25.063313    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af2e15df-b573-4f4c-8d00-bd04763918d6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sll6k\" (UID: \"af2e15df-b573-4f4c-8d00-bd04763918d6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"
Sep 17 21:53:25.073235 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:25.073209    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xw6s\" (UniqueName: \"kubernetes.io/projected/af2e15df-b573-4f4c-8d00-bd04763918d6-kube-api-access-9xw6s\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sll6k\" (UID: \"af2e15df-b573-4f4c-8d00-bd04763918d6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"
Sep 17 21:53:25.180571 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:25.180537    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"
Sep 17 21:53:25.455346 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:25.455266    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"]
Sep 17 21:53:25.457563 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:53:25.457538    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2e15df_b573_4f4c_8d00_bd04763918d6.slice/crio-09e589f6b61ddf9d983f8952e205f9704311d35e221e984236cd7d797b843fbe WatchSource:0}: Error finding container 09e589f6b61ddf9d983f8952e205f9704311d35e221e984236cd7d797b843fbe: Status 404 returned error can't find the container with id 09e589f6b61ddf9d983f8952e205f9704311d35e221e984236cd7d797b843fbe
Sep 17 21:53:25.966943 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:25.966900    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k" event={"ID":"af2e15df-b573-4f4c-8d00-bd04763918d6","Type":"ContainerStarted","Data":"5cf422d1a11db1120a72df0d3e349366f421cf4c5a826468151cdb5fb16dd15e"}
Sep 17 21:53:25.967414 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:25.966967    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k" event={"ID":"af2e15df-b573-4f4c-8d00-bd04763918d6","Type":"ContainerStarted","Data":"09e589f6b61ddf9d983f8952e205f9704311d35e221e984236cd7d797b843fbe"}
Sep 17 21:53:25.987006 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:25.986888    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k" podStartSLOduration=1.805853444 podStartE2EDuration="1.986868191s" totalImagesPullingTime="181.014747ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:53:25.986719296 +0000 UTC m=+721.805045610" watchObservedRunningTime="2026-09-17 21:53:25.986868191 +0000 UTC m=+721.805194504"
Sep 17 21:53:35.180824 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:35.180786    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"
Sep 17 21:53:35.181340 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:35.180844    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"
Sep 17 21:53:35.193644 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:35.193620    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"
Sep 17 21:53:36.025827 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:53:36.025799    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sll6k"
Sep 17 21:55:13.850451 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:13.850411    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-646fdff977-nr5dj"]
Sep 17 21:55:13.850924 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:13.850687    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-646fdff977-nr5dj" podUID="69af91e2-a1d2-478a-abfd-9240cb3b6f89" containerName="authorino" containerID="cri-o://f2eae77553dd958161f266a7812b5f160f520306382b041a3b28550235029778" gracePeriod=30
Sep 17 21:55:14.105670 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:14.105609    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-646fdff977-nr5dj"
Sep 17 21:55:14.208458 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:14.208424    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9lgm\" (UniqueName: \"kubernetes.io/projected/69af91e2-a1d2-478a-abfd-9240cb3b6f89-kube-api-access-f9lgm\") pod \"69af91e2-a1d2-478a-abfd-9240cb3b6f89\" (UID: \"69af91e2-a1d2-478a-abfd-9240cb3b6f89\") "
Sep 17 21:55:14.208647 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:14.208522    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/69af91e2-a1d2-478a-abfd-9240cb3b6f89-tls-cert\") pod \"69af91e2-a1d2-478a-abfd-9240cb3b6f89\" (UID: \"69af91e2-a1d2-478a-abfd-9240cb3b6f89\") "
Sep 17 21:55:14.210731 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:14.210701    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69af91e2-a1d2-478a-abfd-9240cb3b6f89-kube-api-access-f9lgm" (OuterVolumeSpecName: "kube-api-access-f9lgm") pod "69af91e2-a1d2-478a-abfd-9240cb3b6f89" (UID: "69af91e2-a1d2-478a-abfd-9240cb3b6f89"). InnerVolumeSpecName "kube-api-access-f9lgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:55:14.219132 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:14.219106    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69af91e2-a1d2-478a-abfd-9240cb3b6f89-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "69af91e2-a1d2-478a-abfd-9240cb3b6f89" (UID: "69af91e2-a1d2-478a-abfd-9240cb3b6f89"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:55:14.310021 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:14.309989    2564 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/69af91e2-a1d2-478a-abfd-9240cb3b6f89-tls-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:55:14.310021 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:14.310019    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9lgm\" (UniqueName: \"kubernetes.io/projected/69af91e2-a1d2-478a-abfd-9240cb3b6f89-kube-api-access-f9lgm\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:55:14.474714 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:14.474681    2564 generic.go:356] "Generic (PLEG): container finished" podID="69af91e2-a1d2-478a-abfd-9240cb3b6f89" containerID="f2eae77553dd958161f266a7812b5f160f520306382b041a3b28550235029778" exitCode=0
Sep 17 21:55:14.474891 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:14.474724    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-646fdff977-nr5dj"
Sep 17 21:55:14.474891 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:14.474757    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-646fdff977-nr5dj" event={"ID":"69af91e2-a1d2-478a-abfd-9240cb3b6f89","Type":"ContainerDied","Data":"f2eae77553dd958161f266a7812b5f160f520306382b041a3b28550235029778"}
Sep 17 21:55:14.474891 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:14.474801    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-646fdff977-nr5dj" event={"ID":"69af91e2-a1d2-478a-abfd-9240cb3b6f89","Type":"ContainerDied","Data":"7f5f27b92c68815e3944baf652ede92faddbbeceb2468da8a36a420622c7b41e"}
Sep 17 21:55:14.474891 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:14.474826    2564 scope.go:117] "RemoveContainer" containerID="f2eae77553dd958161f266a7812b5f160f520306382b041a3b28550235029778"
Sep 17 21:55:14.484427 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:14.484408    2564 scope.go:117] "RemoveContainer" containerID="f2eae77553dd958161f266a7812b5f160f520306382b041a3b28550235029778"
Sep 17 21:55:14.484710 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:55:14.484694    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2eae77553dd958161f266a7812b5f160f520306382b041a3b28550235029778\": container with ID starting with f2eae77553dd958161f266a7812b5f160f520306382b041a3b28550235029778 not found: ID does not exist" containerID="f2eae77553dd958161f266a7812b5f160f520306382b041a3b28550235029778"
Sep 17 21:55:14.484752 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:14.484720    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2eae77553dd958161f266a7812b5f160f520306382b041a3b28550235029778"} err="failed to get container status \"f2eae77553dd958161f266a7812b5f160f520306382b041a3b28550235029778\": rpc error: code = NotFound desc = could not find container \"f2eae77553dd958161f266a7812b5f160f520306382b041a3b28550235029778\": container with ID starting with f2eae77553dd958161f266a7812b5f160f520306382b041a3b28550235029778 not found: ID does not exist"
Sep 17 21:55:14.525506 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:14.525468    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-646fdff977-nr5dj"]
Sep 17 21:55:14.558089 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:14.558058    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-646fdff977-nr5dj"]
Sep 17 21:55:14.753842 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:14.753758    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69af91e2-a1d2-478a-abfd-9240cb3b6f89" path="/var/lib/kubelet/pods/69af91e2-a1d2-478a-abfd-9240cb3b6f89/volumes"
Sep 17 21:55:32.008858 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.008822    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"]
Sep 17 21:55:32.009245 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.009219    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69af91e2-a1d2-478a-abfd-9240cb3b6f89" containerName="authorino"
Sep 17 21:55:32.009245 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.009233    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="69af91e2-a1d2-478a-abfd-9240cb3b6f89" containerName="authorino"
Sep 17 21:55:32.009317 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.009295    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="69af91e2-a1d2-478a-abfd-9240cb3b6f89" containerName="authorino"
Sep 17 21:55:32.013181 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.013151    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.016651 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.016625    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f996ae-openshift-default-dockercfg-45z85\""
Sep 17 21:55:32.049852 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.049819    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"]
Sep 17 21:55:32.061320 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.061273    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-workload-certs\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.061486 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.061329    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-istio-envoy\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.061486 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.061373    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-workload-socket\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.061486 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.061404    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9febd7d-9eea-4806-8103-09aa77022f2f-istio-token\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.061486 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.061453    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9febd7d-9eea-4806-8103-09aa77022f2f-istio-podinfo\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.061653 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.061488    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9febd7d-9eea-4806-8103-09aa77022f2f-istiod-ca-cert\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.061653 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.061589    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-istio-data\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.061653 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.061631    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdggw\" (UniqueName: \"kubernetes.io/projected/e9febd7d-9eea-4806-8103-09aa77022f2f-kube-api-access-rdggw\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.061771 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.061706    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-credential-socket\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.162540 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.162505    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9febd7d-9eea-4806-8103-09aa77022f2f-istiod-ca-cert\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.162720 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.162556    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-istio-data\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.162720 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.162590    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdggw\" (UniqueName: \"kubernetes.io/projected/e9febd7d-9eea-4806-8103-09aa77022f2f-kube-api-access-rdggw\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.162720 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.162641    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-credential-socket\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.162720 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.162700    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-workload-certs\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.163008 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.162988    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-istio-envoy\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.163111 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.163035    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-workload-socket\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.163111 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.163048    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-credential-socket\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.163111 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.163074    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9febd7d-9eea-4806-8103-09aa77022f2f-istio-token\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.163111 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.162988    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-workload-certs\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.163443 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.163105    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-istio-data\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.163443 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.163125    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9febd7d-9eea-4806-8103-09aa77022f2f-istio-podinfo\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.163443 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.163352    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9febd7d-9eea-4806-8103-09aa77022f2f-istiod-ca-cert\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.163612 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.163472    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-workload-socket\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.165252 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.165226    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-istio-envoy\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.165551 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.165534    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9febd7d-9eea-4806-8103-09aa77022f2f-istio-podinfo\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.174077 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.174050    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"]
Sep 17 21:55:32.178936 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.178916    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.185693 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.185673    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6a6c05-openshift-default-dockercfg-gxhdb\""
Sep 17 21:55:32.211088 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.211050    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"]
Sep 17 21:55:32.211693 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.211670    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9febd7d-9eea-4806-8103-09aa77022f2f-istio-token\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.213098 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.213077    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdggw\" (UniqueName: \"kubernetes.io/projected/e9febd7d-9eea-4806-8103-09aa77022f2f-kube-api-access-rdggw\") pod \"e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") " pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.264279 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.264157    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-workload-certs\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.264483 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.264282    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-istio-data\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.264483 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.264357    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b112866e-0590-411f-bc6f-8f5fe79ed928-istio-podinfo\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.264483 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.264454    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-workload-socket\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.264669 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.264487    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b112866e-0590-411f-bc6f-8f5fe79ed928-istiod-ca-cert\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.264669 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.264514    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9qrt\" (UniqueName: \"kubernetes.io/projected/b112866e-0590-411f-bc6f-8f5fe79ed928-kube-api-access-n9qrt\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.264669 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.264555    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-credential-socket\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.264669 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.264604    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b112866e-0590-411f-bc6f-8f5fe79ed928-istio-token\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.264669 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.264640    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-istio-envoy\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.324844 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.324811    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:32.366439 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.365682    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b112866e-0590-411f-bc6f-8f5fe79ed928-istio-podinfo\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.366439 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.365744    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-workload-socket\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.366439 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.365772    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b112866e-0590-411f-bc6f-8f5fe79ed928-istiod-ca-cert\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.366439 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.365796    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9qrt\" (UniqueName: \"kubernetes.io/projected/b112866e-0590-411f-bc6f-8f5fe79ed928-kube-api-access-n9qrt\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.366439 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.365835    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-credential-socket\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.366439 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.365874    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b112866e-0590-411f-bc6f-8f5fe79ed928-istio-token\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.366439 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.365906    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-istio-envoy\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.366439 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.365967    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-workload-certs\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.366439 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.365997    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-istio-data\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.367712 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.367573    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-credential-socket\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.367712 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.367577    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-workload-socket\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.367712 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.367675    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-workload-certs\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.368239 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.368157    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-istio-data\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.369677 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.368698    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b112866e-0590-411f-bc6f-8f5fe79ed928-istiod-ca-cert\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.375342 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.373346    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-istio-envoy\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.379680 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.379653    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b112866e-0590-411f-bc6f-8f5fe79ed928-istio-podinfo\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.415839 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.415803    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b112866e-0590-411f-bc6f-8f5fe79ed928-istio-token\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.445544 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.445512    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9qrt\" (UniqueName: \"kubernetes.io/projected/b112866e-0590-411f-bc6f-8f5fe79ed928-kube-api-access-n9qrt\") pod \"e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") " pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.490532 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.490498    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:32.649515 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:55:32.649486    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9febd7d_9eea_4806_8103_09aa77022f2f.slice/crio-c1e92a1bb98a4d6969f08fa27b9876976681012dcddd7906df7f8d3e251478cf WatchSource:0}: Error finding container c1e92a1bb98a4d6969f08fa27b9876976681012dcddd7906df7f8d3e251478cf: Status 404 returned error can't find the container with id c1e92a1bb98a4d6969f08fa27b9876976681012dcddd7906df7f8d3e251478cf
Sep 17 21:55:32.651764 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.651726    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:55:32.651860 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.651803    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:55:32.651860 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.651833    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:55:32.656767 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.656743    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"]
Sep 17 21:55:32.785247 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.785218    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"]
Sep 17 21:55:32.787358 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:55:32.787328    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb112866e_0590_411f_bc6f_8f5fe79ed928.slice/crio-6fe94a9b205cf7d6b8e6bbf65e7b158eaff4ebb6740837ca7d61935d7e02e75f WatchSource:0}: Error finding container 6fe94a9b205cf7d6b8e6bbf65e7b158eaff4ebb6740837ca7d61935d7e02e75f: Status 404 returned error can't find the container with id 6fe94a9b205cf7d6b8e6bbf65e7b158eaff4ebb6740837ca7d61935d7e02e75f
Sep 17 21:55:32.790745 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.790706    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:55:32.790860 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.790776    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:55:32.790860 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:32.790807    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:55:33.566333 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:33.566291    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv" event={"ID":"b112866e-0590-411f-bc6f-8f5fe79ed928","Type":"ContainerStarted","Data":"ec4d54500c65c479b8a1b3bfc5abc4ba8fd061943b3453ee25bd28e8f74e7449"}
Sep 17 21:55:33.566830 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:33.566351    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv" event={"ID":"b112866e-0590-411f-bc6f-8f5fe79ed928","Type":"ContainerStarted","Data":"6fe94a9b205cf7d6b8e6bbf65e7b158eaff4ebb6740837ca7d61935d7e02e75f"}
Sep 17 21:55:33.567966 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:33.567940    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz" event={"ID":"e9febd7d-9eea-4806-8103-09aa77022f2f","Type":"ContainerStarted","Data":"34035e9125a4900c90e3f2cf227155f8e54ecfd6ec2446a771f3e080e757abf3"}
Sep 17 21:55:33.568073 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:33.567989    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz" event={"ID":"e9febd7d-9eea-4806-8103-09aa77022f2f","Type":"ContainerStarted","Data":"c1e92a1bb98a4d6969f08fa27b9876976681012dcddd7906df7f8d3e251478cf"}
Sep 17 21:55:33.611772 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:33.611722    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv" podStartSLOduration=1.6117050929999999 podStartE2EDuration="1.611705093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:55:33.611376801 +0000 UTC m=+849.429703112" watchObservedRunningTime="2026-09-17 21:55:33.611705093 +0000 UTC m=+849.430031406"
Sep 17 21:55:33.650678 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:33.650623    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz" podStartSLOduration=2.650606887 podStartE2EDuration="2.650606887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:55:33.649742744 +0000 UTC m=+849.468069056" watchObservedRunningTime="2026-09-17 21:55:33.650606887 +0000 UTC m=+849.468933198"
Sep 17 21:55:34.325712 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:34.325670    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:34.330833 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:34.330803    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:34.490841 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:34.490805    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:34.497268 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:34.497233    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:34.573412 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:34.573371    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:34.573951 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:34.573441    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:34.574254 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:34.574233    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:34.574412 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:34.574396    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:38.378394 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:38.378352    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t"]
Sep 17 21:55:38.382014 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:38.381991    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t"
Sep 17 21:55:38.392052 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:38.392026    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f996ae-dockercfg-ssnbg\""
Sep 17 21:55:38.393259 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:38.393241    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f996ae\""
Sep 17 21:55:38.403284 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:38.403259    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t"]
Sep 17 21:55:38.520968 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:38.520928    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx87h\" (UniqueName: \"kubernetes.io/projected/6afd63a8-2cb1-4ace-9483-a70b2298929f-kube-api-access-hx87h\") pod \"payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t\" (UID: \"6afd63a8-2cb1-4ace-9483-a70b2298929f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t"
Sep 17 21:55:38.521147 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:38.520982    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afd63a8-2cb1-4ace-9483-a70b2298929f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t\" (UID: \"6afd63a8-2cb1-4ace-9483-a70b2298929f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t"
Sep 17 21:55:38.621701 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:38.621663    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx87h\" (UniqueName: \"kubernetes.io/projected/6afd63a8-2cb1-4ace-9483-a70b2298929f-kube-api-access-hx87h\") pod \"payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t\" (UID: \"6afd63a8-2cb1-4ace-9483-a70b2298929f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t"
Sep 17 21:55:38.621873 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:38.621712    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afd63a8-2cb1-4ace-9483-a70b2298929f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t\" (UID: \"6afd63a8-2cb1-4ace-9483-a70b2298929f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t"
Sep 17 21:55:38.622461 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:38.622442    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afd63a8-2cb1-4ace-9483-a70b2298929f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t\" (UID: \"6afd63a8-2cb1-4ace-9483-a70b2298929f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t"
Sep 17 21:55:38.639346 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:38.639283    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx87h\" (UniqueName: \"kubernetes.io/projected/6afd63a8-2cb1-4ace-9483-a70b2298929f-kube-api-access-hx87h\") pod \"payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t\" (UID: \"6afd63a8-2cb1-4ace-9483-a70b2298929f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t"
Sep 17 21:55:38.694348 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:38.694315    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t"
Sep 17 21:55:39.007342 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:39.007311    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t"]
Sep 17 21:55:39.009668 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:55:39.009636    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6afd63a8_2cb1_4ace_9483_a70b2298929f.slice/crio-34df1260251a8a80e3154f33e48b2280c7b2d5ac6c9e3a09871dea648f90c17e WatchSource:0}: Error finding container 34df1260251a8a80e3154f33e48b2280c7b2d5ac6c9e3a09871dea648f90c17e: Status 404 returned error can't find the container with id 34df1260251a8a80e3154f33e48b2280c7b2d5ac6c9e3a09871dea648f90c17e
Sep 17 21:55:39.601014 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:39.600966    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t" event={"ID":"6afd63a8-2cb1-4ace-9483-a70b2298929f","Type":"ContainerStarted","Data":"41296f576876ad992e2c6bf8b95a931b72c9b269d7e4d20533eb1797698a5e8d"}
Sep 17 21:55:39.601535 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:39.601033    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t" event={"ID":"6afd63a8-2cb1-4ace-9483-a70b2298929f","Type":"ContainerStarted","Data":"34df1260251a8a80e3154f33e48b2280c7b2d5ac6c9e3a09871dea648f90c17e"}
Sep 17 21:55:39.601535 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:39.601076    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t"
Sep 17 21:55:40.786002 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:40.785950    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t" podStartSLOduration=2.785933257 podStartE2EDuration="2.785933257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:55:39.657044903 +0000 UTC m=+855.475371215" watchObservedRunningTime="2026-09-17 21:55:40.785933257 +0000 UTC m=+856.604259568"
Sep 17 21:55:40.786873 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:40.786850    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"]
Sep 17 21:55:40.790865 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:40.790844    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:40.794145 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:40.794124    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-9e9409-openshift-default-dockercfg-6nzdj\""
Sep 17 21:55:40.815254 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:40.815218    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"]
Sep 17 21:55:40.946967 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:40.946927    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-workload-socket\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:40.946967 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:40.946971    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-istio-data\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:40.947290 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:40.947003    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-workload-certs\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:40.947290 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:40.947040    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d06b164-d263-4f1b-a9ab-c060136cd754-istio-podinfo\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:40.947290 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:40.947060    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d06b164-d263-4f1b-a9ab-c060136cd754-istiod-ca-cert\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:40.947290 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:40.947122    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-credential-socket\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:40.947290 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:40.947225    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqk4j\" (UniqueName: \"kubernetes.io/projected/4d06b164-d263-4f1b-a9ab-c060136cd754-kube-api-access-mqk4j\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:40.947290 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:40.947262    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-istio-envoy\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:40.947552 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:40.947337    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d06b164-d263-4f1b-a9ab-c060136cd754-istio-token\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:41.048860 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.048769    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-istio-envoy\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:41.048860 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.048816    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d06b164-d263-4f1b-a9ab-c060136cd754-istio-token\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:41.048860 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.048850    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-workload-socket\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:41.049141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.048867    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-istio-data\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:41.049141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.048888    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-workload-certs\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:41.049141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.048913    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d06b164-d263-4f1b-a9ab-c060136cd754-istio-podinfo\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:41.049141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.048938    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d06b164-d263-4f1b-a9ab-c060136cd754-istiod-ca-cert\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:41.049141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.048974    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-credential-socket\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:41.049141 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.049089    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqk4j\" (UniqueName: \"kubernetes.io/projected/4d06b164-d263-4f1b-a9ab-c060136cd754-kube-api-access-mqk4j\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:41.049484 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.049370    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-workload-socket\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:41.049484 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.049414    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-workload-certs\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:41.049596 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.049523    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-credential-socket\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:41.049653 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.049605    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-istio-data\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:41.049832 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.049806    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d06b164-d263-4f1b-a9ab-c060136cd754-istiod-ca-cert\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:41.051501 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.051480    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-istio-envoy\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:41.051589 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.051570    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d06b164-d263-4f1b-a9ab-c060136cd754-istio-podinfo\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:41.157312 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.157283    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d06b164-d263-4f1b-a9ab-c060136cd754-istio-token\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:41.165449 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.165422    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqk4j\" (UniqueName: \"kubernetes.io/projected/4d06b164-d263-4f1b-a9ab-c060136cd754-kube-api-access-mqk4j\") pod \"e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") " pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:41.402368 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.402331    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:41.750260 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.750164    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"]
Sep 17 21:55:41.753593 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:55:41.753570    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d06b164_d263_4f1b_a9ab_c060136cd754.slice/crio-6eef10a56bc4ca03abf71637b386dfb184ad64c120ce1529ee03dfc130436e7c WatchSource:0}: Error finding container 6eef10a56bc4ca03abf71637b386dfb184ad64c120ce1529ee03dfc130436e7c: Status 404 returned error can't find the container with id 6eef10a56bc4ca03abf71637b386dfb184ad64c120ce1529ee03dfc130436e7c
Sep 17 21:55:41.755961 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.755931    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:55:41.756034 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.755992    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:55:41.756034 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.756023    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:55:41.892114 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.892071    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l"]
Sep 17 21:55:41.896260 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.896235    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l"
Sep 17 21:55:41.901772 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.901747    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6a6c05\""
Sep 17 21:55:41.901914 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.901837    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6a6c05-dockercfg-q6c4z\""
Sep 17 21:55:41.950793 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:41.950755    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l"]
Sep 17 21:55:42.059770 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.059675    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pccbj\" (UniqueName: \"kubernetes.io/projected/95f8375e-116e-478d-b3d6-f634936aa0f0-kube-api-access-pccbj\") pod \"payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l\" (UID: \"95f8375e-116e-478d-b3d6-f634936aa0f0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l"
Sep 17 21:55:42.059924 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.059809    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95f8375e-116e-478d-b3d6-f634936aa0f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l\" (UID: \"95f8375e-116e-478d-b3d6-f634936aa0f0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l"
Sep 17 21:55:42.161318 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.161267    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pccbj\" (UniqueName: \"kubernetes.io/projected/95f8375e-116e-478d-b3d6-f634936aa0f0-kube-api-access-pccbj\") pod \"payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l\" (UID: \"95f8375e-116e-478d-b3d6-f634936aa0f0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l"
Sep 17 21:55:42.161468 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.161358    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95f8375e-116e-478d-b3d6-f634936aa0f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l\" (UID: \"95f8375e-116e-478d-b3d6-f634936aa0f0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l"
Sep 17 21:55:42.162259 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.162238    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95f8375e-116e-478d-b3d6-f634936aa0f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l\" (UID: \"95f8375e-116e-478d-b3d6-f634936aa0f0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l"
Sep 17 21:55:42.180941 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.180905    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pccbj\" (UniqueName: \"kubernetes.io/projected/95f8375e-116e-478d-b3d6-f634936aa0f0-kube-api-access-pccbj\") pod \"payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l\" (UID: \"95f8375e-116e-478d-b3d6-f634936aa0f0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l"
Sep 17 21:55:42.207876 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.207846    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l"
Sep 17 21:55:42.343085 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.342995    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf"]
Sep 17 21:55:42.347563 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.347540    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf"
Sep 17 21:55:42.378576 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.378540    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf"]
Sep 17 21:55:42.465265 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.465225    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcfb0f22-9ccc-4012-a92d-7875c900d3da-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf\" (UID: \"dcfb0f22-9ccc-4012-a92d-7875c900d3da\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf"
Sep 17 21:55:42.465440 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.465285    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pd6h\" (UniqueName: \"kubernetes.io/projected/dcfb0f22-9ccc-4012-a92d-7875c900d3da-kube-api-access-8pd6h\") pod \"payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf\" (UID: \"dcfb0f22-9ccc-4012-a92d-7875c900d3da\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf"
Sep 17 21:55:42.543385 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.543358    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l"]
Sep 17 21:55:42.545215 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:55:42.545166    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f8375e_116e_478d_b3d6_f634936aa0f0.slice/crio-235f8a67fabae9a87021b72940c85b1e54ff84d46fd9c8b2f888ed7faf6daf64 WatchSource:0}: Error finding container 235f8a67fabae9a87021b72940c85b1e54ff84d46fd9c8b2f888ed7faf6daf64: Status 404 returned error can't find the container with id 235f8a67fabae9a87021b72940c85b1e54ff84d46fd9c8b2f888ed7faf6daf64
Sep 17 21:55:42.566446 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.566420    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcfb0f22-9ccc-4012-a92d-7875c900d3da-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf\" (UID: \"dcfb0f22-9ccc-4012-a92d-7875c900d3da\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf"
Sep 17 21:55:42.566547 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.566498    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pd6h\" (UniqueName: \"kubernetes.io/projected/dcfb0f22-9ccc-4012-a92d-7875c900d3da-kube-api-access-8pd6h\") pod \"payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf\" (UID: \"dcfb0f22-9ccc-4012-a92d-7875c900d3da\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf"
Sep 17 21:55:42.567283 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.567261    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcfb0f22-9ccc-4012-a92d-7875c900d3da-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf\" (UID: \"dcfb0f22-9ccc-4012-a92d-7875c900d3da\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf"
Sep 17 21:55:42.601372 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.601215    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pd6h\" (UniqueName: \"kubernetes.io/projected/dcfb0f22-9ccc-4012-a92d-7875c900d3da-kube-api-access-8pd6h\") pod \"payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf\" (UID: \"dcfb0f22-9ccc-4012-a92d-7875c900d3da\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf"
Sep 17 21:55:42.620537 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.620495    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l" event={"ID":"95f8375e-116e-478d-b3d6-f634936aa0f0","Type":"ContainerStarted","Data":"a2a591c21d8414800c95e3ddefdab5f2ab77c2582ac76dd53c5d1eff276bea51"}
Sep 17 21:55:42.620707 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.620588    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l" event={"ID":"95f8375e-116e-478d-b3d6-f634936aa0f0","Type":"ContainerStarted","Data":"235f8a67fabae9a87021b72940c85b1e54ff84d46fd9c8b2f888ed7faf6daf64"}
Sep 17 21:55:42.620780 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.620757    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l"
Sep 17 21:55:42.622707 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.622666    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp" event={"ID":"4d06b164-d263-4f1b-a9ab-c060136cd754","Type":"ContainerStarted","Data":"6da4e6799b27077abac5e558862cd4712e46b4060cdae6cf683dea94fa274134"}
Sep 17 21:55:42.622821 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.622729    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp" event={"ID":"4d06b164-d263-4f1b-a9ab-c060136cd754","Type":"ContainerStarted","Data":"6eef10a56bc4ca03abf71637b386dfb184ad64c120ce1529ee03dfc130436e7c"}
Sep 17 21:55:42.659450 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.659385    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l" podStartSLOduration=1.659363591 podStartE2EDuration="1.659363591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:55:42.655260678 +0000 UTC m=+858.473586993" watchObservedRunningTime="2026-09-17 21:55:42.659363591 +0000 UTC m=+858.477689908"
Sep 17 21:55:42.659649 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.659592    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf"
Sep 17 21:55:42.713963 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.713719    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp" podStartSLOduration=2.713697484 podStartE2EDuration="2.713697484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:55:42.713419373 +0000 UTC m=+858.531745689" watchObservedRunningTime="2026-09-17 21:55:42.713697484 +0000 UTC m=+858.532023798"
Sep 17 21:55:42.964644 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:42.964617    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf"]
Sep 17 21:55:42.965967 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:55:42.965939    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcfb0f22_9ccc_4012_a92d_7875c900d3da.slice/crio-6e611a5868855f811616a28d877f103507107d8d40cb33cf71edacff824aaf13 WatchSource:0}: Error finding container 6e611a5868855f811616a28d877f103507107d8d40cb33cf71edacff824aaf13: Status 404 returned error can't find the container with id 6e611a5868855f811616a28d877f103507107d8d40cb33cf71edacff824aaf13
Sep 17 21:55:43.402701 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:43.402656    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:43.408079 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:43.408050    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:43.630603 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:43.630564    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf" event={"ID":"dcfb0f22-9ccc-4012-a92d-7875c900d3da","Type":"ContainerStarted","Data":"22ea0452fae25218746738dc8a1c275d86ef939bd6680dc91bdfe23aec345284"}
Sep 17 21:55:43.631125 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:43.631097    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:43.631290 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:43.631152    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf"
Sep 17 21:55:43.631290 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:43.631195    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf" event={"ID":"dcfb0f22-9ccc-4012-a92d-7875c900d3da","Type":"ContainerStarted","Data":"6e611a5868855f811616a28d877f103507107d8d40cb33cf71edacff824aaf13"}
Sep 17 21:55:43.632195 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:43.632149    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:55:43.668836 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:43.668736    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf" podStartSLOduration=1.668721179 podStartE2EDuration="1.668721179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:55:43.664690321 +0000 UTC m=+859.483016631" watchObservedRunningTime="2026-09-17 21:55:43.668721179 +0000 UTC m=+859.487047491"
Sep 17 21:55:44.414709 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:44.414666    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5"]
Sep 17 21:55:44.419209 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:44.419164    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5"
Sep 17 21:55:44.423071 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:44.423050    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-dbe8a7-dockercfg-9v9nl\""
Sep 17 21:55:44.423420 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:44.423400    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-dbe8a7\""
Sep 17 21:55:44.553288 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:44.553250    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5"]
Sep 17 21:55:44.585240 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:44.585210    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhjsz\" (UniqueName: \"kubernetes.io/projected/3e041119-b2c9-4c79-8e6a-13076bd4a91f-kube-api-access-nhjsz\") pod \"payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5\" (UID: \"3e041119-b2c9-4c79-8e6a-13076bd4a91f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5"
Sep 17 21:55:44.585401 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:44.585264    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e041119-b2c9-4c79-8e6a-13076bd4a91f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5\" (UID: \"3e041119-b2c9-4c79-8e6a-13076bd4a91f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5"
Sep 17 21:55:44.686895 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:44.686799    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e041119-b2c9-4c79-8e6a-13076bd4a91f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5\" (UID: \"3e041119-b2c9-4c79-8e6a-13076bd4a91f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5"
Sep 17 21:55:44.687051 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:44.686972    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhjsz\" (UniqueName: \"kubernetes.io/projected/3e041119-b2c9-4c79-8e6a-13076bd4a91f-kube-api-access-nhjsz\") pod \"payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5\" (UID: \"3e041119-b2c9-4c79-8e6a-13076bd4a91f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5"
Sep 17 21:55:44.687969 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:44.687940    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e041119-b2c9-4c79-8e6a-13076bd4a91f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5\" (UID: \"3e041119-b2c9-4c79-8e6a-13076bd4a91f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5"
Sep 17 21:55:44.727033 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:44.727004    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhjsz\" (UniqueName: \"kubernetes.io/projected/3e041119-b2c9-4c79-8e6a-13076bd4a91f-kube-api-access-nhjsz\") pod \"payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5\" (UID: \"3e041119-b2c9-4c79-8e6a-13076bd4a91f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5"
Sep 17 21:55:44.731818 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:44.731795    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5"
Sep 17 21:55:45.043834 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:45.043809    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5"]
Sep 17 21:55:45.049364 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:55:45.046277    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e041119_b2c9_4c79_8e6a_13076bd4a91f.slice/crio-be7fea490e143bd7b6c6a3c70e4980d5d39b31ded12be9254ca440066e04809c WatchSource:0}: Error finding container be7fea490e143bd7b6c6a3c70e4980d5d39b31ded12be9254ca440066e04809c: Status 404 returned error can't find the container with id be7fea490e143bd7b6c6a3c70e4980d5d39b31ded12be9254ca440066e04809c
Sep 17 21:55:45.642526 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:45.642482    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5" event={"ID":"3e041119-b2c9-4c79-8e6a-13076bd4a91f","Type":"ContainerStarted","Data":"515d15d20a032576dce192490db16194e8e69041de0007159d13246f2c11576f"}
Sep 17 21:55:45.642993 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:45.642549    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5" event={"ID":"3e041119-b2c9-4c79-8e6a-13076bd4a91f","Type":"ContainerStarted","Data":"be7fea490e143bd7b6c6a3c70e4980d5d39b31ded12be9254ca440066e04809c"}
Sep 17 21:55:45.642993 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:45.642820    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5"
Sep 17 21:55:45.686977 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:45.686908    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5" podStartSLOduration=1.6868876739999998 podStartE2EDuration="1.686887674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:55:45.682140308 +0000 UTC m=+861.500466621" watchObservedRunningTime="2026-09-17 21:55:45.686887674 +0000 UTC m=+861.505213987"
Sep 17 21:55:50.608400 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:50.608367    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t"
Sep 17 21:55:53.633396 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:53.633363    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l"
Sep 17 21:55:54.639045 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:54.639012    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf"
Sep 17 21:55:55.351139 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:55.351098    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9e9409-5d657977f7-btvn7"]
Sep 17 21:55:55.357965 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:55.357939    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9e9409-5d657977f7-btvn7"
Sep 17 21:55:55.362509 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:55.362477    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-9e9409\""
Sep 17 21:55:55.362736 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:55.362710    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-9e9409-dockercfg-r6z88\""
Sep 17 21:55:55.507473 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:55.507432    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4x5g\" (UniqueName: \"kubernetes.io/projected/0234a335-e7c6-4d1e-825f-1aa77c91fcd2-kube-api-access-s4x5g\") pod \"payload-processing-e2e-autores-9e9409-5d657977f7-btvn7\" (UID: \"0234a335-e7c6-4d1e-825f-1aa77c91fcd2\") " pod="openshift-ingress/payload-processing-e2e-autores-9e9409-5d657977f7-btvn7"
Sep 17 21:55:55.507651 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:55.507561    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0234a335-e7c6-4d1e-825f-1aa77c91fcd2-plugins-config-volume\") pod \"payload-processing-e2e-autores-9e9409-5d657977f7-btvn7\" (UID: \"0234a335-e7c6-4d1e-825f-1aa77c91fcd2\") " pod="openshift-ingress/payload-processing-e2e-autores-9e9409-5d657977f7-btvn7"
Sep 17 21:55:55.517917 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:55.517885    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9e9409-5d657977f7-btvn7"]
Sep 17 21:55:55.608661 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:55.608584    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4x5g\" (UniqueName: \"kubernetes.io/projected/0234a335-e7c6-4d1e-825f-1aa77c91fcd2-kube-api-access-s4x5g\") pod \"payload-processing-e2e-autores-9e9409-5d657977f7-btvn7\" (UID: \"0234a335-e7c6-4d1e-825f-1aa77c91fcd2\") " pod="openshift-ingress/payload-processing-e2e-autores-9e9409-5d657977f7-btvn7"
Sep 17 21:55:55.608661 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:55.608655    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0234a335-e7c6-4d1e-825f-1aa77c91fcd2-plugins-config-volume\") pod \"payload-processing-e2e-autores-9e9409-5d657977f7-btvn7\" (UID: \"0234a335-e7c6-4d1e-825f-1aa77c91fcd2\") " pod="openshift-ingress/payload-processing-e2e-autores-9e9409-5d657977f7-btvn7"
Sep 17 21:55:55.609644 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:55.609615    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0234a335-e7c6-4d1e-825f-1aa77c91fcd2-plugins-config-volume\") pod \"payload-processing-e2e-autores-9e9409-5d657977f7-btvn7\" (UID: \"0234a335-e7c6-4d1e-825f-1aa77c91fcd2\") " pod="openshift-ingress/payload-processing-e2e-autores-9e9409-5d657977f7-btvn7"
Sep 17 21:55:55.655818 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:55.655774    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4x5g\" (UniqueName: \"kubernetes.io/projected/0234a335-e7c6-4d1e-825f-1aa77c91fcd2-kube-api-access-s4x5g\") pod \"payload-processing-e2e-autores-9e9409-5d657977f7-btvn7\" (UID: \"0234a335-e7c6-4d1e-825f-1aa77c91fcd2\") " pod="openshift-ingress/payload-processing-e2e-autores-9e9409-5d657977f7-btvn7"
Sep 17 21:55:55.669447 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:55.669407    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9e9409-5d657977f7-btvn7"
Sep 17 21:55:55.988736 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:55.988707    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9e9409-5d657977f7-btvn7"]
Sep 17 21:55:56.029485 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:55:56.029454    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0234a335_e7c6_4d1e_825f_1aa77c91fcd2.slice/crio-55c435838244d77def6c3799ad88694fcf8b43b2aa9a65e045db3f08749a6d09 WatchSource:0}: Error finding container 55c435838244d77def6c3799ad88694fcf8b43b2aa9a65e045db3f08749a6d09: Status 404 returned error can't find the container with id 55c435838244d77def6c3799ad88694fcf8b43b2aa9a65e045db3f08749a6d09
Sep 17 21:55:56.660269 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:56.655755    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5"
Sep 17 21:55:56.720964 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:56.720933    2564 generic.go:356] "Generic (PLEG): container finished" podID="b112866e-0590-411f-bc6f-8f5fe79ed928" containerID="ec4d54500c65c479b8a1b3bfc5abc4ba8fd061943b3453ee25bd28e8f74e7449" exitCode=0
Sep 17 21:55:56.721156 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:56.721014    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv" event={"ID":"b112866e-0590-411f-bc6f-8f5fe79ed928","Type":"ContainerDied","Data":"ec4d54500c65c479b8a1b3bfc5abc4ba8fd061943b3453ee25bd28e8f74e7449"}
Sep 17 21:55:56.721544 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:56.721521    2564 scope.go:117] "RemoveContainer" containerID="ec4d54500c65c479b8a1b3bfc5abc4ba8fd061943b3453ee25bd28e8f74e7449"
Sep 17 21:55:56.722879 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:56.722840    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:55:56.722998 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:56.722931    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:55:56.722998 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:56.722965    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9e9409-5d657977f7-btvn7" event={"ID":"0234a335-e7c6-4d1e-825f-1aa77c91fcd2","Type":"ContainerStarted","Data":"708cab970228c9879ee70cccf62bfb53b7752511a64c2918d8a23f25aee254a1"}
Sep 17 21:55:56.722998 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:56.722977    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:55:56.723226 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:56.723018    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9e9409-5d657977f7-btvn7" event={"ID":"0234a335-e7c6-4d1e-825f-1aa77c91fcd2","Type":"ContainerStarted","Data":"55c435838244d77def6c3799ad88694fcf8b43b2aa9a65e045db3f08749a6d09"}
Sep 17 21:55:56.723226 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:56.723051    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-9e9409-5d657977f7-btvn7"
Sep 17 21:55:56.724828 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:56.724803    2564 generic.go:356] "Generic (PLEG): container finished" podID="e9febd7d-9eea-4806-8103-09aa77022f2f" containerID="34035e9125a4900c90e3f2cf227155f8e54ecfd6ec2446a771f3e080e757abf3" exitCode=0
Sep 17 21:55:56.724941 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:56.724835    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz" event={"ID":"e9febd7d-9eea-4806-8103-09aa77022f2f","Type":"ContainerDied","Data":"34035e9125a4900c90e3f2cf227155f8e54ecfd6ec2446a771f3e080e757abf3"}
Sep 17 21:55:56.725216 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:56.725196    2564 scope.go:117] "RemoveContainer" containerID="34035e9125a4900c90e3f2cf227155f8e54ecfd6ec2446a771f3e080e757abf3"
Sep 17 21:55:56.726316 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:56.726285    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:55:56.726391 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:56.726365    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:55:56.726431 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:56.726406    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:55:56.912271 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:56.910756    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-9e9409-5d657977f7-btvn7" podStartSLOduration=1.9107339840000002 podStartE2EDuration="1.910733984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:55:56.905442229 +0000 UTC m=+872.723768545" watchObservedRunningTime="2026-09-17 21:55:56.910733984 +0000 UTC m=+872.729060296"
Sep 17 21:55:57.732498 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:57.732464    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv" event={"ID":"b112866e-0590-411f-bc6f-8f5fe79ed928","Type":"ContainerStarted","Data":"9d756e8853db6437f6f042ec036f96ec0313e549aa865c4a0b78bd9555ee952a"}
Sep 17 21:55:57.734431 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:57.734409    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz" event={"ID":"e9febd7d-9eea-4806-8103-09aa77022f2f","Type":"ContainerStarted","Data":"86215180a22172d251a875fe933975569d247bcbbfdff88702e2bd52d8a3d6f5"}
Sep 17 21:55:58.325107 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:58.325065    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:55:58.325470 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:58.325433    2564 patch_prober.go:28] interesting pod/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 17 21:55:58.325607 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:58.325488    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz" podUID="e9febd7d-9eea-4806-8103-09aa77022f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 17 21:55:58.491680 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:58.491641    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:55:58.492282 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:58.492251    2564 patch_prober.go:28] interesting pod/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 17 21:55:58.492484 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:58.492456    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv" podUID="b112866e-0590-411f-bc6f-8f5fe79ed928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 17 21:55:59.325349 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:59.325303    2564 patch_prober.go:28] interesting pod/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 17 21:55:59.325859 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:59.325373    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz" podUID="e9febd7d-9eea-4806-8103-09aa77022f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 17 21:55:59.492061 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:59.492004    2564 patch_prober.go:28] interesting pod/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 17 21:55:59.492437 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:55:59.492403    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv" podUID="b112866e-0590-411f-bc6f-8f5fe79ed928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 17 21:56:00.325402 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:00.325365    2564 patch_prober.go:28] interesting pod/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 17 21:56:00.326014 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:00.325978    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz" podUID="e9febd7d-9eea-4806-8103-09aa77022f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 17 21:56:00.491252 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:00.491215    2564 patch_prober.go:28] interesting pod/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 17 21:56:00.491518 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:00.491491    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv" podUID="b112866e-0590-411f-bc6f-8f5fe79ed928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 17 21:56:01.326025 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:01.325857    2564 patch_prober.go:28] interesting pod/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 17 21:56:01.326816 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:01.326779    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz" podUID="e9febd7d-9eea-4806-8103-09aa77022f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 17 21:56:01.491726 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:01.491678    2564 patch_prober.go:28] interesting pod/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 17 21:56:01.491933 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:01.491752    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv" podUID="b112866e-0590-411f-bc6f-8f5fe79ed928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 17 21:56:02.326234 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:02.326188    2564 patch_prober.go:28] interesting pod/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 17 21:56:02.326734 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:02.326276    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz" podUID="e9febd7d-9eea-4806-8103-09aa77022f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 17 21:56:02.491463 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:02.491400    2564 patch_prober.go:28] interesting pod/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 17 21:56:02.491669 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:02.491516    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv" podUID="b112866e-0590-411f-bc6f-8f5fe79ed928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 17 21:56:03.325900 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:03.325853    2564 patch_prober.go:28] interesting pod/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 17 21:56:03.326112 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:03.325930    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz" podUID="e9febd7d-9eea-4806-8103-09aa77022f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 17 21:56:03.491963 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:03.491921    2564 patch_prober.go:28] interesting pod/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 17 21:56:03.492494 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:03.491987    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv" podUID="b112866e-0590-411f-bc6f-8f5fe79ed928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 17 21:56:04.325824 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:04.325779    2564 patch_prober.go:28] interesting pod/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 17 21:56:04.326198 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:04.326140    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz" podUID="e9febd7d-9eea-4806-8103-09aa77022f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 17 21:56:04.526914 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:04.526540    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:56:04.527999 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:04.527976    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:56:04.530316 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:04.530298    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 21:56:04.573552 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:04.573510    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:56:05.329573 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:05.329538    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:56:05.330727 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:05.330708    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 21:56:06.459507 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:06.459469    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4"]
Sep 17 21:56:06.461994 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:06.461978    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4"
Sep 17 21:56:06.466135 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:06.466116    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4b89cb58\""
Sep 17 21:56:06.484237 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:06.484217    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk"]
Sep 17 21:56:06.492434 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:06.492410    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk"
Sep 17 21:56:06.500463 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:06.500248    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4b89cb58-dockercfg-wkn2d\""
Sep 17 21:56:06.527144 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:06.527112    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab828caa-3067-4d57-9386-0f79ac4d1ef1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4\" (UID: \"ab828caa-3067-4d57-9386-0f79ac4d1ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4"
Sep 17 21:56:06.527338 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:06.527149    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxfvl\" (UniqueName: \"kubernetes.io/projected/15af5e17-af25-40b5-82ae-63e5e4b087ff-kube-api-access-pxfvl\") pod \"payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk\" (UID: \"15af5e17-af25-40b5-82ae-63e5e4b087ff\") " pod="openshift-ingress/payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk"
Sep 17 21:56:06.527338 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:06.527219    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltzbd\" (UniqueName: \"kubernetes.io/projected/ab828caa-3067-4d57-9386-0f79ac4d1ef1-kube-api-access-ltzbd\") pod \"payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4\" (UID: \"ab828caa-3067-4d57-9386-0f79ac4d1ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4"
Sep 17 21:56:06.527338 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:06.527251    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15af5e17-af25-40b5-82ae-63e5e4b087ff-plugins-config-volume\") pod \"payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk\" (UID: \"15af5e17-af25-40b5-82ae-63e5e4b087ff\") " pod="openshift-ingress/payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk"
Sep 17 21:56:06.604634 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:06.604597    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4"]
Sep 17 21:56:06.628480 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:06.628441    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab828caa-3067-4d57-9386-0f79ac4d1ef1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4\" (UID: \"ab828caa-3067-4d57-9386-0f79ac4d1ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4"
Sep 17 21:56:06.628651 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:06.628498    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxfvl\" (UniqueName: \"kubernetes.io/projected/15af5e17-af25-40b5-82ae-63e5e4b087ff-kube-api-access-pxfvl\") pod \"payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk\" (UID: \"15af5e17-af25-40b5-82ae-63e5e4b087ff\") " pod="openshift-ingress/payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk"
Sep 17 21:56:06.628651 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:06.628550    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltzbd\" (UniqueName: \"kubernetes.io/projected/ab828caa-3067-4d57-9386-0f79ac4d1ef1-kube-api-access-ltzbd\") pod \"payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4\" (UID: \"ab828caa-3067-4d57-9386-0f79ac4d1ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4"
Sep 17 21:56:06.628651 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:06.628612    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15af5e17-af25-40b5-82ae-63e5e4b087ff-plugins-config-volume\") pod \"payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk\" (UID: \"15af5e17-af25-40b5-82ae-63e5e4b087ff\") " pod="openshift-ingress/payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk"
Sep 17 21:56:06.629292 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:06.629268    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab828caa-3067-4d57-9386-0f79ac4d1ef1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4\" (UID: \"ab828caa-3067-4d57-9386-0f79ac4d1ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4"
Sep 17 21:56:06.629588 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:06.629565    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15af5e17-af25-40b5-82ae-63e5e4b087ff-plugins-config-volume\") pod \"payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk\" (UID: \"15af5e17-af25-40b5-82ae-63e5e4b087ff\") " pod="openshift-ingress/payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk"
Sep 17 21:56:06.645488 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:06.645458    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltzbd\" (UniqueName: \"kubernetes.io/projected/ab828caa-3067-4d57-9386-0f79ac4d1ef1-kube-api-access-ltzbd\") pod \"payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4\" (UID: \"ab828caa-3067-4d57-9386-0f79ac4d1ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4"
Sep 17 21:56:06.650014 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:06.649989    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxfvl\" (UniqueName: \"kubernetes.io/projected/15af5e17-af25-40b5-82ae-63e5e4b087ff-kube-api-access-pxfvl\") pod \"payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk\" (UID: \"15af5e17-af25-40b5-82ae-63e5e4b087ff\") " pod="openshift-ingress/payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk"
Sep 17 21:56:06.715804 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:06.715708    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk"]
Sep 17 21:56:06.773072 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:06.773034    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4"
Sep 17 21:56:06.811201 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:06.810329    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk"
Sep 17 21:56:07.127879 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:07.127828    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk"]
Sep 17 21:56:07.132205 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:07.131702    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4"]
Sep 17 21:56:07.137683 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:56:07.134927    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab828caa_3067_4d57_9386_0f79ac4d1ef1.slice/crio-9848d839c83ce59a384071ed1027de1be7fcd2ea14685d09b4746a0d06a18ba4 WatchSource:0}: Error finding container 9848d839c83ce59a384071ed1027de1be7fcd2ea14685d09b4746a0d06a18ba4: Status 404 returned error can't find the container with id 9848d839c83ce59a384071ed1027de1be7fcd2ea14685d09b4746a0d06a18ba4
Sep 17 21:56:07.736787 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:07.736760    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-9e9409-5d657977f7-btvn7"
Sep 17 21:56:07.833431 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:07.833362    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk" event={"ID":"15af5e17-af25-40b5-82ae-63e5e4b087ff","Type":"ContainerStarted","Data":"f6b6d0c7bfc8ccb1927151727d53ad779cdef341586d5261b8495f0d4457b772"}
Sep 17 21:56:07.833431 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:07.833431    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk" event={"ID":"15af5e17-af25-40b5-82ae-63e5e4b087ff","Type":"ContainerStarted","Data":"430d0d625d3e28a9afa2b9fc03050588601848ffc1e7a7baf08013762a70b398"}
Sep 17 21:56:07.833687 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:07.833485    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk"
Sep 17 21:56:07.835630 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:07.835467    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4" event={"ID":"ab828caa-3067-4d57-9386-0f79ac4d1ef1","Type":"ContainerStarted","Data":"50e778703b25ffbd55de46fe9f8f8c302ec337c9b313f7d018ee943bc63f6efd"}
Sep 17 21:56:07.835630 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:07.835515    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4" event={"ID":"ab828caa-3067-4d57-9386-0f79ac4d1ef1","Type":"ContainerStarted","Data":"9848d839c83ce59a384071ed1027de1be7fcd2ea14685d09b4746a0d06a18ba4"}
Sep 17 21:56:07.835821 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:07.835638    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4"
Sep 17 21:56:07.876123 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:07.876066    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk" podStartSLOduration=1.876049767 podStartE2EDuration="1.876049767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:56:07.874371115 +0000 UTC m=+883.692697428" watchObservedRunningTime="2026-09-17 21:56:07.876049767 +0000 UTC m=+883.694376079"
Sep 17 21:56:07.920369 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:07.920301    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4" podStartSLOduration=1.9202827980000001 podStartE2EDuration="1.920282798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:56:07.918470508 +0000 UTC m=+883.736796821" watchObservedRunningTime="2026-09-17 21:56:07.920282798 +0000 UTC m=+883.738609111"
Sep 17 21:56:10.856130 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:10.856046    2564 generic.go:356] "Generic (PLEG): container finished" podID="4d06b164-d263-4f1b-a9ab-c060136cd754" containerID="6da4e6799b27077abac5e558862cd4712e46b4060cdae6cf683dea94fa274134" exitCode=0
Sep 17 21:56:10.856528 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:10.856118    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp" event={"ID":"4d06b164-d263-4f1b-a9ab-c060136cd754","Type":"ContainerDied","Data":"6da4e6799b27077abac5e558862cd4712e46b4060cdae6cf683dea94fa274134"}
Sep 17 21:56:10.856595 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:10.856581    2564 scope.go:117] "RemoveContainer" containerID="6da4e6799b27077abac5e558862cd4712e46b4060cdae6cf683dea94fa274134"
Sep 17 21:56:10.857769 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:10.857712    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:56:10.857849 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:10.857834    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:56:10.857897 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:10.857886    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:56:11.870225 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:11.867373    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp" event={"ID":"4d06b164-d263-4f1b-a9ab-c060136cd754","Type":"ContainerStarted","Data":"b4eb3f0791dbbaae352d58df7d2d461653ec9dc7354de370523febe2a064227c"}
Sep 17 21:56:12.402898 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:12.402859    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:56:12.404265 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:12.404223    2564 patch_prober.go:28] interesting pod/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 17 21:56:12.404416 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:12.404316    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp" podUID="4d06b164-d263-4f1b-a9ab-c060136cd754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 17 21:56:13.060532 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:13.060499    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf"]
Sep 17 21:56:13.064232 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:13.064213    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf"
Sep 17 21:56:13.071524 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:13.071497    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-3d5c73\""
Sep 17 21:56:13.071645 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:13.071623    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-3d5c73-dockercfg-4bzlt\""
Sep 17 21:56:13.103200 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:13.103046    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74l5d\" (UniqueName: \"kubernetes.io/projected/104b5a96-2d3c-40b1-be05-8f44d1619aef-kube-api-access-74l5d\") pod \"payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf\" (UID: \"104b5a96-2d3c-40b1-be05-8f44d1619aef\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf"
Sep 17 21:56:13.103200 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:13.103130    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/104b5a96-2d3c-40b1-be05-8f44d1619aef-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf\" (UID: \"104b5a96-2d3c-40b1-be05-8f44d1619aef\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf"
Sep 17 21:56:13.109196 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:13.107468    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf"]
Sep 17 21:56:13.205194 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:13.204513    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74l5d\" (UniqueName: \"kubernetes.io/projected/104b5a96-2d3c-40b1-be05-8f44d1619aef-kube-api-access-74l5d\") pod \"payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf\" (UID: \"104b5a96-2d3c-40b1-be05-8f44d1619aef\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf"
Sep 17 21:56:13.205194 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:13.204615    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/104b5a96-2d3c-40b1-be05-8f44d1619aef-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf\" (UID: \"104b5a96-2d3c-40b1-be05-8f44d1619aef\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf"
Sep 17 21:56:13.209198 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:13.206476    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/104b5a96-2d3c-40b1-be05-8f44d1619aef-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf\" (UID: \"104b5a96-2d3c-40b1-be05-8f44d1619aef\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf"
Sep 17 21:56:13.253954 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:13.253919    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74l5d\" (UniqueName: \"kubernetes.io/projected/104b5a96-2d3c-40b1-be05-8f44d1619aef-kube-api-access-74l5d\") pod \"payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf\" (UID: \"104b5a96-2d3c-40b1-be05-8f44d1619aef\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf"
Sep 17 21:56:13.375283 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:13.375161    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf"
Sep 17 21:56:13.405680 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:13.405646    2564 patch_prober.go:28] interesting pod/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 17 21:56:13.405956 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:13.405926    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp" podUID="4d06b164-d263-4f1b-a9ab-c060136cd754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 17 21:56:13.631099 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:13.631007    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:56:13.708479 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:13.708437    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf"]
Sep 17 21:56:13.714263 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:56:13.713727    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104b5a96_2d3c_40b1_be05_8f44d1619aef.slice/crio-d60d9da78eef087cfd2b4a3ab3846324ca03f14627c86a2583dc7100bf28d878 WatchSource:0}: Error finding container d60d9da78eef087cfd2b4a3ab3846324ca03f14627c86a2583dc7100bf28d878: Status 404 returned error can't find the container with id d60d9da78eef087cfd2b4a3ab3846324ca03f14627c86a2583dc7100bf28d878
Sep 17 21:56:13.880903 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:13.880867    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf" event={"ID":"104b5a96-2d3c-40b1-be05-8f44d1619aef","Type":"ContainerStarted","Data":"9abc11a74300e12fbb721168e4d681358b969ae5baa22e8a16c0ecbfa8748bcd"}
Sep 17 21:56:13.881134 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:13.880930    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf" event={"ID":"104b5a96-2d3c-40b1-be05-8f44d1619aef","Type":"ContainerStarted","Data":"d60d9da78eef087cfd2b4a3ab3846324ca03f14627c86a2583dc7100bf28d878"}
Sep 17 21:56:13.881246 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:13.881201    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf"
Sep 17 21:56:14.403841 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:14.403402    2564 patch_prober.go:28] interesting pod/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 17 21:56:14.403841 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:14.403467    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp" podUID="4d06b164-d263-4f1b-a9ab-c060136cd754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 17 21:56:15.402844 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:15.402810    2564 patch_prober.go:28] interesting pod/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 17 21:56:15.403131 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:15.403104    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp" podUID="4d06b164-d263-4f1b-a9ab-c060136cd754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 17 21:56:16.403302 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:16.403248    2564 patch_prober.go:28] interesting pod/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 17 21:56:16.404022 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:16.403973    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp" podUID="4d06b164-d263-4f1b-a9ab-c060136cd754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 17 21:56:17.403642 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:17.403598    2564 patch_prober.go:28] interesting pod/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 17 21:56:17.404032 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:17.403658    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp" podUID="4d06b164-d263-4f1b-a9ab-c060136cd754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 17 21:56:18.407508 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:18.407477    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:56:18.408681 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:18.408659    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:56:18.439048 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:18.438991    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf" podStartSLOduration=5.438973787 podStartE2EDuration="5.438973787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:56:13.932932448 +0000 UTC m=+889.751258763" watchObservedRunningTime="2026-09-17 21:56:18.438973787 +0000 UTC m=+894.257300138"
Sep 17 21:56:18.612615 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:18.612578    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b89cb58-qww6l"]
Sep 17 21:56:18.616462 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:18.616441    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b89cb58-qww6l"
Sep 17 21:56:18.621764 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:18.621740    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 21:56:18.621764 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:18.621740    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmdbp\""
Sep 17 21:56:18.632821 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:18.632787    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b89cb58-qww6l"]
Sep 17 21:56:18.660603 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:18.660505    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5428aafa-7a75-42ae-bf80-73e9705d2361-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b89cb58-qww6l\" (UID: \"5428aafa-7a75-42ae-bf80-73e9705d2361\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b89cb58-qww6l"
Sep 17 21:56:18.762058 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:18.762027    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5428aafa-7a75-42ae-bf80-73e9705d2361-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b89cb58-qww6l\" (UID: \"5428aafa-7a75-42ae-bf80-73e9705d2361\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b89cb58-qww6l"
Sep 17 21:56:18.762721 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:18.762700    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5428aafa-7a75-42ae-bf80-73e9705d2361-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b89cb58-qww6l\" (UID: \"5428aafa-7a75-42ae-bf80-73e9705d2361\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b89cb58-qww6l"
Sep 17 21:56:18.843685 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:18.843649    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk"
Sep 17 21:56:18.843879 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:18.843782    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4"
Sep 17 21:56:18.927655 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:18.927578    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b89cb58-qww6l"
Sep 17 21:56:19.215909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:19.215881    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b89cb58-qww6l"]
Sep 17 21:56:19.217513 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:56:19.217478    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5428aafa_7a75_42ae_bf80_73e9705d2361.slice/crio-8af9e8452446f9ff8ed4c88a310fe903da9edfa406d4670be90fbc300a37fb75 WatchSource:0}: Error finding container 8af9e8452446f9ff8ed4c88a310fe903da9edfa406d4670be90fbc300a37fb75: Status 404 returned error can't find the container with id 8af9e8452446f9ff8ed4c88a310fe903da9edfa406d4670be90fbc300a37fb75
Sep 17 21:56:19.921001 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:19.920968    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b89cb58-qww6l" event={"ID":"5428aafa-7a75-42ae-bf80-73e9705d2361","Type":"ContainerStarted","Data":"8af9e8452446f9ff8ed4c88a310fe903da9edfa406d4670be90fbc300a37fb75"}
Sep 17 21:56:21.933227 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:21.933190    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b89cb58-qww6l" event={"ID":"5428aafa-7a75-42ae-bf80-73e9705d2361","Type":"ContainerStarted","Data":"8872575afde23a6aa02d7ab2e88e64f259572e94b05b5e2a83250eb1bc57d73e"}
Sep 17 21:56:21.957827 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:21.957768    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b89cb58-qww6l" podStartSLOduration=1.5628070109999999 podStartE2EDuration="3.957750561s" totalImagesPullingTime="2.39494355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:56:21.95610923 +0000 UTC m=+897.774435544" watchObservedRunningTime="2026-09-17 21:56:21.957750561 +0000 UTC m=+897.776076873"
Sep 17 21:56:22.940143 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:22.940110    2564 generic.go:356] "Generic (PLEG): container finished" podID="5428aafa-7a75-42ae-bf80-73e9705d2361" containerID="8872575afde23a6aa02d7ab2e88e64f259572e94b05b5e2a83250eb1bc57d73e" exitCode=0
Sep 17 21:56:22.940578 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:22.940196    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b89cb58-qww6l" event={"ID":"5428aafa-7a75-42ae-bf80-73e9705d2361","Type":"ContainerDied","Data":"8872575afde23a6aa02d7ab2e88e64f259572e94b05b5e2a83250eb1bc57d73e"}
Sep 17 21:56:24.086078 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:24.086054    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b89cb58-qww6l"
Sep 17 21:56:24.122156 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:24.122124    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5428aafa-7a75-42ae-bf80-73e9705d2361-maas-api-service-ca\") pod \"5428aafa-7a75-42ae-bf80-73e9705d2361\" (UID: \"5428aafa-7a75-42ae-bf80-73e9705d2361\") "
Sep 17 21:56:24.122759 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:24.122724    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5428aafa-7a75-42ae-bf80-73e9705d2361-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5428aafa-7a75-42ae-bf80-73e9705d2361" (UID: "5428aafa-7a75-42ae-bf80-73e9705d2361"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:56:24.223547 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:24.223462    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5428aafa-7a75-42ae-bf80-73e9705d2361-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:56:24.678103 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:24.678073    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdcnw_6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2/ovn-acl-logging/0.log"
Sep 17 21:56:24.682101 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:24.682080    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdcnw_6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2/ovn-acl-logging/0.log"
Sep 17 21:56:24.887946 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:24.887915    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf"
Sep 17 21:56:24.953959 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:24.953861    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b89cb58-qww6l" event={"ID":"5428aafa-7a75-42ae-bf80-73e9705d2361","Type":"ContainerDied","Data":"8af9e8452446f9ff8ed4c88a310fe903da9edfa406d4670be90fbc300a37fb75"}
Sep 17 21:56:24.953959 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:24.953892    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b89cb58-qww6l"
Sep 17 21:56:24.953959 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:24.953934    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8af9e8452446f9ff8ed4c88a310fe903da9edfa406d4670be90fbc300a37fb75"
Sep 17 21:56:35.878233 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:35.878198    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk"]
Sep 17 21:56:35.878687 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:35.878479    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk" podUID="15af5e17-af25-40b5-82ae-63e5e4b087ff" containerName="payload-processing" containerID="cri-o://f6b6d0c7bfc8ccb1927151727d53ad779cdef341586d5261b8495f0d4457b772" gracePeriod=30
Sep 17 21:56:35.891466 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:35.891437    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4"]
Sep 17 21:56:35.891679 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:35.891651    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4" podUID="ab828caa-3067-4d57-9386-0f79ac4d1ef1" containerName="payload-pre-processing" containerID="cri-o://50e778703b25ffbd55de46fe9f8f8c302ec337c9b313f7d018ee943bc63f6efd" gracePeriod=30
Sep 17 21:56:36.017403 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:36.017368    2564 generic.go:356] "Generic (PLEG): container finished" podID="ab828caa-3067-4d57-9386-0f79ac4d1ef1" containerID="50e778703b25ffbd55de46fe9f8f8c302ec337c9b313f7d018ee943bc63f6efd" exitCode=0
Sep 17 21:56:36.017544 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:36.017514    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4" event={"ID":"ab828caa-3067-4d57-9386-0f79ac4d1ef1","Type":"ContainerDied","Data":"50e778703b25ffbd55de46fe9f8f8c302ec337c9b313f7d018ee943bc63f6efd"}
Sep 17 21:56:36.019675 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:36.019646    2564 generic.go:356] "Generic (PLEG): container finished" podID="15af5e17-af25-40b5-82ae-63e5e4b087ff" containerID="f6b6d0c7bfc8ccb1927151727d53ad779cdef341586d5261b8495f0d4457b772" exitCode=0
Sep 17 21:56:36.019798 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:36.019710    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk" event={"ID":"15af5e17-af25-40b5-82ae-63e5e4b087ff","Type":"ContainerDied","Data":"f6b6d0c7bfc8ccb1927151727d53ad779cdef341586d5261b8495f0d4457b772"}
Sep 17 21:56:36.191096 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:36.191067    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk"
Sep 17 21:56:36.191540 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:36.191519    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b89cb58-qww6l"]
Sep 17 21:56:36.196915 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:36.196895    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b89cb58-qww6l"]
Sep 17 21:56:36.205981 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:36.205957    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4"
Sep 17 21:56:36.342440 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:36.342403    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15af5e17-af25-40b5-82ae-63e5e4b087ff-plugins-config-volume\") pod \"15af5e17-af25-40b5-82ae-63e5e4b087ff\" (UID: \"15af5e17-af25-40b5-82ae-63e5e4b087ff\") "
Sep 17 21:56:36.342613 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:36.342487    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab828caa-3067-4d57-9386-0f79ac4d1ef1-plugins-config-volume\") pod \"ab828caa-3067-4d57-9386-0f79ac4d1ef1\" (UID: \"ab828caa-3067-4d57-9386-0f79ac4d1ef1\") "
Sep 17 21:56:36.342613 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:36.342534    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxfvl\" (UniqueName: \"kubernetes.io/projected/15af5e17-af25-40b5-82ae-63e5e4b087ff-kube-api-access-pxfvl\") pod \"15af5e17-af25-40b5-82ae-63e5e4b087ff\" (UID: \"15af5e17-af25-40b5-82ae-63e5e4b087ff\") "
Sep 17 21:56:36.342613 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:36.342563    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltzbd\" (UniqueName: \"kubernetes.io/projected/ab828caa-3067-4d57-9386-0f79ac4d1ef1-kube-api-access-ltzbd\") pod \"ab828caa-3067-4d57-9386-0f79ac4d1ef1\" (UID: \"ab828caa-3067-4d57-9386-0f79ac4d1ef1\") "
Sep 17 21:56:36.342935 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:36.342913    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15af5e17-af25-40b5-82ae-63e5e4b087ff-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "15af5e17-af25-40b5-82ae-63e5e4b087ff" (UID: "15af5e17-af25-40b5-82ae-63e5e4b087ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:56:36.343016 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:36.342932    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab828caa-3067-4d57-9386-0f79ac4d1ef1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ab828caa-3067-4d57-9386-0f79ac4d1ef1" (UID: "ab828caa-3067-4d57-9386-0f79ac4d1ef1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:56:36.345154 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:36.345114    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15af5e17-af25-40b5-82ae-63e5e4b087ff-kube-api-access-pxfvl" (OuterVolumeSpecName: "kube-api-access-pxfvl") pod "15af5e17-af25-40b5-82ae-63e5e4b087ff" (UID: "15af5e17-af25-40b5-82ae-63e5e4b087ff"). InnerVolumeSpecName "kube-api-access-pxfvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:56:36.345271 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:36.345195    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab828caa-3067-4d57-9386-0f79ac4d1ef1-kube-api-access-ltzbd" (OuterVolumeSpecName: "kube-api-access-ltzbd") pod "ab828caa-3067-4d57-9386-0f79ac4d1ef1" (UID: "ab828caa-3067-4d57-9386-0f79ac4d1ef1"). InnerVolumeSpecName "kube-api-access-ltzbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:56:36.443870 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:36.443777    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab828caa-3067-4d57-9386-0f79ac4d1ef1-plugins-config-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:56:36.443870 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:36.443808    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxfvl\" (UniqueName: \"kubernetes.io/projected/15af5e17-af25-40b5-82ae-63e5e4b087ff-kube-api-access-pxfvl\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:56:36.443870 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:36.443820    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltzbd\" (UniqueName: \"kubernetes.io/projected/ab828caa-3067-4d57-9386-0f79ac4d1ef1-kube-api-access-ltzbd\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:56:36.443870 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:36.443831    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15af5e17-af25-40b5-82ae-63e5e4b087ff-plugins-config-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:56:36.754053 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:36.753971    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5428aafa-7a75-42ae-bf80-73e9705d2361" path="/var/lib/kubelet/pods/5428aafa-7a75-42ae-bf80-73e9705d2361/volumes"
Sep 17 21:56:37.026748 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.026651    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4"
Sep 17 21:56:37.026748 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.026665    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4" event={"ID":"ab828caa-3067-4d57-9386-0f79ac4d1ef1","Type":"ContainerDied","Data":"9848d839c83ce59a384071ed1027de1be7fcd2ea14685d09b4746a0d06a18ba4"}
Sep 17 21:56:37.027289 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.026757    2564 scope.go:117] "RemoveContainer" containerID="50e778703b25ffbd55de46fe9f8f8c302ec337c9b313f7d018ee943bc63f6efd"
Sep 17 21:56:37.028273 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.028254    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk"
Sep 17 21:56:37.028373 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.028253    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk" event={"ID":"15af5e17-af25-40b5-82ae-63e5e4b087ff","Type":"ContainerDied","Data":"430d0d625d3e28a9afa2b9fc03050588601848ffc1e7a7baf08013762a70b398"}
Sep 17 21:56:37.036378 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.036357    2564 scope.go:117] "RemoveContainer" containerID="f6b6d0c7bfc8ccb1927151727d53ad779cdef341586d5261b8495f0d4457b772"
Sep 17 21:56:37.055061 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.055035    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4"]
Sep 17 21:56:37.061675 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.061648    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b89cb58-6684484d7d-zvtj4"]
Sep 17 21:56:37.080853 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.080825    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk"]
Sep 17 21:56:37.088946 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.088920    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4b89cb58-579d5787bf-z6qgk"]
Sep 17 21:56:37.517902 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.517869    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"]
Sep 17 21:56:37.518297 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.518284    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab828caa-3067-4d57-9386-0f79ac4d1ef1" containerName="payload-pre-processing"
Sep 17 21:56:37.518347 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.518299    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab828caa-3067-4d57-9386-0f79ac4d1ef1" containerName="payload-pre-processing"
Sep 17 21:56:37.518347 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.518308    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5428aafa-7a75-42ae-bf80-73e9705d2361" containerName="revoke-keys"
Sep 17 21:56:37.518347 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.518313    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5428aafa-7a75-42ae-bf80-73e9705d2361" containerName="revoke-keys"
Sep 17 21:56:37.518347 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.518330    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15af5e17-af25-40b5-82ae-63e5e4b087ff" containerName="payload-processing"
Sep 17 21:56:37.518347 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.518335    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="15af5e17-af25-40b5-82ae-63e5e4b087ff" containerName="payload-processing"
Sep 17 21:56:37.518517 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.518402    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab828caa-3067-4d57-9386-0f79ac4d1ef1" containerName="payload-pre-processing"
Sep 17 21:56:37.518517 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.518410    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5428aafa-7a75-42ae-bf80-73e9705d2361" containerName="revoke-keys"
Sep 17 21:56:37.518517 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.518419    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="15af5e17-af25-40b5-82ae-63e5e4b087ff" containerName="payload-processing"
Sep 17 21:56:37.523452 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.523431    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 21:56:37.531329 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.531307    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f996ae\"/\"default-dockercfg-tqvp5\""
Sep 17 21:56:37.531452 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.531307    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f996ae\"/\"openshift-service-ca.crt\""
Sep 17 21:56:37.532544 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.532524    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f996ae\"/\"simulator-w0-f996ae-kserve-self-signed-certs\""
Sep 17 21:56:37.536033 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.536016    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f996ae\"/\"kube-root-ca.crt\""
Sep 17 21:56:37.655895 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.655858    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcn4k\" (UniqueName: \"kubernetes.io/projected/6971204f-1b69-42e6-8327-ebd78192e939-kube-api-access-vcn4k\") pod \"simulator-w0-f996ae-kserve-67677f8797-n52gz\" (UID: \"6971204f-1b69-42e6-8327-ebd78192e939\") " pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 21:56:37.656069 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.655915    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6971204f-1b69-42e6-8327-ebd78192e939-tmp-dir\") pod \"simulator-w0-f996ae-kserve-67677f8797-n52gz\" (UID: \"6971204f-1b69-42e6-8327-ebd78192e939\") " pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 21:56:37.656069 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.655978    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6971204f-1b69-42e6-8327-ebd78192e939-tls-certs\") pod \"simulator-w0-f996ae-kserve-67677f8797-n52gz\" (UID: \"6971204f-1b69-42e6-8327-ebd78192e939\") " pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 21:56:37.656243 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.656074    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6971204f-1b69-42e6-8327-ebd78192e939-model-cache\") pod \"simulator-w0-f996ae-kserve-67677f8797-n52gz\" (UID: \"6971204f-1b69-42e6-8327-ebd78192e939\") " pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 21:56:37.656243 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.656131    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6971204f-1b69-42e6-8327-ebd78192e939-dshm\") pod \"simulator-w0-f996ae-kserve-67677f8797-n52gz\" (UID: \"6971204f-1b69-42e6-8327-ebd78192e939\") " pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 21:56:37.656243 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.656162    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6971204f-1b69-42e6-8327-ebd78192e939-home\") pod \"simulator-w0-f996ae-kserve-67677f8797-n52gz\" (UID: \"6971204f-1b69-42e6-8327-ebd78192e939\") " pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 21:56:37.698923 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.698878    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"]
Sep 17 21:56:37.757210 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.757157    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcn4k\" (UniqueName: \"kubernetes.io/projected/6971204f-1b69-42e6-8327-ebd78192e939-kube-api-access-vcn4k\") pod \"simulator-w0-f996ae-kserve-67677f8797-n52gz\" (UID: \"6971204f-1b69-42e6-8327-ebd78192e939\") " pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 21:56:37.757413 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.757240    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6971204f-1b69-42e6-8327-ebd78192e939-tmp-dir\") pod \"simulator-w0-f996ae-kserve-67677f8797-n52gz\" (UID: \"6971204f-1b69-42e6-8327-ebd78192e939\") " pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 21:56:37.757413 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.757272    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6971204f-1b69-42e6-8327-ebd78192e939-tls-certs\") pod \"simulator-w0-f996ae-kserve-67677f8797-n52gz\" (UID: \"6971204f-1b69-42e6-8327-ebd78192e939\") " pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 21:56:37.757413 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.757317    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6971204f-1b69-42e6-8327-ebd78192e939-model-cache\") pod \"simulator-w0-f996ae-kserve-67677f8797-n52gz\" (UID: \"6971204f-1b69-42e6-8327-ebd78192e939\") " pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 21:56:37.757413 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.757356    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6971204f-1b69-42e6-8327-ebd78192e939-dshm\") pod \"simulator-w0-f996ae-kserve-67677f8797-n52gz\" (UID: \"6971204f-1b69-42e6-8327-ebd78192e939\") " pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 21:56:37.757413 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.757382    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6971204f-1b69-42e6-8327-ebd78192e939-home\") pod \"simulator-w0-f996ae-kserve-67677f8797-n52gz\" (UID: \"6971204f-1b69-42e6-8327-ebd78192e939\") " pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 21:56:37.757669 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.757617    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6971204f-1b69-42e6-8327-ebd78192e939-tmp-dir\") pod \"simulator-w0-f996ae-kserve-67677f8797-n52gz\" (UID: \"6971204f-1b69-42e6-8327-ebd78192e939\") " pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 21:56:37.757732 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.757683    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6971204f-1b69-42e6-8327-ebd78192e939-model-cache\") pod \"simulator-w0-f996ae-kserve-67677f8797-n52gz\" (UID: \"6971204f-1b69-42e6-8327-ebd78192e939\") " pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 21:56:37.757780 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.757738    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6971204f-1b69-42e6-8327-ebd78192e939-home\") pod \"simulator-w0-f996ae-kserve-67677f8797-n52gz\" (UID: \"6971204f-1b69-42e6-8327-ebd78192e939\") " pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 21:56:37.759747 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.759722    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6971204f-1b69-42e6-8327-ebd78192e939-dshm\") pod \"simulator-w0-f996ae-kserve-67677f8797-n52gz\" (UID: \"6971204f-1b69-42e6-8327-ebd78192e939\") " pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 21:56:37.759989 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.759973    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6971204f-1b69-42e6-8327-ebd78192e939-tls-certs\") pod \"simulator-w0-f996ae-kserve-67677f8797-n52gz\" (UID: \"6971204f-1b69-42e6-8327-ebd78192e939\") " pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 21:56:37.776061 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.775994    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcn4k\" (UniqueName: \"kubernetes.io/projected/6971204f-1b69-42e6-8327-ebd78192e939-kube-api-access-vcn4k\") pod \"simulator-w0-f996ae-kserve-67677f8797-n52gz\" (UID: \"6971204f-1b69-42e6-8327-ebd78192e939\") " pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 21:56:37.832872 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:37.832821    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 21:56:38.117524 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:38.117499    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"]
Sep 17 21:56:38.119263 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:56:38.119230    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6971204f_1b69_42e6_8327_ebd78192e939.slice/crio-ff629cbc5ed27fbf9a41c36831c923ea133a04510e3d3cc6e41cfcc88edde8ff WatchSource:0}: Error finding container ff629cbc5ed27fbf9a41c36831c923ea133a04510e3d3cc6e41cfcc88edde8ff: Status 404 returned error can't find the container with id ff629cbc5ed27fbf9a41c36831c923ea133a04510e3d3cc6e41cfcc88edde8ff
Sep 17 21:56:38.755259 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:38.755220    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15af5e17-af25-40b5-82ae-63e5e4b087ff" path="/var/lib/kubelet/pods/15af5e17-af25-40b5-82ae-63e5e4b087ff/volumes"
Sep 17 21:56:38.755695 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:38.755674    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab828caa-3067-4d57-9386-0f79ac4d1ef1" path="/var/lib/kubelet/pods/ab828caa-3067-4d57-9386-0f79ac4d1ef1/volumes"
Sep 17 21:56:39.044784 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:39.044690    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz" event={"ID":"6971204f-1b69-42e6-8327-ebd78192e939","Type":"ContainerStarted","Data":"0cff843bf7e98ea17c72f23b0e6aa9b6269a065b241c8f8ef7f8a59c58d7647b"}
Sep 17 21:56:39.044784 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:39.044736    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz" event={"ID":"6971204f-1b69-42e6-8327-ebd78192e939","Type":"ContainerStarted","Data":"ff629cbc5ed27fbf9a41c36831c923ea133a04510e3d3cc6e41cfcc88edde8ff"}
Sep 17 21:56:39.079214 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:39.079151    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz" podStartSLOduration=2.079134842 podStartE2EDuration="2.079134842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:56:39.071150983 +0000 UTC m=+914.889477294" watchObservedRunningTime="2026-09-17 21:56:39.079134842 +0000 UTC m=+914.897461197"
Sep 17 21:56:47.833969 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:47.833931    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 21:56:47.834414 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:47.833985    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 21:56:47.846442 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:47.846414    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 21:56:48.103354 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:48.103267    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 21:56:49.820804 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:49.820774    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"]
Sep 17 21:56:49.824303 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:49.824288    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 21:56:49.828509 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:49.828486    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-dbe8a7\"/\"simulator-w4-dbe8a7-kserve-self-signed-certs\""
Sep 17 21:56:49.829428 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:49.829410    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-dbe8a7\"/\"openshift-service-ca.crt\""
Sep 17 21:56:49.829528 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:49.829415    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-dbe8a7\"/\"kube-root-ca.crt\""
Sep 17 21:56:49.829887 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:49.829869    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-dbe8a7\"/\"default-dockercfg-r9shz\""
Sep 17 21:56:49.853416 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:49.853385    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"]
Sep 17 21:56:49.977835 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:49.977788    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-home\") pod \"simulator-w4-dbe8a7-kserve-599b998f44-mzvq4\" (UID: \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\") " pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 21:56:49.978023 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:49.977851    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-model-cache\") pod \"simulator-w4-dbe8a7-kserve-599b998f44-mzvq4\" (UID: \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\") " pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 21:56:49.978023 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:49.977888    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-tmp-dir\") pod \"simulator-w4-dbe8a7-kserve-599b998f44-mzvq4\" (UID: \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\") " pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 21:56:49.978023 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:49.977952    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-tls-certs\") pod \"simulator-w4-dbe8a7-kserve-599b998f44-mzvq4\" (UID: \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\") " pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 21:56:49.978023 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:49.977991    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-dshm\") pod \"simulator-w4-dbe8a7-kserve-599b998f44-mzvq4\" (UID: \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\") " pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 21:56:49.978023 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:49.978010    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ddw8\" (UniqueName: \"kubernetes.io/projected/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-kube-api-access-8ddw8\") pod \"simulator-w4-dbe8a7-kserve-599b998f44-mzvq4\" (UID: \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\") " pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 21:56:50.079493 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:50.079399    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-model-cache\") pod \"simulator-w4-dbe8a7-kserve-599b998f44-mzvq4\" (UID: \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\") " pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 21:56:50.079493 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:50.079441    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-tmp-dir\") pod \"simulator-w4-dbe8a7-kserve-599b998f44-mzvq4\" (UID: \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\") " pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 21:56:50.079679 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:50.079518    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-tls-certs\") pod \"simulator-w4-dbe8a7-kserve-599b998f44-mzvq4\" (UID: \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\") " pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 21:56:50.079679 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:50.079569    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-dshm\") pod \"simulator-w4-dbe8a7-kserve-599b998f44-mzvq4\" (UID: \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\") " pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 21:56:50.079679 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:50.079593    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ddw8\" (UniqueName: \"kubernetes.io/projected/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-kube-api-access-8ddw8\") pod \"simulator-w4-dbe8a7-kserve-599b998f44-mzvq4\" (UID: \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\") " pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 21:56:50.079818 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:50.079682    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-home\") pod \"simulator-w4-dbe8a7-kserve-599b998f44-mzvq4\" (UID: \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\") " pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 21:56:50.079886 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:50.079864    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-tmp-dir\") pod \"simulator-w4-dbe8a7-kserve-599b998f44-mzvq4\" (UID: \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\") " pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 21:56:50.079939 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:50.079910    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-model-cache\") pod \"simulator-w4-dbe8a7-kserve-599b998f44-mzvq4\" (UID: \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\") " pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 21:56:50.080041 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:50.080017    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-home\") pod \"simulator-w4-dbe8a7-kserve-599b998f44-mzvq4\" (UID: \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\") " pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 21:56:50.082030 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:50.082008    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-dshm\") pod \"simulator-w4-dbe8a7-kserve-599b998f44-mzvq4\" (UID: \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\") " pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 21:56:50.082116 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:50.082080    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-tls-certs\") pod \"simulator-w4-dbe8a7-kserve-599b998f44-mzvq4\" (UID: \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\") " pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 21:56:50.089096 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:50.089061    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ddw8\" (UniqueName: \"kubernetes.io/projected/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-kube-api-access-8ddw8\") pod \"simulator-w4-dbe8a7-kserve-599b998f44-mzvq4\" (UID: \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\") " pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 21:56:50.133854 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:50.133814    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 21:56:50.430850 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:50.430818    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"]
Sep 17 21:56:50.433739 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:56:50.433705    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d02dd9f_3fb4_4f4b_88e2_c2ae9240fc46.slice/crio-ac0fba64aab8b66f9320e36e6f91d1409a5eef10892306bb60bffd2084305575 WatchSource:0}: Error finding container ac0fba64aab8b66f9320e36e6f91d1409a5eef10892306bb60bffd2084305575: Status 404 returned error can't find the container with id ac0fba64aab8b66f9320e36e6f91d1409a5eef10892306bb60bffd2084305575
Sep 17 21:56:51.109816 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:51.109777    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4" event={"ID":"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46","Type":"ContainerStarted","Data":"a850f05e42f48018c9864c05a05ce72e36320bb3135cb642a8dbbf58f3ca2dc0"}
Sep 17 21:56:51.110469 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:51.109827    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4" event={"ID":"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46","Type":"ContainerStarted","Data":"ac0fba64aab8b66f9320e36e6f91d1409a5eef10892306bb60bffd2084305575"}
Sep 17 21:56:51.135734 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:56:51.135664    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4" podStartSLOduration=2.135637852 podStartE2EDuration="2.135637852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:56:51.132564455 +0000 UTC m=+926.950890779" watchObservedRunningTime="2026-09-17 21:56:51.135637852 +0000 UTC m=+926.953964165"
Sep 17 21:57:00.134348 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:00.134310    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 21:57:00.134897 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:00.134362    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 21:57:00.147137 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:00.147114    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 21:57:00.167867 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:00.167838    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 21:57:10.558319 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:10.558282    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0a97fdf-4xrk5"]
Sep 17 21:57:10.563740 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:10.563719    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0a97fdf-4xrk5"
Sep 17 21:57:10.566615 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:10.566595    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 21:57:10.566706 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:10.566674    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmdbp\""
Sep 17 21:57:10.571279 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:10.571255    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0a97fdf-4xrk5"]
Sep 17 21:57:10.671644 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:10.671599    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc835af-ba00-4237-8fa6-f8c5d132e277-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a0a97fdf-4xrk5\" (UID: \"edc835af-ba00-4237-8fa6-f8c5d132e277\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0a97fdf-4xrk5"
Sep 17 21:57:10.772493 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:10.772453    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc835af-ba00-4237-8fa6-f8c5d132e277-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a0a97fdf-4xrk5\" (UID: \"edc835af-ba00-4237-8fa6-f8c5d132e277\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0a97fdf-4xrk5"
Sep 17 21:57:10.773055 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:10.773035    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc835af-ba00-4237-8fa6-f8c5d132e277-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a0a97fdf-4xrk5\" (UID: \"edc835af-ba00-4237-8fa6-f8c5d132e277\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0a97fdf-4xrk5"
Sep 17 21:57:10.876255 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:10.876155    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0a97fdf-4xrk5"
Sep 17 21:57:11.154092 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:11.154068    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0a97fdf-4xrk5"]
Sep 17 21:57:11.156524 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:57:11.156499    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc835af_ba00_4237_8fa6_f8c5d132e277.slice/crio-98235ce058ece60b96a2ad9706ab493fe2beb37ccda781059fda4f3f6fc473e9 WatchSource:0}: Error finding container 98235ce058ece60b96a2ad9706ab493fe2beb37ccda781059fda4f3f6fc473e9: Status 404 returned error can't find the container with id 98235ce058ece60b96a2ad9706ab493fe2beb37ccda781059fda4f3f6fc473e9
Sep 17 21:57:11.216298 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:11.216266    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0a97fdf-4xrk5" event={"ID":"edc835af-ba00-4237-8fa6-f8c5d132e277","Type":"ContainerStarted","Data":"98235ce058ece60b96a2ad9706ab493fe2beb37ccda781059fda4f3f6fc473e9"}
Sep 17 21:57:12.222909 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:12.222871    2564 generic.go:356] "Generic (PLEG): container finished" podID="edc835af-ba00-4237-8fa6-f8c5d132e277" containerID="a449acbf13910db6387f67b665854eacf0671b2e48610971524c59cc87e67279" exitCode=0
Sep 17 21:57:12.223414 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:12.222925    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0a97fdf-4xrk5" event={"ID":"edc835af-ba00-4237-8fa6-f8c5d132e277","Type":"ContainerDied","Data":"a449acbf13910db6387f67b665854eacf0671b2e48610971524c59cc87e67279"}
Sep 17 21:57:13.373708 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:13.373676    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0a97fdf-4xrk5"
Sep 17 21:57:13.501231 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:13.501106    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc835af-ba00-4237-8fa6-f8c5d132e277-maas-api-service-ca\") pod \"edc835af-ba00-4237-8fa6-f8c5d132e277\" (UID: \"edc835af-ba00-4237-8fa6-f8c5d132e277\") "
Sep 17 21:57:13.501463 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:13.501438    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edc835af-ba00-4237-8fa6-f8c5d132e277-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "edc835af-ba00-4237-8fa6-f8c5d132e277" (UID: "edc835af-ba00-4237-8fa6-f8c5d132e277"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:57:13.501592 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:13.501573    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc835af-ba00-4237-8fa6-f8c5d132e277-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:57:14.240631 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:14.240598    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0a97fdf-4xrk5"
Sep 17 21:57:14.240809 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:14.240639    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0a97fdf-4xrk5" event={"ID":"edc835af-ba00-4237-8fa6-f8c5d132e277","Type":"ContainerDied","Data":"98235ce058ece60b96a2ad9706ab493fe2beb37ccda781059fda4f3f6fc473e9"}
Sep 17 21:57:14.240809 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:14.240691    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98235ce058ece60b96a2ad9706ab493fe2beb37ccda781059fda4f3f6fc473e9"
Sep 17 21:57:22.986381 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:22.986339    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0a97fdf-4xrk5"]
Sep 17 21:57:22.993886 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:22.993853    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0a97fdf-4xrk5"]
Sep 17 21:57:24.754757 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:24.754727    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edc835af-ba00-4237-8fa6-f8c5d132e277" path="/var/lib/kubelet/pods/edc835af-ba00-4237-8fa6-f8c5d132e277/volumes"
Sep 17 21:57:37.860134 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:37.860102    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"]
Sep 17 21:57:37.860560 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:37.860546    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edc835af-ba00-4237-8fa6-f8c5d132e277" containerName="revoke-keys"
Sep 17 21:57:37.860607 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:37.860561    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="edc835af-ba00-4237-8fa6-f8c5d132e277" containerName="revoke-keys"
Sep 17 21:57:37.860645 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:37.860637    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="edc835af-ba00-4237-8fa6-f8c5d132e277" containerName="revoke-keys"
Sep 17 21:57:37.865046 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:37.865028    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 21:57:37.873341 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:37.873317    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6a6c05\"/\"openshift-service-ca.crt\""
Sep 17 21:57:37.873459 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:37.873340    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6a6c05\"/\"default-dockercfg-mrssq\""
Sep 17 21:57:37.873459 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:37.873315    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6a6c05\"/\"test-model-6a6c05-kserve-self-signed-certs\""
Sep 17 21:57:37.875836 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:37.875823    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6a6c05\"/\"kube-root-ca.crt\""
Sep 17 21:57:37.893728 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:37.893701    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"]
Sep 17 21:57:37.956732 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:37.956700    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-dshm\") pod \"test-model-6a6c05-kserve-5966cfccf5-lc5ns\" (UID: \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\") " pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 21:57:37.956911 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:37.956784    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-tls-certs\") pod \"test-model-6a6c05-kserve-5966cfccf5-lc5ns\" (UID: \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\") " pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 21:57:37.956911 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:37.956804    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-model-cache\") pod \"test-model-6a6c05-kserve-5966cfccf5-lc5ns\" (UID: \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\") " pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 21:57:37.956911 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:37.956840    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lt7g\" (UniqueName: \"kubernetes.io/projected/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-kube-api-access-8lt7g\") pod \"test-model-6a6c05-kserve-5966cfccf5-lc5ns\" (UID: \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\") " pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 21:57:37.957012 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:37.956916    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-home\") pod \"test-model-6a6c05-kserve-5966cfccf5-lc5ns\" (UID: \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\") " pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 21:57:37.957012 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:37.956938    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-tmp-dir\") pod \"test-model-6a6c05-kserve-5966cfccf5-lc5ns\" (UID: \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\") " pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 21:57:38.057731 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:38.057693    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-tls-certs\") pod \"test-model-6a6c05-kserve-5966cfccf5-lc5ns\" (UID: \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\") " pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 21:57:38.057731 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:38.057734    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-model-cache\") pod \"test-model-6a6c05-kserve-5966cfccf5-lc5ns\" (UID: \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\") " pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 21:57:38.057968 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:38.057795    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lt7g\" (UniqueName: \"kubernetes.io/projected/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-kube-api-access-8lt7g\") pod \"test-model-6a6c05-kserve-5966cfccf5-lc5ns\" (UID: \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\") " pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 21:57:38.057968 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:38.057840    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-home\") pod \"test-model-6a6c05-kserve-5966cfccf5-lc5ns\" (UID: \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\") " pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 21:57:38.057968 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:38.057862    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-tmp-dir\") pod \"test-model-6a6c05-kserve-5966cfccf5-lc5ns\" (UID: \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\") " pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 21:57:38.057968 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:38.057896    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-dshm\") pod \"test-model-6a6c05-kserve-5966cfccf5-lc5ns\" (UID: \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\") " pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 21:57:38.058292 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:38.058273    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-model-cache\") pod \"test-model-6a6c05-kserve-5966cfccf5-lc5ns\" (UID: \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\") " pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 21:57:38.058360 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:38.058334    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-home\") pod \"test-model-6a6c05-kserve-5966cfccf5-lc5ns\" (UID: \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\") " pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 21:57:38.058415 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:38.058370    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-tmp-dir\") pod \"test-model-6a6c05-kserve-5966cfccf5-lc5ns\" (UID: \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\") " pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 21:57:38.060272 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:38.060248    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-dshm\") pod \"test-model-6a6c05-kserve-5966cfccf5-lc5ns\" (UID: \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\") " pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 21:57:38.060436 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:38.060414    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-tls-certs\") pod \"test-model-6a6c05-kserve-5966cfccf5-lc5ns\" (UID: \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\") " pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 21:57:38.069192 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:38.069155    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lt7g\" (UniqueName: \"kubernetes.io/projected/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-kube-api-access-8lt7g\") pod \"test-model-6a6c05-kserve-5966cfccf5-lc5ns\" (UID: \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\") " pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 21:57:38.174163 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:38.174128    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 21:57:38.468730 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:38.468647    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"]
Sep 17 21:57:38.473243 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:57:38.473214    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12ae6330_0fbf_41ff_bc85_ff4e2322c76e.slice/crio-152b43a456008120bf556d9ddcbd697d821ef9a846dbc0d8763ac798815f36e2 WatchSource:0}: Error finding container 152b43a456008120bf556d9ddcbd697d821ef9a846dbc0d8763ac798815f36e2: Status 404 returned error can't find the container with id 152b43a456008120bf556d9ddcbd697d821ef9a846dbc0d8763ac798815f36e2
Sep 17 21:57:39.376484 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:39.376449    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns" event={"ID":"12ae6330-0fbf-41ff-bc85-ff4e2322c76e","Type":"ContainerStarted","Data":"0008b2ac017d0c13a37e4e4434241fd92b999bcaee104af8526d91d77c3f174d"}
Sep 17 21:57:39.376876 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:39.376494    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns" event={"ID":"12ae6330-0fbf-41ff-bc85-ff4e2322c76e","Type":"ContainerStarted","Data":"152b43a456008120bf556d9ddcbd697d821ef9a846dbc0d8763ac798815f36e2"}
Sep 17 21:57:39.406758 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:39.406709    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns" podStartSLOduration=2.406694439 podStartE2EDuration="2.406694439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:57:39.404313227 +0000 UTC m=+975.222639541" watchObservedRunningTime="2026-09-17 21:57:39.406694439 +0000 UTC m=+975.225020750"
Sep 17 21:57:48.174621 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:48.174519    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 21:57:48.174621 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:48.174583    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 21:57:48.187415 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:48.187388    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 21:57:48.446879 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:48.446797    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 21:57:53.706601 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:53.706568    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e9409-gxvht"]
Sep 17 21:57:53.710120 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:53.710103    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e9409-gxvht"
Sep 17 21:57:53.713048 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:53.713026    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 21:57:53.713195 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:53.713046    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmdbp\""
Sep 17 21:57:53.718928 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:53.718707    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e9409-gxvht"]
Sep 17 21:57:53.814723 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:53.814689    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a587d809-cda7-4122-8986-58190ff3acab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-9e9409-gxvht\" (UID: \"a587d809-cda7-4122-8986-58190ff3acab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e9409-gxvht"
Sep 17 21:57:53.916065 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:53.916024    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a587d809-cda7-4122-8986-58190ff3acab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-9e9409-gxvht\" (UID: \"a587d809-cda7-4122-8986-58190ff3acab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e9409-gxvht"
Sep 17 21:57:53.916689 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:53.916668    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a587d809-cda7-4122-8986-58190ff3acab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-9e9409-gxvht\" (UID: \"a587d809-cda7-4122-8986-58190ff3acab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e9409-gxvht"
Sep 17 21:57:54.023725 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:54.023639    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e9409-gxvht"
Sep 17 21:57:54.315578 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:54.315553    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e9409-gxvht"]
Sep 17 21:57:54.317352 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:57:54.317320    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda587d809_cda7_4122_8986_58190ff3acab.slice/crio-dfc9b9583d458f600df70228ee17145d509b1fe852377f553a0169548f6a31ed WatchSource:0}: Error finding container dfc9b9583d458f600df70228ee17145d509b1fe852377f553a0169548f6a31ed: Status 404 returned error can't find the container with id dfc9b9583d458f600df70228ee17145d509b1fe852377f553a0169548f6a31ed
Sep 17 21:57:54.475967 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:54.475934    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e9409-gxvht" event={"ID":"a587d809-cda7-4122-8986-58190ff3acab","Type":"ContainerStarted","Data":"dfc9b9583d458f600df70228ee17145d509b1fe852377f553a0169548f6a31ed"}
Sep 17 21:57:55.482904 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:55.482872    2564 generic.go:356] "Generic (PLEG): container finished" podID="a587d809-cda7-4122-8986-58190ff3acab" containerID="8d0a36c34df710fb4f18138ab836da8f537cc6fee976d9ec2bc7a22b72edcf6a" exitCode=0
Sep 17 21:57:55.483329 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:55.482954    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e9409-gxvht" event={"ID":"a587d809-cda7-4122-8986-58190ff3acab","Type":"ContainerDied","Data":"8d0a36c34df710fb4f18138ab836da8f537cc6fee976d9ec2bc7a22b72edcf6a"}
Sep 17 21:57:56.637116 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:56.637090    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e9409-gxvht"
Sep 17 21:57:56.743595 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:56.743510    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a587d809-cda7-4122-8986-58190ff3acab-maas-api-service-ca\") pod \"a587d809-cda7-4122-8986-58190ff3acab\" (UID: \"a587d809-cda7-4122-8986-58190ff3acab\") "
Sep 17 21:57:56.743886 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:56.743862    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a587d809-cda7-4122-8986-58190ff3acab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a587d809-cda7-4122-8986-58190ff3acab" (UID: "a587d809-cda7-4122-8986-58190ff3acab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:57:56.845477 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:56.845440    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a587d809-cda7-4122-8986-58190ff3acab-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:57:57.496806 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:57.496769    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e9409-gxvht"
Sep 17 21:57:57.496985 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:57.496772    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e9409-gxvht" event={"ID":"a587d809-cda7-4122-8986-58190ff3acab","Type":"ContainerDied","Data":"dfc9b9583d458f600df70228ee17145d509b1fe852377f553a0169548f6a31ed"}
Sep 17 21:57:57.496985 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:57:57.496904    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfc9b9583d458f600df70228ee17145d509b1fe852377f553a0169548f6a31ed"
Sep 17 21:58:02.659337 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:02.659300    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9151699-g44qf"]
Sep 17 21:58:02.659750 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:02.659737    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a587d809-cda7-4122-8986-58190ff3acab" containerName="revoke-keys"
Sep 17 21:58:02.659792 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:02.659751    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a587d809-cda7-4122-8986-58190ff3acab" containerName="revoke-keys"
Sep 17 21:58:02.659824 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:02.659818    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a587d809-cda7-4122-8986-58190ff3acab" containerName="revoke-keys"
Sep 17 21:58:02.662969 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:02.662951    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9151699-g44qf"
Sep 17 21:58:02.665865 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:02.665844    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 21:58:02.665985 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:02.665911    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmdbp\""
Sep 17 21:58:02.670850 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:02.670443    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9151699-g44qf"]
Sep 17 21:58:02.805552 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:02.805506    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a8b847f-9f2d-401c-97de-84427aea3ea5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a9151699-g44qf\" (UID: \"6a8b847f-9f2d-401c-97de-84427aea3ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9151699-g44qf"
Sep 17 21:58:02.906841 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:02.906796    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a8b847f-9f2d-401c-97de-84427aea3ea5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a9151699-g44qf\" (UID: \"6a8b847f-9f2d-401c-97de-84427aea3ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9151699-g44qf"
Sep 17 21:58:02.907496 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:02.907473    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a8b847f-9f2d-401c-97de-84427aea3ea5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a9151699-g44qf\" (UID: \"6a8b847f-9f2d-401c-97de-84427aea3ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9151699-g44qf"
Sep 17 21:58:02.974755 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:02.974670    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9151699-g44qf"
Sep 17 21:58:03.251761 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:03.251731    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9151699-g44qf"]
Sep 17 21:58:03.252667 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:58:03.252639    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a8b847f_9f2d_401c_97de_84427aea3ea5.slice/crio-6f971d7c2c090cca87807bb20556ac4576cda81d09f9d6ef3212fe0bef783979 WatchSource:0}: Error finding container 6f971d7c2c090cca87807bb20556ac4576cda81d09f9d6ef3212fe0bef783979: Status 404 returned error can't find the container with id 6f971d7c2c090cca87807bb20556ac4576cda81d09f9d6ef3212fe0bef783979
Sep 17 21:58:03.530988 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:03.530949    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9151699-g44qf" event={"ID":"6a8b847f-9f2d-401c-97de-84427aea3ea5","Type":"ContainerStarted","Data":"6f971d7c2c090cca87807bb20556ac4576cda81d09f9d6ef3212fe0bef783979"}
Sep 17 21:58:04.538558 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:04.538523    2564 generic.go:356] "Generic (PLEG): container finished" podID="6a8b847f-9f2d-401c-97de-84427aea3ea5" containerID="8b93f226cf90f2fbba7c031c56381b4739d49b392eba2e17430af5afff751008" exitCode=0
Sep 17 21:58:04.538954 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:04.538591    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9151699-g44qf" event={"ID":"6a8b847f-9f2d-401c-97de-84427aea3ea5","Type":"ContainerDied","Data":"8b93f226cf90f2fbba7c031c56381b4739d49b392eba2e17430af5afff751008"}
Sep 17 21:58:05.686900 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:05.686874    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9151699-g44qf"
Sep 17 21:58:05.834445 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:05.834351    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a8b847f-9f2d-401c-97de-84427aea3ea5-maas-api-service-ca\") pod \"6a8b847f-9f2d-401c-97de-84427aea3ea5\" (UID: \"6a8b847f-9f2d-401c-97de-84427aea3ea5\") "
Sep 17 21:58:05.834751 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:05.834727    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a8b847f-9f2d-401c-97de-84427aea3ea5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6a8b847f-9f2d-401c-97de-84427aea3ea5" (UID: "6a8b847f-9f2d-401c-97de-84427aea3ea5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:58:05.936038 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:05.935995    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a8b847f-9f2d-401c-97de-84427aea3ea5-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:58:06.551456 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:06.551427    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9151699-g44qf"
Sep 17 21:58:06.551456 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:06.551448    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9151699-g44qf" event={"ID":"6a8b847f-9f2d-401c-97de-84427aea3ea5","Type":"ContainerDied","Data":"6f971d7c2c090cca87807bb20556ac4576cda81d09f9d6ef3212fe0bef783979"}
Sep 17 21:58:06.551679 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:06.551497    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f971d7c2c090cca87807bb20556ac4576cda81d09f9d6ef3212fe0bef783979"
Sep 17 21:58:07.342908 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:07.342875    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9e9409-5d657977f7-btvn7"]
Sep 17 21:58:07.343346 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:07.343136    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-9e9409-5d657977f7-btvn7" podUID="0234a335-e7c6-4d1e-825f-1aa77c91fcd2" containerName="payload-processing" containerID="cri-o://708cab970228c9879ee70cccf62bfb53b7752511a64c2918d8a23f25aee254a1" gracePeriod=30
Sep 17 21:58:07.569928 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:07.569324    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e9409-gxvht"]
Sep 17 21:58:07.579115 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:07.579077    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e9409-gxvht"]
Sep 17 21:58:07.582523 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:07.582493    2564 generic.go:356] "Generic (PLEG): container finished" podID="0234a335-e7c6-4d1e-825f-1aa77c91fcd2" containerID="708cab970228c9879ee70cccf62bfb53b7752511a64c2918d8a23f25aee254a1" exitCode=0
Sep 17 21:58:07.582678 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:07.582570    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9e9409-5d657977f7-btvn7" event={"ID":"0234a335-e7c6-4d1e-825f-1aa77c91fcd2","Type":"ContainerDied","Data":"708cab970228c9879ee70cccf62bfb53b7752511a64c2918d8a23f25aee254a1"}
Sep 17 21:58:07.632839 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:07.632811    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9e9409-5d657977f7-btvn7"
Sep 17 21:58:07.755482 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:07.755437    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4x5g\" (UniqueName: \"kubernetes.io/projected/0234a335-e7c6-4d1e-825f-1aa77c91fcd2-kube-api-access-s4x5g\") pod \"0234a335-e7c6-4d1e-825f-1aa77c91fcd2\" (UID: \"0234a335-e7c6-4d1e-825f-1aa77c91fcd2\") "
Sep 17 21:58:07.755706 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:07.755549    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0234a335-e7c6-4d1e-825f-1aa77c91fcd2-plugins-config-volume\") pod \"0234a335-e7c6-4d1e-825f-1aa77c91fcd2\" (UID: \"0234a335-e7c6-4d1e-825f-1aa77c91fcd2\") "
Sep 17 21:58:07.755904 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:07.755880    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0234a335-e7c6-4d1e-825f-1aa77c91fcd2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0234a335-e7c6-4d1e-825f-1aa77c91fcd2" (UID: "0234a335-e7c6-4d1e-825f-1aa77c91fcd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:58:07.757866 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:07.757835    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0234a335-e7c6-4d1e-825f-1aa77c91fcd2-kube-api-access-s4x5g" (OuterVolumeSpecName: "kube-api-access-s4x5g") pod "0234a335-e7c6-4d1e-825f-1aa77c91fcd2" (UID: "0234a335-e7c6-4d1e-825f-1aa77c91fcd2"). InnerVolumeSpecName "kube-api-access-s4x5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:58:07.856981 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:07.856950    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4x5g\" (UniqueName: \"kubernetes.io/projected/0234a335-e7c6-4d1e-825f-1aa77c91fcd2-kube-api-access-s4x5g\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:58:07.856981 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:07.856978    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0234a335-e7c6-4d1e-825f-1aa77c91fcd2-plugins-config-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:58:08.590028 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:08.589998    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9e9409-5d657977f7-btvn7"
Sep 17 21:58:08.590502 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:08.589997    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9e9409-5d657977f7-btvn7" event={"ID":"0234a335-e7c6-4d1e-825f-1aa77c91fcd2","Type":"ContainerDied","Data":"55c435838244d77def6c3799ad88694fcf8b43b2aa9a65e045db3f08749a6d09"}
Sep 17 21:58:08.590502 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:08.590163    2564 scope.go:117] "RemoveContainer" containerID="708cab970228c9879ee70cccf62bfb53b7752511a64c2918d8a23f25aee254a1"
Sep 17 21:58:08.614888 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:08.614857    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9e9409-5d657977f7-btvn7"]
Sep 17 21:58:08.621015 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:08.620993    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9e9409-5d657977f7-btvn7"]
Sep 17 21:58:08.754271 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:08.754238    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0234a335-e7c6-4d1e-825f-1aa77c91fcd2" path="/var/lib/kubelet/pods/0234a335-e7c6-4d1e-825f-1aa77c91fcd2/volumes"
Sep 17 21:58:08.754606 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:08.754594    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a587d809-cda7-4122-8986-58190ff3acab" path="/var/lib/kubelet/pods/a587d809-cda7-4122-8986-58190ff3acab/volumes"
Sep 17 21:58:14.846470 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:14.846428    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9151699-g44qf"]
Sep 17 21:58:14.855061 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:14.855015    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9151699-g44qf"]
Sep 17 21:58:16.754263 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:16.754229    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a8b847f-9f2d-401c-97de-84427aea3ea5" path="/var/lib/kubelet/pods/6a8b847f-9f2d-401c-97de-84427aea3ea5/volumes"
Sep 17 21:58:17.999954 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:17.999920    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"]
Sep 17 21:58:18.000425 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:18.000130    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp" podUID="4d06b164-d263-4f1b-a9ab-c060136cd754" containerName="istio-proxy" containerID="cri-o://b4eb3f0791dbbaae352d58df7d2d461653ec9dc7354de370523febe2a064227c" gracePeriod=30
Sep 17 21:58:18.410848 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:18.410812    2564 patch_prober.go:28] interesting pod/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 21:58:18.411046 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:18.410872    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp" podUID="4d06b164-d263-4f1b-a9ab-c060136cd754" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:58:23.268930 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.268852    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:58:23.309712 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.309681    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqk4j\" (UniqueName: \"kubernetes.io/projected/4d06b164-d263-4f1b-a9ab-c060136cd754-kube-api-access-mqk4j\") pod \"4d06b164-d263-4f1b-a9ab-c060136cd754\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") "
Sep 17 21:58:23.309712 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.309719    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-istio-envoy\") pod \"4d06b164-d263-4f1b-a9ab-c060136cd754\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") "
Sep 17 21:58:23.309974 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.309738    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-credential-socket\") pod \"4d06b164-d263-4f1b-a9ab-c060136cd754\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") "
Sep 17 21:58:23.309974 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.309785    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-workload-socket\") pod \"4d06b164-d263-4f1b-a9ab-c060136cd754\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") "
Sep 17 21:58:23.309974 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.309823    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-istio-data\") pod \"4d06b164-d263-4f1b-a9ab-c060136cd754\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") "
Sep 17 21:58:23.309974 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.309872    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d06b164-d263-4f1b-a9ab-c060136cd754-istio-token\") pod \"4d06b164-d263-4f1b-a9ab-c060136cd754\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") "
Sep 17 21:58:23.309974 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.309896    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d06b164-d263-4f1b-a9ab-c060136cd754-istiod-ca-cert\") pod \"4d06b164-d263-4f1b-a9ab-c060136cd754\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") "
Sep 17 21:58:23.309974 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.309930    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-workload-certs\") pod \"4d06b164-d263-4f1b-a9ab-c060136cd754\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") "
Sep 17 21:58:23.310332 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.309981    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d06b164-d263-4f1b-a9ab-c060136cd754-istio-podinfo\") pod \"4d06b164-d263-4f1b-a9ab-c060136cd754\" (UID: \"4d06b164-d263-4f1b-a9ab-c060136cd754\") "
Sep 17 21:58:23.310332 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.310275    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4d06b164-d263-4f1b-a9ab-c060136cd754" (UID: "4d06b164-d263-4f1b-a9ab-c060136cd754"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:58:23.310437 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.310364    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4d06b164-d263-4f1b-a9ab-c060136cd754" (UID: "4d06b164-d263-4f1b-a9ab-c060136cd754"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:58:23.310497 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.310443    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-istio-data" (OuterVolumeSpecName: "istio-data") pod "4d06b164-d263-4f1b-a9ab-c060136cd754" (UID: "4d06b164-d263-4f1b-a9ab-c060136cd754"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:58:23.310497 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.310476    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4d06b164-d263-4f1b-a9ab-c060136cd754" (UID: "4d06b164-d263-4f1b-a9ab-c060136cd754"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:58:23.310838 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.310691    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d06b164-d263-4f1b-a9ab-c060136cd754-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4d06b164-d263-4f1b-a9ab-c060136cd754" (UID: "4d06b164-d263-4f1b-a9ab-c060136cd754"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:58:23.311459 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.311434    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-istio-data\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:58:23.311586 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.311462    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d06b164-d263-4f1b-a9ab-c060136cd754-istiod-ca-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:58:23.311586 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.311477    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-workload-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:58:23.311586 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.311490    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-credential-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:58:23.311586 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.311504    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-workload-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:58:23.312482 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.312448    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d06b164-d263-4f1b-a9ab-c060136cd754-istio-token" (OuterVolumeSpecName: "istio-token") pod "4d06b164-d263-4f1b-a9ab-c060136cd754" (UID: "4d06b164-d263-4f1b-a9ab-c060136cd754"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:58:23.312482 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.312471    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d06b164-d263-4f1b-a9ab-c060136cd754-kube-api-access-mqk4j" (OuterVolumeSpecName: "kube-api-access-mqk4j") pod "4d06b164-d263-4f1b-a9ab-c060136cd754" (UID: "4d06b164-d263-4f1b-a9ab-c060136cd754"). InnerVolumeSpecName "kube-api-access-mqk4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:58:23.312669 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.312532    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4d06b164-d263-4f1b-a9ab-c060136cd754" (UID: "4d06b164-d263-4f1b-a9ab-c060136cd754"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:58:23.312896 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.312878    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d06b164-d263-4f1b-a9ab-c060136cd754-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4d06b164-d263-4f1b-a9ab-c060136cd754" (UID: "4d06b164-d263-4f1b-a9ab-c060136cd754"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 21:58:23.412136 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.412102    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d06b164-d263-4f1b-a9ab-c060136cd754-istio-podinfo\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:58:23.412136 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.412130    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqk4j\" (UniqueName: \"kubernetes.io/projected/4d06b164-d263-4f1b-a9ab-c060136cd754-kube-api-access-mqk4j\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:58:23.412136 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.412141    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d06b164-d263-4f1b-a9ab-c060136cd754-istio-envoy\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:58:23.412391 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.412151    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d06b164-d263-4f1b-a9ab-c060136cd754-istio-token\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:58:23.680335 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.680306    2564 generic.go:356] "Generic (PLEG): container finished" podID="4d06b164-d263-4f1b-a9ab-c060136cd754" containerID="b4eb3f0791dbbaae352d58df7d2d461653ec9dc7354de370523febe2a064227c" exitCode=0
Sep 17 21:58:23.680517 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.680355    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"
Sep 17 21:58:23.680517 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.680391    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp" event={"ID":"4d06b164-d263-4f1b-a9ab-c060136cd754","Type":"ContainerDied","Data":"b4eb3f0791dbbaae352d58df7d2d461653ec9dc7354de370523febe2a064227c"}
Sep 17 21:58:23.680517 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.680447    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp" event={"ID":"4d06b164-d263-4f1b-a9ab-c060136cd754","Type":"ContainerDied","Data":"6eef10a56bc4ca03abf71637b386dfb184ad64c120ce1529ee03dfc130436e7c"}
Sep 17 21:58:23.680517 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.680469    2564 scope.go:117] "RemoveContainer" containerID="b4eb3f0791dbbaae352d58df7d2d461653ec9dc7354de370523febe2a064227c"
Sep 17 21:58:23.692213 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.692160    2564 scope.go:117] "RemoveContainer" containerID="6da4e6799b27077abac5e558862cd4712e46b4060cdae6cf683dea94fa274134"
Sep 17 21:58:23.700459 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.700443    2564 scope.go:117] "RemoveContainer" containerID="b4eb3f0791dbbaae352d58df7d2d461653ec9dc7354de370523febe2a064227c"
Sep 17 21:58:23.700743 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:58:23.700724    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4eb3f0791dbbaae352d58df7d2d461653ec9dc7354de370523febe2a064227c\": container with ID starting with b4eb3f0791dbbaae352d58df7d2d461653ec9dc7354de370523febe2a064227c not found: ID does not exist" containerID="b4eb3f0791dbbaae352d58df7d2d461653ec9dc7354de370523febe2a064227c"
Sep 17 21:58:23.700793 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.700752    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4eb3f0791dbbaae352d58df7d2d461653ec9dc7354de370523febe2a064227c"} err="failed to get container status \"b4eb3f0791dbbaae352d58df7d2d461653ec9dc7354de370523febe2a064227c\": rpc error: code = NotFound desc = could not find container \"b4eb3f0791dbbaae352d58df7d2d461653ec9dc7354de370523febe2a064227c\": container with ID starting with b4eb3f0791dbbaae352d58df7d2d461653ec9dc7354de370523febe2a064227c not found: ID does not exist"
Sep 17 21:58:23.700793 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.700769    2564 scope.go:117] "RemoveContainer" containerID="6da4e6799b27077abac5e558862cd4712e46b4060cdae6cf683dea94fa274134"
Sep 17 21:58:23.701020 ip-10-0-129-67 kubenswrapper[2564]: E0917 21:58:23.700993    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da4e6799b27077abac5e558862cd4712e46b4060cdae6cf683dea94fa274134\": container with ID starting with 6da4e6799b27077abac5e558862cd4712e46b4060cdae6cf683dea94fa274134 not found: ID does not exist" containerID="6da4e6799b27077abac5e558862cd4712e46b4060cdae6cf683dea94fa274134"
Sep 17 21:58:23.701067 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.701025    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da4e6799b27077abac5e558862cd4712e46b4060cdae6cf683dea94fa274134"} err="failed to get container status \"6da4e6799b27077abac5e558862cd4712e46b4060cdae6cf683dea94fa274134\": rpc error: code = NotFound desc = could not find container \"6da4e6799b27077abac5e558862cd4712e46b4060cdae6cf683dea94fa274134\": container with ID starting with 6da4e6799b27077abac5e558862cd4712e46b4060cdae6cf683dea94fa274134 not found: ID does not exist"
Sep 17 21:58:23.742198 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.742145    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"]
Sep 17 21:58:23.749618 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:23.749593    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-9e9409-openshift-default-57657b7b78-8hmvp"]
Sep 17 21:58:24.753866 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:24.753835    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d06b164-d263-4f1b-a9ab-c060136cd754" path="/var/lib/kubelet/pods/4d06b164-d263-4f1b-a9ab-c060136cd754/volumes"
Sep 17 21:58:35.201678 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.201646    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj"]
Sep 17 21:58:35.202344 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.202323    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d06b164-d263-4f1b-a9ab-c060136cd754" containerName="istio-proxy"
Sep 17 21:58:35.202344 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.202345    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d06b164-d263-4f1b-a9ab-c060136cd754" containerName="istio-proxy"
Sep 17 21:58:35.202520 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.202364    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0234a335-e7c6-4d1e-825f-1aa77c91fcd2" containerName="payload-processing"
Sep 17 21:58:35.202520 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.202372    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0234a335-e7c6-4d1e-825f-1aa77c91fcd2" containerName="payload-processing"
Sep 17 21:58:35.202520 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.202386    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a8b847f-9f2d-401c-97de-84427aea3ea5" containerName="revoke-keys"
Sep 17 21:58:35.202520 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.202394    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a8b847f-9f2d-401c-97de-84427aea3ea5" containerName="revoke-keys"
Sep 17 21:58:35.202520 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.202423    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d06b164-d263-4f1b-a9ab-c060136cd754" containerName="istio-proxy"
Sep 17 21:58:35.202520 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.202431    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d06b164-d263-4f1b-a9ab-c060136cd754" containerName="istio-proxy"
Sep 17 21:58:35.202796 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.202535    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d06b164-d263-4f1b-a9ab-c060136cd754" containerName="istio-proxy"
Sep 17 21:58:35.202796 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.202548    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0234a335-e7c6-4d1e-825f-1aa77c91fcd2" containerName="payload-processing"
Sep 17 21:58:35.202796 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.202559    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a8b847f-9f2d-401c-97de-84427aea3ea5" containerName="revoke-keys"
Sep 17 21:58:35.207680 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.207656    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj"
Sep 17 21:58:35.216738 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.216717    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-4bd0ef\""
Sep 17 21:58:35.241105 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.241061    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf"]
Sep 17 21:58:35.241617 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.241604    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d06b164-d263-4f1b-a9ab-c060136cd754" containerName="istio-proxy"
Sep 17 21:58:35.244767 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.244753    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf"
Sep 17 21:58:35.247806 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.247786    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-4bd0ef-dockercfg-ccxbq\""
Sep 17 21:58:35.253803 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.253778    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj"]
Sep 17 21:58:35.281088 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.281060    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf"]
Sep 17 21:58:35.327779 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.327747    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a7ee96-4642-405d-b9f4-82111bc86046-plugins-config-volume\") pod \"payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf\" (UID: \"96a7ee96-4642-405d-b9f4-82111bc86046\") " pod="openshift-ingress/payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf"
Sep 17 21:58:35.327948 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.327786    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e383431-da15-4c63-a8c4-13217af0e721-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj\" (UID: \"6e383431-da15-4c63-a8c4-13217af0e721\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj"
Sep 17 21:58:35.327948 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.327822    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm7wq\" (UniqueName: \"kubernetes.io/projected/96a7ee96-4642-405d-b9f4-82111bc86046-kube-api-access-qm7wq\") pod \"payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf\" (UID: \"96a7ee96-4642-405d-b9f4-82111bc86046\") " pod="openshift-ingress/payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf"
Sep 17 21:58:35.327948 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.327917    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bs5d\" (UniqueName: \"kubernetes.io/projected/6e383431-da15-4c63-a8c4-13217af0e721-kube-api-access-9bs5d\") pod \"payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj\" (UID: \"6e383431-da15-4c63-a8c4-13217af0e721\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj"
Sep 17 21:58:35.428886 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.428848    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bs5d\" (UniqueName: \"kubernetes.io/projected/6e383431-da15-4c63-a8c4-13217af0e721-kube-api-access-9bs5d\") pod \"payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj\" (UID: \"6e383431-da15-4c63-a8c4-13217af0e721\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj"
Sep 17 21:58:35.429066 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.428912    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a7ee96-4642-405d-b9f4-82111bc86046-plugins-config-volume\") pod \"payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf\" (UID: \"96a7ee96-4642-405d-b9f4-82111bc86046\") " pod="openshift-ingress/payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf"
Sep 17 21:58:35.429066 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.428929    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e383431-da15-4c63-a8c4-13217af0e721-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj\" (UID: \"6e383431-da15-4c63-a8c4-13217af0e721\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj"
Sep 17 21:58:35.429066 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.428957    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm7wq\" (UniqueName: \"kubernetes.io/projected/96a7ee96-4642-405d-b9f4-82111bc86046-kube-api-access-qm7wq\") pod \"payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf\" (UID: \"96a7ee96-4642-405d-b9f4-82111bc86046\") " pod="openshift-ingress/payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf"
Sep 17 21:58:35.429752 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.429727    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e383431-da15-4c63-a8c4-13217af0e721-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj\" (UID: \"6e383431-da15-4c63-a8c4-13217af0e721\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj"
Sep 17 21:58:35.429841 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.429757    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a7ee96-4642-405d-b9f4-82111bc86046-plugins-config-volume\") pod \"payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf\" (UID: \"96a7ee96-4642-405d-b9f4-82111bc86046\") " pod="openshift-ingress/payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf"
Sep 17 21:58:35.449771 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.449741    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bs5d\" (UniqueName: \"kubernetes.io/projected/6e383431-da15-4c63-a8c4-13217af0e721-kube-api-access-9bs5d\") pod \"payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj\" (UID: \"6e383431-da15-4c63-a8c4-13217af0e721\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj"
Sep 17 21:58:35.451185 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.451142    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm7wq\" (UniqueName: \"kubernetes.io/projected/96a7ee96-4642-405d-b9f4-82111bc86046-kube-api-access-qm7wq\") pod \"payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf\" (UID: \"96a7ee96-4642-405d-b9f4-82111bc86046\") " pod="openshift-ingress/payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf"
Sep 17 21:58:35.519082 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.518989    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj"
Sep 17 21:58:35.556418 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.556384    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf"
Sep 17 21:58:35.829649 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.829626    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj"]
Sep 17 21:58:35.832017 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:58:35.831985    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e383431_da15_4c63_a8c4_13217af0e721.slice/crio-c0205869b47ccd7c9c7ea2a3306236be5cc88a07dcce3ec88ff7d7f34719eb18 WatchSource:0}: Error finding container c0205869b47ccd7c9c7ea2a3306236be5cc88a07dcce3ec88ff7d7f34719eb18: Status 404 returned error can't find the container with id c0205869b47ccd7c9c7ea2a3306236be5cc88a07dcce3ec88ff7d7f34719eb18
Sep 17 21:58:35.907689 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:35.907575    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf"]
Sep 17 21:58:35.910347 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:58:35.910319    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a7ee96_4642_405d_b9f4_82111bc86046.slice/crio-532e45d55962adba600a945812f87459b566952c617fac752c5827a13162d4a5 WatchSource:0}: Error finding container 532e45d55962adba600a945812f87459b566952c617fac752c5827a13162d4a5: Status 404 returned error can't find the container with id 532e45d55962adba600a945812f87459b566952c617fac752c5827a13162d4a5
Sep 17 21:58:36.757147 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:36.757107    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj" event={"ID":"6e383431-da15-4c63-a8c4-13217af0e721","Type":"ContainerStarted","Data":"ee7b851c001693ebbf47757f9c941fbe65be710503d4292973f83caf3b81c0dc"}
Sep 17 21:58:36.757607 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:36.757188    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj" event={"ID":"6e383431-da15-4c63-a8c4-13217af0e721","Type":"ContainerStarted","Data":"c0205869b47ccd7c9c7ea2a3306236be5cc88a07dcce3ec88ff7d7f34719eb18"}
Sep 17 21:58:36.757607 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:36.757313    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj"
Sep 17 21:58:36.758810 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:36.758780    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf" event={"ID":"96a7ee96-4642-405d-b9f4-82111bc86046","Type":"ContainerStarted","Data":"554c9d59d15b05cea3558995f220e76f6be68a1b3ddf1692f93567bdad12b75a"}
Sep 17 21:58:36.758934 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:36.758824    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf" event={"ID":"96a7ee96-4642-405d-b9f4-82111bc86046","Type":"ContainerStarted","Data":"532e45d55962adba600a945812f87459b566952c617fac752c5827a13162d4a5"}
Sep 17 21:58:36.758934 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:36.758881    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf"
Sep 17 21:58:36.800881 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:36.800828    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj" podStartSLOduration=1.800807147 podStartE2EDuration="1.800807147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:36.798634099 +0000 UTC m=+1032.616960410" watchObservedRunningTime="2026-09-17 21:58:36.800807147 +0000 UTC m=+1032.619133460"
Sep 17 21:58:36.863836 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:36.863776    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf" podStartSLOduration=1.863754717 podStartE2EDuration="1.863754717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:36.863011631 +0000 UTC m=+1032.681337943" watchObservedRunningTime="2026-09-17 21:58:36.863754717 +0000 UTC m=+1032.682081032"
Sep 17 21:58:40.075095 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:40.075061    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c85107f9-cqsk7"]
Sep 17 21:58:40.079089 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:40.079071    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c85107f9-cqsk7"
Sep 17 21:58:40.082334 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:40.082317    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 21:58:40.082457 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:40.082403    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmdbp\""
Sep 17 21:58:40.088735 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:40.088707    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c85107f9-cqsk7"]
Sep 17 21:58:40.177613 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:40.177576    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/983a7479-f973-4b5f-a8ac-fe48542f1246-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c85107f9-cqsk7\" (UID: \"983a7479-f973-4b5f-a8ac-fe48542f1246\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c85107f9-cqsk7"
Sep 17 21:58:40.278245 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:40.278204    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/983a7479-f973-4b5f-a8ac-fe48542f1246-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c85107f9-cqsk7\" (UID: \"983a7479-f973-4b5f-a8ac-fe48542f1246\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c85107f9-cqsk7"
Sep 17 21:58:40.278853 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:40.278833    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/983a7479-f973-4b5f-a8ac-fe48542f1246-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c85107f9-cqsk7\" (UID: \"983a7479-f973-4b5f-a8ac-fe48542f1246\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c85107f9-cqsk7"
Sep 17 21:58:40.392535 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:40.392449    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c85107f9-cqsk7"
Sep 17 21:58:40.668113 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:40.668085    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c85107f9-cqsk7"]
Sep 17 21:58:40.669541 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:58:40.669513    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983a7479_f973_4b5f_a8ac_fe48542f1246.slice/crio-b93a2ec96d8c5dc2fc7c955947773e2e5339f9ad29578e500454bbc689f83681 WatchSource:0}: Error finding container b93a2ec96d8c5dc2fc7c955947773e2e5339f9ad29578e500454bbc689f83681: Status 404 returned error can't find the container with id b93a2ec96d8c5dc2fc7c955947773e2e5339f9ad29578e500454bbc689f83681
Sep 17 21:58:40.780599 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:40.780568    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c85107f9-cqsk7" event={"ID":"983a7479-f973-4b5f-a8ac-fe48542f1246","Type":"ContainerStarted","Data":"b93a2ec96d8c5dc2fc7c955947773e2e5339f9ad29578e500454bbc689f83681"}
Sep 17 21:58:41.789959 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:41.789869    2564 generic.go:356] "Generic (PLEG): container finished" podID="983a7479-f973-4b5f-a8ac-fe48542f1246" containerID="2d64368dfac8de3bb105526cded05736f140d7df304f6856a33f561f4f248df2" exitCode=0
Sep 17 21:58:41.790400 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:41.789944    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c85107f9-cqsk7" event={"ID":"983a7479-f973-4b5f-a8ac-fe48542f1246","Type":"ContainerDied","Data":"2d64368dfac8de3bb105526cded05736f140d7df304f6856a33f561f4f248df2"}
Sep 17 21:58:42.945527 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:42.945499    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c85107f9-cqsk7"
Sep 17 21:58:43.004523 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:43.004486    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/983a7479-f973-4b5f-a8ac-fe48542f1246-maas-api-service-ca\") pod \"983a7479-f973-4b5f-a8ac-fe48542f1246\" (UID: \"983a7479-f973-4b5f-a8ac-fe48542f1246\") "
Sep 17 21:58:43.004829 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:43.004807    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/983a7479-f973-4b5f-a8ac-fe48542f1246-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "983a7479-f973-4b5f-a8ac-fe48542f1246" (UID: "983a7479-f973-4b5f-a8ac-fe48542f1246"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:58:43.106107 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:43.106030    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/983a7479-f973-4b5f-a8ac-fe48542f1246-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 21:58:43.803896 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:43.803863    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c85107f9-cqsk7"
Sep 17 21:58:43.804073 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:43.803914    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c85107f9-cqsk7" event={"ID":"983a7479-f973-4b5f-a8ac-fe48542f1246","Type":"ContainerDied","Data":"b93a2ec96d8c5dc2fc7c955947773e2e5339f9ad29578e500454bbc689f83681"}
Sep 17 21:58:43.804073 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:43.803961    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b93a2ec96d8c5dc2fc7c955947773e2e5339f9ad29578e500454bbc689f83681"
Sep 17 21:58:47.765787 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:47.765756    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf"
Sep 17 21:58:47.766250 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:47.765835    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj"
Sep 17 21:58:53.929971 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:53.929933    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c85107f9-cqsk7"]
Sep 17 21:58:53.944592 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:53.944553    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c85107f9-cqsk7"]
Sep 17 21:58:54.755647 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:58:54.755616    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="983a7479-f973-4b5f-a8ac-fe48542f1246" path="/var/lib/kubelet/pods/983a7479-f973-4b5f-a8ac-fe48542f1246/volumes"
Sep 17 21:59:57.915996 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:57.915952    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"]
Sep 17 21:59:57.916540 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:57.916405    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="983a7479-f973-4b5f-a8ac-fe48542f1246" containerName="revoke-keys"
Sep 17 21:59:57.916540 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:57.916418    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="983a7479-f973-4b5f-a8ac-fe48542f1246" containerName="revoke-keys"
Sep 17 21:59:57.916540 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:57.916483    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="983a7479-f973-4b5f-a8ac-fe48542f1246" containerName="revoke-keys"
Sep 17 21:59:57.919755 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:57.919739    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 21:59:57.924143 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:57.924123    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-4bd0ef\"/\"kube-root-ca.crt\""
Sep 17 21:59:57.924340 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:57.924122    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-4bd0ef\"/\"default-dockercfg-m9w7k\""
Sep 17 21:59:57.924340 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:57.924275    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-4bd0ef\"/\"explicit-model-4bd0ef-kserve-self-signed-certs\""
Sep 17 21:59:57.924340 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:57.924312    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-4bd0ef\"/\"openshift-service-ca.crt\""
Sep 17 21:59:57.949035 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:57.949011    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"]
Sep 17 21:59:58.051648 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:58.051613    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6284ed91-64e6-4acf-b78f-a1741bc6979b-tmp-dir\") pod \"explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq\" (UID: \"6284ed91-64e6-4acf-b78f-a1741bc6979b\") " pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 21:59:58.051862 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:58.051677    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6284ed91-64e6-4acf-b78f-a1741bc6979b-home\") pod \"explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq\" (UID: \"6284ed91-64e6-4acf-b78f-a1741bc6979b\") " pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 21:59:58.051862 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:58.051724    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rvfm\" (UniqueName: \"kubernetes.io/projected/6284ed91-64e6-4acf-b78f-a1741bc6979b-kube-api-access-5rvfm\") pod \"explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq\" (UID: \"6284ed91-64e6-4acf-b78f-a1741bc6979b\") " pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 21:59:58.051862 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:58.051771    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6284ed91-64e6-4acf-b78f-a1741bc6979b-dshm\") pod \"explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq\" (UID: \"6284ed91-64e6-4acf-b78f-a1741bc6979b\") " pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 21:59:58.051862 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:58.051804    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6284ed91-64e6-4acf-b78f-a1741bc6979b-tls-certs\") pod \"explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq\" (UID: \"6284ed91-64e6-4acf-b78f-a1741bc6979b\") " pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 21:59:58.052005 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:58.051866    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6284ed91-64e6-4acf-b78f-a1741bc6979b-model-cache\") pod \"explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq\" (UID: \"6284ed91-64e6-4acf-b78f-a1741bc6979b\") " pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 21:59:58.153457 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:58.153409    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6284ed91-64e6-4acf-b78f-a1741bc6979b-tmp-dir\") pod \"explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq\" (UID: \"6284ed91-64e6-4acf-b78f-a1741bc6979b\") " pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 21:59:58.153653 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:58.153479    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6284ed91-64e6-4acf-b78f-a1741bc6979b-home\") pod \"explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq\" (UID: \"6284ed91-64e6-4acf-b78f-a1741bc6979b\") " pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 21:59:58.153653 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:58.153509    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rvfm\" (UniqueName: \"kubernetes.io/projected/6284ed91-64e6-4acf-b78f-a1741bc6979b-kube-api-access-5rvfm\") pod \"explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq\" (UID: \"6284ed91-64e6-4acf-b78f-a1741bc6979b\") " pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 21:59:58.153653 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:58.153546    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6284ed91-64e6-4acf-b78f-a1741bc6979b-dshm\") pod \"explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq\" (UID: \"6284ed91-64e6-4acf-b78f-a1741bc6979b\") " pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 21:59:58.153653 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:58.153588    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6284ed91-64e6-4acf-b78f-a1741bc6979b-tls-certs\") pod \"explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq\" (UID: \"6284ed91-64e6-4acf-b78f-a1741bc6979b\") " pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 21:59:58.153653 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:58.153635    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6284ed91-64e6-4acf-b78f-a1741bc6979b-model-cache\") pod \"explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq\" (UID: \"6284ed91-64e6-4acf-b78f-a1741bc6979b\") " pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 21:59:58.153948 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:58.153871    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6284ed91-64e6-4acf-b78f-a1741bc6979b-home\") pod \"explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq\" (UID: \"6284ed91-64e6-4acf-b78f-a1741bc6979b\") " pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 21:59:58.153948 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:58.153917    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6284ed91-64e6-4acf-b78f-a1741bc6979b-tmp-dir\") pod \"explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq\" (UID: \"6284ed91-64e6-4acf-b78f-a1741bc6979b\") " pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 21:59:58.154030 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:58.153975    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6284ed91-64e6-4acf-b78f-a1741bc6979b-model-cache\") pod \"explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq\" (UID: \"6284ed91-64e6-4acf-b78f-a1741bc6979b\") " pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 21:59:58.155993 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:58.155972    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6284ed91-64e6-4acf-b78f-a1741bc6979b-dshm\") pod \"explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq\" (UID: \"6284ed91-64e6-4acf-b78f-a1741bc6979b\") " pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 21:59:58.156259 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:58.156241    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6284ed91-64e6-4acf-b78f-a1741bc6979b-tls-certs\") pod \"explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq\" (UID: \"6284ed91-64e6-4acf-b78f-a1741bc6979b\") " pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 21:59:58.165354 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:58.165328    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rvfm\" (UniqueName: \"kubernetes.io/projected/6284ed91-64e6-4acf-b78f-a1741bc6979b-kube-api-access-5rvfm\") pod \"explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq\" (UID: \"6284ed91-64e6-4acf-b78f-a1741bc6979b\") " pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 21:59:58.229612 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:58.229530    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 21:59:58.516099 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:58.516071    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"]
Sep 17 21:59:58.517809 ip-10-0-129-67 kubenswrapper[2564]: W0917 21:59:58.517777    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6284ed91_64e6_4acf_b78f_a1741bc6979b.slice/crio-dc9c1d386425fa43420b210790844b9768099dcc63a96f5c014f422e57e00ea1 WatchSource:0}: Error finding container dc9c1d386425fa43420b210790844b9768099dcc63a96f5c014f422e57e00ea1: Status 404 returned error can't find the container with id dc9c1d386425fa43420b210790844b9768099dcc63a96f5c014f422e57e00ea1
Sep 17 21:59:59.228508 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:59.228475    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq" event={"ID":"6284ed91-64e6-4acf-b78f-a1741bc6979b","Type":"ContainerStarted","Data":"8198205b3275b23a94ccb720c794be035fc8cf9c7a6166ccf9aafc3191059203"}
Sep 17 21:59:59.228508 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:59.228517    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq" event={"ID":"6284ed91-64e6-4acf-b78f-a1741bc6979b","Type":"ContainerStarted","Data":"dc9c1d386425fa43420b210790844b9768099dcc63a96f5c014f422e57e00ea1"}
Sep 17 21:59:59.270542 ip-10-0-129-67 kubenswrapper[2564]: I0917 21:59:59.270477    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq" podStartSLOduration=2.27045997 podStartE2EDuration="2.27045997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:59.268777898 +0000 UTC m=+1115.087104211" watchObservedRunningTime="2026-09-17 21:59:59.27045997 +0000 UTC m=+1115.088786283"
Sep 17 22:00:00.190492 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:00.190451    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f996ae-29828040-d6qfj"]
Sep 17 22:00:00.194795 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:00.194765    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3d5c73-29828040-php7j"]
Sep 17 22:00:00.194951 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:00.194890    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f996ae-29828040-d6qfj"
Sep 17 22:00:00.197871 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:00.197848    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmdbp\""
Sep 17 22:00:00.198350 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:00.198333    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3d5c73-29828040-php7j"
Sep 17 22:00:00.198858 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:00.198841    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828040-kxmw9"]
Sep 17 22:00:00.202319 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:00.202301    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828040-kxmw9"
Sep 17 22:00:00.207110 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:00.207089    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f996ae-29828040-d6qfj"
Sep 17 22:00:00.213341 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:00.213243    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f996ae-29828040-d6qfj"]
Sep 17 22:00:00.218420 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:00.218386    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3d5c73-29828040-php7j"
Sep 17 22:00:00.218723 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:00.218695    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3d5c73-29828040-php7j"]
Sep 17 22:00:00.238920 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:00.238342    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a6c05-29828040-n57qk"]
Sep 17 22:00:00.254666 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:00.254488    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828040-kxmw9"]
Sep 17 22:00:00.254666 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:00.254616    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a6c05-29828040-n57qk"
Sep 17 22:00:00.258046 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:00.257826    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828040-kxmw9"
Sep 17 22:00:00.260762 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:00.260680    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a6c05-29828040-n57qk"]
Sep 17 22:00:00.272584 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:00.270514    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a6c05-29828040-n57qk"
Sep 17 22:00:00.507533 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:00.507470    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f996ae-29828040-d6qfj"]
Sep 17 22:00:00.510259 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:00:00.510229    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca15b6e_4aa8_4347_9556_f2f77814463d.slice/crio-311ac1d363eee3765a1081fd19be356f99e88bc6e3fd9772784bceaa8a93ebe6 WatchSource:0}: Error finding container 311ac1d363eee3765a1081fd19be356f99e88bc6e3fd9772784bceaa8a93ebe6: Status 404 returned error can't find the container with id 311ac1d363eee3765a1081fd19be356f99e88bc6e3fd9772784bceaa8a93ebe6
Sep 17 22:00:00.533988 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:00.533968    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3d5c73-29828040-php7j"]
Sep 17 22:00:00.536158 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:00:00.536133    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc523d0e7_283a_4668_b776_59cfe4800cbf.slice/crio-d19c2d3b381b9f3d8ca563a650538c88376eadb8f2e9ee8935bc342cad9f608c WatchSource:0}: Error finding container d19c2d3b381b9f3d8ca563a650538c88376eadb8f2e9ee8935bc342cad9f608c: Status 404 returned error can't find the container with id d19c2d3b381b9f3d8ca563a650538c88376eadb8f2e9ee8935bc342cad9f608c
Sep 17 22:00:00.572881 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:00.572856    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828040-kxmw9"]
Sep 17 22:00:00.575070 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:00:00.575045    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e74d340_4b98_4849_a48d_fdc890e79751.slice/crio-ecee3de151a6952a5da9e52cb33c5262def3cfece2a7f4d8ee7e61d975f34e5b WatchSource:0}: Error finding container ecee3de151a6952a5da9e52cb33c5262def3cfece2a7f4d8ee7e61d975f34e5b: Status 404 returned error can't find the container with id ecee3de151a6952a5da9e52cb33c5262def3cfece2a7f4d8ee7e61d975f34e5b
Sep 17 22:00:00.598628 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:00.594535    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a6c05-29828040-n57qk"]
Sep 17 22:00:00.600986 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:00:00.600963    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e0b9d01_6e7d_43c5_bde8_ddefe4313d4f.slice/crio-433a6e6fc0bf1a528d1696643bb803ea259cdd62a78a3b902b99a88e08f1fdde WatchSource:0}: Error finding container 433a6e6fc0bf1a528d1696643bb803ea259cdd62a78a3b902b99a88e08f1fdde: Status 404 returned error can't find the container with id 433a6e6fc0bf1a528d1696643bb803ea259cdd62a78a3b902b99a88e08f1fdde
Sep 17 22:00:01.244406 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:01.244365    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a6c05-29828040-n57qk" event={"ID":"3e0b9d01-6e7d-43c5-bde8-ddefe4313d4f","Type":"ContainerStarted","Data":"89c7d5e15e535aa9de8f31c2f744fdfeff3f83e3b83e5d3e975d77931478407e"}
Sep 17 22:00:01.244833 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:01.244433    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a6c05-29828040-n57qk" event={"ID":"3e0b9d01-6e7d-43c5-bde8-ddefe4313d4f","Type":"ContainerStarted","Data":"433a6e6fc0bf1a528d1696643bb803ea259cdd62a78a3b902b99a88e08f1fdde"}
Sep 17 22:00:01.245957 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:01.245931    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3d5c73-29828040-php7j" event={"ID":"c523d0e7-283a-4668-b776-59cfe4800cbf","Type":"ContainerStarted","Data":"7f8ccbc18b6a86a2679e489b9bcf68ff46e8fa0fb1cf2235b3c694f823a9afc8"}
Sep 17 22:00:01.246050 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:01.245980    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3d5c73-29828040-php7j" event={"ID":"c523d0e7-283a-4668-b776-59cfe4800cbf","Type":"ContainerStarted","Data":"d19c2d3b381b9f3d8ca563a650538c88376eadb8f2e9ee8935bc342cad9f608c"}
Sep 17 22:00:01.247387 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:01.247361    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f996ae-29828040-d6qfj" event={"ID":"2ca15b6e-4aa8-4347-9556-f2f77814463d","Type":"ContainerStarted","Data":"c1ec83e01bd4e2dafd0acba19806dedeb75f9738280d556506fa9c0b715bb7b5"}
Sep 17 22:00:01.247496 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:01.247401    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f996ae-29828040-d6qfj" event={"ID":"2ca15b6e-4aa8-4347-9556-f2f77814463d","Type":"ContainerStarted","Data":"311ac1d363eee3765a1081fd19be356f99e88bc6e3fd9772784bceaa8a93ebe6"}
Sep 17 22:00:01.248861 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:01.248845    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828040-kxmw9" event={"ID":"5e74d340-4b98-4849-a48d-fdc890e79751","Type":"ContainerStarted","Data":"7f359ff8051790f7c4a2b9c9ec404ae07c6b9aa6f382fc7db8fac7181582d5fb"}
Sep 17 22:00:01.248948 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:01.248876    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828040-kxmw9" event={"ID":"5e74d340-4b98-4849-a48d-fdc890e79751","Type":"ContainerStarted","Data":"ecee3de151a6952a5da9e52cb33c5262def3cfece2a7f4d8ee7e61d975f34e5b"}
Sep 17 22:00:01.275069 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:01.275017    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a6c05-29828040-n57qk" podStartSLOduration=1.2750025329999999 podStartE2EDuration="1.275002533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:01.273696068 +0000 UTC m=+1117.092022379" watchObservedRunningTime="2026-09-17 22:00:01.275002533 +0000 UTC m=+1117.093328868"
Sep 17 22:00:01.296389 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:01.296342    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3d5c73-29828040-php7j" podStartSLOduration=1.296325975 podStartE2EDuration="1.296325975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:01.294842959 +0000 UTC m=+1117.113169274" watchObservedRunningTime="2026-09-17 22:00:01.296325975 +0000 UTC m=+1117.114652327"
Sep 17 22:00:01.316624 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:01.316571    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828040-kxmw9" podStartSLOduration=1.31655339 podStartE2EDuration="1.31655339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:01.31357203 +0000 UTC m=+1117.131898341" watchObservedRunningTime="2026-09-17 22:00:01.31655339 +0000 UTC m=+1117.134879701"
Sep 17 22:00:01.340489 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:01.340437    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f996ae-29828040-d6qfj" podStartSLOduration=1.340420636 podStartE2EDuration="1.340420636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:01.339428103 +0000 UTC m=+1117.157754416" watchObservedRunningTime="2026-09-17 22:00:01.340420636 +0000 UTC m=+1117.158746947"
Sep 17 22:00:02.255647 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:02.255611    2564 generic.go:356] "Generic (PLEG): container finished" podID="5e74d340-4b98-4849-a48d-fdc890e79751" containerID="7f359ff8051790f7c4a2b9c9ec404ae07c6b9aa6f382fc7db8fac7181582d5fb" exitCode=0
Sep 17 22:00:02.256067 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:02.255659    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828040-kxmw9" event={"ID":"5e74d340-4b98-4849-a48d-fdc890e79751","Type":"ContainerDied","Data":"7f359ff8051790f7c4a2b9c9ec404ae07c6b9aa6f382fc7db8fac7181582d5fb"}
Sep 17 22:00:02.257383 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:02.257357    2564 generic.go:356] "Generic (PLEG): container finished" podID="3e0b9d01-6e7d-43c5-bde8-ddefe4313d4f" containerID="89c7d5e15e535aa9de8f31c2f744fdfeff3f83e3b83e5d3e975d77931478407e" exitCode=0
Sep 17 22:00:02.257485 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:02.257393    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a6c05-29828040-n57qk" event={"ID":"3e0b9d01-6e7d-43c5-bde8-ddefe4313d4f","Type":"ContainerDied","Data":"89c7d5e15e535aa9de8f31c2f744fdfeff3f83e3b83e5d3e975d77931478407e"}
Sep 17 22:00:02.258970 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:02.258949    2564 generic.go:356] "Generic (PLEG): container finished" podID="c523d0e7-283a-4668-b776-59cfe4800cbf" containerID="7f8ccbc18b6a86a2679e489b9bcf68ff46e8fa0fb1cf2235b3c694f823a9afc8" exitCode=0
Sep 17 22:00:02.259078 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:02.259049    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3d5c73-29828040-php7j" event={"ID":"c523d0e7-283a-4668-b776-59cfe4800cbf","Type":"ContainerDied","Data":"7f8ccbc18b6a86a2679e489b9bcf68ff46e8fa0fb1cf2235b3c694f823a9afc8"}
Sep 17 22:00:02.260680 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:02.260659    2564 generic.go:356] "Generic (PLEG): container finished" podID="2ca15b6e-4aa8-4347-9556-f2f77814463d" containerID="c1ec83e01bd4e2dafd0acba19806dedeb75f9738280d556506fa9c0b715bb7b5" exitCode=0
Sep 17 22:00:02.260761 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:02.260690    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f996ae-29828040-d6qfj" event={"ID":"2ca15b6e-4aa8-4347-9556-f2f77814463d","Type":"ContainerDied","Data":"c1ec83e01bd4e2dafd0acba19806dedeb75f9738280d556506fa9c0b715bb7b5"}
Sep 17 22:00:03.426861 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:03.426836    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a6c05-29828040-n57qk"
Sep 17 22:00:03.483751 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:03.483716    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3d5c73-29828040-php7j"
Sep 17 22:00:03.518041 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:03.518013    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f996ae-29828040-d6qfj"
Sep 17 22:00:03.521576 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:03.521551    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828040-kxmw9"
Sep 17 22:00:04.274317 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:04.274277    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828040-kxmw9"
Sep 17 22:00:04.274317 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:04.274294    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828040-kxmw9" event={"ID":"5e74d340-4b98-4849-a48d-fdc890e79751","Type":"ContainerDied","Data":"ecee3de151a6952a5da9e52cb33c5262def3cfece2a7f4d8ee7e61d975f34e5b"}
Sep 17 22:00:04.274558 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:04.274350    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecee3de151a6952a5da9e52cb33c5262def3cfece2a7f4d8ee7e61d975f34e5b"
Sep 17 22:00:04.275679 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:04.275656    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a6c05-29828040-n57qk"
Sep 17 22:00:04.275679 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:04.275669    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a6c05-29828040-n57qk" event={"ID":"3e0b9d01-6e7d-43c5-bde8-ddefe4313d4f","Type":"ContainerDied","Data":"433a6e6fc0bf1a528d1696643bb803ea259cdd62a78a3b902b99a88e08f1fdde"}
Sep 17 22:00:04.275866 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:04.275718    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="433a6e6fc0bf1a528d1696643bb803ea259cdd62a78a3b902b99a88e08f1fdde"
Sep 17 22:00:04.277007 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:04.276992    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3d5c73-29828040-php7j"
Sep 17 22:00:04.277121 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:04.277052    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3d5c73-29828040-php7j" event={"ID":"c523d0e7-283a-4668-b776-59cfe4800cbf","Type":"ContainerDied","Data":"d19c2d3b381b9f3d8ca563a650538c88376eadb8f2e9ee8935bc342cad9f608c"}
Sep 17 22:00:04.277121 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:04.277092    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d19c2d3b381b9f3d8ca563a650538c88376eadb8f2e9ee8935bc342cad9f608c"
Sep 17 22:00:04.280990 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:04.278859    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f996ae-29828040-d6qfj" event={"ID":"2ca15b6e-4aa8-4347-9556-f2f77814463d","Type":"ContainerDied","Data":"311ac1d363eee3765a1081fd19be356f99e88bc6e3fd9772784bceaa8a93ebe6"}
Sep 17 22:00:04.280990 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:04.278983    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f996ae-29828040-d6qfj"
Sep 17 22:00:04.280990 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:04.280459    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="311ac1d363eee3765a1081fd19be356f99e88bc6e3fd9772784bceaa8a93ebe6"
Sep 17 22:00:08.230605 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:08.230566    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 22:00:08.231005 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:08.230622    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 22:00:08.243149 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:08.243127    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 22:00:08.316503 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:08.316473    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 22:00:31.232032 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:31.231993    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4bd0ef-s87nt"]
Sep 17 22:00:31.232550 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:31.232448    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e74d340-4b98-4849-a48d-fdc890e79751" containerName="cleanup"
Sep 17 22:00:31.232550 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:31.232461    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e74d340-4b98-4849-a48d-fdc890e79751" containerName="cleanup"
Sep 17 22:00:31.232550 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:31.232470    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e0b9d01-6e7d-43c5-bde8-ddefe4313d4f" containerName="cleanup"
Sep 17 22:00:31.232550 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:31.232475    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e0b9d01-6e7d-43c5-bde8-ddefe4313d4f" containerName="cleanup"
Sep 17 22:00:31.232550 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:31.232483    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c523d0e7-283a-4668-b776-59cfe4800cbf" containerName="cleanup"
Sep 17 22:00:31.232550 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:31.232489    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c523d0e7-283a-4668-b776-59cfe4800cbf" containerName="cleanup"
Sep 17 22:00:31.232550 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:31.232494    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ca15b6e-4aa8-4347-9556-f2f77814463d" containerName="cleanup"
Sep 17 22:00:31.232550 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:31.232498    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ca15b6e-4aa8-4347-9556-f2f77814463d" containerName="cleanup"
Sep 17 22:00:31.232800 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:31.232564    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c523d0e7-283a-4668-b776-59cfe4800cbf" containerName="cleanup"
Sep 17 22:00:31.232800 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:31.232574    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e74d340-4b98-4849-a48d-fdc890e79751" containerName="cleanup"
Sep 17 22:00:31.232800 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:31.232582    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e0b9d01-6e7d-43c5-bde8-ddefe4313d4f" containerName="cleanup"
Sep 17 22:00:31.232800 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:31.232587    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ca15b6e-4aa8-4347-9556-f2f77814463d" containerName="cleanup"
Sep 17 22:00:31.235883 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:31.235862    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4bd0ef-s87nt"
Sep 17 22:00:31.240304 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:31.240279    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 22:00:31.240453 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:31.240345    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmdbp\""
Sep 17 22:00:31.248991 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:31.248959    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4bd0ef-s87nt"]
Sep 17 22:00:31.278604 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:31.278570    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc15d138-31bc-4676-9994-04c0ac28a5cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4bd0ef-s87nt\" (UID: \"dc15d138-31bc-4676-9994-04c0ac28a5cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4bd0ef-s87nt"
Sep 17 22:00:31.379237 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:31.379191    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc15d138-31bc-4676-9994-04c0ac28a5cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4bd0ef-s87nt\" (UID: \"dc15d138-31bc-4676-9994-04c0ac28a5cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4bd0ef-s87nt"
Sep 17 22:00:31.379837 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:31.379817    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc15d138-31bc-4676-9994-04c0ac28a5cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4bd0ef-s87nt\" (UID: \"dc15d138-31bc-4676-9994-04c0ac28a5cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4bd0ef-s87nt"
Sep 17 22:00:31.547370 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:31.547285    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4bd0ef-s87nt"
Sep 17 22:00:31.860323 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:31.860298    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4bd0ef-s87nt"]
Sep 17 22:00:31.862117 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:00:31.862089    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc15d138_31bc_4676_9994_04c0ac28a5cc.slice/crio-d13b0a2063cfbe2a5a407d10bf239cdc80c19d268bf62ecb43ab07c09f4ada8b WatchSource:0}: Error finding container d13b0a2063cfbe2a5a407d10bf239cdc80c19d268bf62ecb43ab07c09f4ada8b: Status 404 returned error can't find the container with id d13b0a2063cfbe2a5a407d10bf239cdc80c19d268bf62ecb43ab07c09f4ada8b
Sep 17 22:00:32.445487 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:32.445448    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4bd0ef-s87nt" event={"ID":"dc15d138-31bc-4676-9994-04c0ac28a5cc","Type":"ContainerStarted","Data":"040f609f9a9b11765afb437b1cd45c42b93d9a5b68c550eecba5f54b7d96a93e"}
Sep 17 22:00:32.445870 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:32.445512    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4bd0ef-s87nt" event={"ID":"dc15d138-31bc-4676-9994-04c0ac28a5cc","Type":"ContainerStarted","Data":"d13b0a2063cfbe2a5a407d10bf239cdc80c19d268bf62ecb43ab07c09f4ada8b"}
Sep 17 22:00:32.504314 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:32.504196    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4bd0ef-s87nt" podStartSLOduration=1.5041808859999999 podStartE2EDuration="1.504180886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:32.503658794 +0000 UTC m=+1148.321985105" watchObservedRunningTime="2026-09-17 22:00:32.504180886 +0000 UTC m=+1148.322507190"
Sep 17 22:00:33.452303 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:33.452267    2564 generic.go:356] "Generic (PLEG): container finished" podID="dc15d138-31bc-4676-9994-04c0ac28a5cc" containerID="040f609f9a9b11765afb437b1cd45c42b93d9a5b68c550eecba5f54b7d96a93e" exitCode=0
Sep 17 22:00:33.452725 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:33.452360    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4bd0ef-s87nt" event={"ID":"dc15d138-31bc-4676-9994-04c0ac28a5cc","Type":"ContainerDied","Data":"040f609f9a9b11765afb437b1cd45c42b93d9a5b68c550eecba5f54b7d96a93e"}
Sep 17 22:00:34.603527 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:34.603501    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4bd0ef-s87nt"
Sep 17 22:00:34.713869 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:34.713767    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc15d138-31bc-4676-9994-04c0ac28a5cc-maas-api-service-ca\") pod \"dc15d138-31bc-4676-9994-04c0ac28a5cc\" (UID: \"dc15d138-31bc-4676-9994-04c0ac28a5cc\") "
Sep 17 22:00:34.714101 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:34.714080    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc15d138-31bc-4676-9994-04c0ac28a5cc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dc15d138-31bc-4676-9994-04c0ac28a5cc" (UID: "dc15d138-31bc-4676-9994-04c0ac28a5cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:34.815577 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:34.815533    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc15d138-31bc-4676-9994-04c0ac28a5cc-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:00:35.464916 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:35.464873    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4bd0ef-s87nt"
Sep 17 22:00:35.464916 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:35.464895    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4bd0ef-s87nt" event={"ID":"dc15d138-31bc-4676-9994-04c0ac28a5cc","Type":"ContainerDied","Data":"d13b0a2063cfbe2a5a407d10bf239cdc80c19d268bf62ecb43ab07c09f4ada8b"}
Sep 17 22:00:35.465140 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:35.464938    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d13b0a2063cfbe2a5a407d10bf239cdc80c19d268bf62ecb43ab07c09f4ada8b"
Sep 17 22:00:42.135597 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.135559    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf"]
Sep 17 22:00:42.136083 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.135803    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf" podUID="96a7ee96-4642-405d-b9f4-82111bc86046" containerName="payload-processing" containerID="cri-o://554c9d59d15b05cea3558995f220e76f6be68a1b3ddf1692f93567bdad12b75a" gracePeriod=30
Sep 17 22:00:42.143143 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.143114    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj"]
Sep 17 22:00:42.143429 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.143391    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj" podUID="6e383431-da15-4c63-a8c4-13217af0e721" containerName="payload-pre-processing" containerID="cri-o://ee7b851c001693ebbf47757f9c941fbe65be710503d4292973f83caf3b81c0dc" gracePeriod=30
Sep 17 22:00:42.427712 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.427679    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4bd0ef-s87nt"]
Sep 17 22:00:42.433165 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.433140    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4bd0ef-s87nt"]
Sep 17 22:00:42.457891 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.457862    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf"
Sep 17 22:00:42.464496 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.464461    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj"
Sep 17 22:00:42.510315 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.510279    2564 generic.go:356] "Generic (PLEG): container finished" podID="96a7ee96-4642-405d-b9f4-82111bc86046" containerID="554c9d59d15b05cea3558995f220e76f6be68a1b3ddf1692f93567bdad12b75a" exitCode=0
Sep 17 22:00:42.510497 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.510374    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf"
Sep 17 22:00:42.510497 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.510417    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf" event={"ID":"96a7ee96-4642-405d-b9f4-82111bc86046","Type":"ContainerDied","Data":"554c9d59d15b05cea3558995f220e76f6be68a1b3ddf1692f93567bdad12b75a"}
Sep 17 22:00:42.510497 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.510476    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf" event={"ID":"96a7ee96-4642-405d-b9f4-82111bc86046","Type":"ContainerDied","Data":"532e45d55962adba600a945812f87459b566952c617fac752c5827a13162d4a5"}
Sep 17 22:00:42.510678 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.510509    2564 scope.go:117] "RemoveContainer" containerID="554c9d59d15b05cea3558995f220e76f6be68a1b3ddf1692f93567bdad12b75a"
Sep 17 22:00:42.511936 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.511911    2564 generic.go:356] "Generic (PLEG): container finished" podID="6e383431-da15-4c63-a8c4-13217af0e721" containerID="ee7b851c001693ebbf47757f9c941fbe65be710503d4292973f83caf3b81c0dc" exitCode=0
Sep 17 22:00:42.512055 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.511987    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj" event={"ID":"6e383431-da15-4c63-a8c4-13217af0e721","Type":"ContainerDied","Data":"ee7b851c001693ebbf47757f9c941fbe65be710503d4292973f83caf3b81c0dc"}
Sep 17 22:00:42.512126 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.512054    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj" event={"ID":"6e383431-da15-4c63-a8c4-13217af0e721","Type":"ContainerDied","Data":"c0205869b47ccd7c9c7ea2a3306236be5cc88a07dcce3ec88ff7d7f34719eb18"}
Sep 17 22:00:42.512278 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.512256    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj"
Sep 17 22:00:42.522363 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.522343    2564 scope.go:117] "RemoveContainer" containerID="554c9d59d15b05cea3558995f220e76f6be68a1b3ddf1692f93567bdad12b75a"
Sep 17 22:00:42.522690 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:00:42.522670    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"554c9d59d15b05cea3558995f220e76f6be68a1b3ddf1692f93567bdad12b75a\": container with ID starting with 554c9d59d15b05cea3558995f220e76f6be68a1b3ddf1692f93567bdad12b75a not found: ID does not exist" containerID="554c9d59d15b05cea3558995f220e76f6be68a1b3ddf1692f93567bdad12b75a"
Sep 17 22:00:42.522781 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.522702    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"554c9d59d15b05cea3558995f220e76f6be68a1b3ddf1692f93567bdad12b75a"} err="failed to get container status \"554c9d59d15b05cea3558995f220e76f6be68a1b3ddf1692f93567bdad12b75a\": rpc error: code = NotFound desc = could not find container \"554c9d59d15b05cea3558995f220e76f6be68a1b3ddf1692f93567bdad12b75a\": container with ID starting with 554c9d59d15b05cea3558995f220e76f6be68a1b3ddf1692f93567bdad12b75a not found: ID does not exist"
Sep 17 22:00:42.522781 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.522725    2564 scope.go:117] "RemoveContainer" containerID="ee7b851c001693ebbf47757f9c941fbe65be710503d4292973f83caf3b81c0dc"
Sep 17 22:00:42.534249 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.534222    2564 scope.go:117] "RemoveContainer" containerID="ee7b851c001693ebbf47757f9c941fbe65be710503d4292973f83caf3b81c0dc"
Sep 17 22:00:42.534701 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:00:42.534681    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee7b851c001693ebbf47757f9c941fbe65be710503d4292973f83caf3b81c0dc\": container with ID starting with ee7b851c001693ebbf47757f9c941fbe65be710503d4292973f83caf3b81c0dc not found: ID does not exist" containerID="ee7b851c001693ebbf47757f9c941fbe65be710503d4292973f83caf3b81c0dc"
Sep 17 22:00:42.534771 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.534713    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee7b851c001693ebbf47757f9c941fbe65be710503d4292973f83caf3b81c0dc"} err="failed to get container status \"ee7b851c001693ebbf47757f9c941fbe65be710503d4292973f83caf3b81c0dc\": rpc error: code = NotFound desc = could not find container \"ee7b851c001693ebbf47757f9c941fbe65be710503d4292973f83caf3b81c0dc\": container with ID starting with ee7b851c001693ebbf47757f9c941fbe65be710503d4292973f83caf3b81c0dc not found: ID does not exist"
Sep 17 22:00:42.593288 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.593254    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e383431-da15-4c63-a8c4-13217af0e721-plugins-config-volume\") pod \"6e383431-da15-4c63-a8c4-13217af0e721\" (UID: \"6e383431-da15-4c63-a8c4-13217af0e721\") "
Sep 17 22:00:42.593482 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.593354    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a7ee96-4642-405d-b9f4-82111bc86046-plugins-config-volume\") pod \"96a7ee96-4642-405d-b9f4-82111bc86046\" (UID: \"96a7ee96-4642-405d-b9f4-82111bc86046\") "
Sep 17 22:00:42.593482 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.593410    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bs5d\" (UniqueName: \"kubernetes.io/projected/6e383431-da15-4c63-a8c4-13217af0e721-kube-api-access-9bs5d\") pod \"6e383431-da15-4c63-a8c4-13217af0e721\" (UID: \"6e383431-da15-4c63-a8c4-13217af0e721\") "
Sep 17 22:00:42.593482 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.593447    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qm7wq\" (UniqueName: \"kubernetes.io/projected/96a7ee96-4642-405d-b9f4-82111bc86046-kube-api-access-qm7wq\") pod \"96a7ee96-4642-405d-b9f4-82111bc86046\" (UID: \"96a7ee96-4642-405d-b9f4-82111bc86046\") "
Sep 17 22:00:42.593711 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.593685    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a7ee96-4642-405d-b9f4-82111bc86046-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "96a7ee96-4642-405d-b9f4-82111bc86046" (UID: "96a7ee96-4642-405d-b9f4-82111bc86046"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:42.593711 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.593691    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e383431-da15-4c63-a8c4-13217af0e721-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e383431-da15-4c63-a8c4-13217af0e721" (UID: "6e383431-da15-4c63-a8c4-13217af0e721"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:42.595793 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.595768    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e383431-da15-4c63-a8c4-13217af0e721-kube-api-access-9bs5d" (OuterVolumeSpecName: "kube-api-access-9bs5d") pod "6e383431-da15-4c63-a8c4-13217af0e721" (UID: "6e383431-da15-4c63-a8c4-13217af0e721"). InnerVolumeSpecName "kube-api-access-9bs5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:00:42.595892 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.595842    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a7ee96-4642-405d-b9f4-82111bc86046-kube-api-access-qm7wq" (OuterVolumeSpecName: "kube-api-access-qm7wq") pod "96a7ee96-4642-405d-b9f4-82111bc86046" (UID: "96a7ee96-4642-405d-b9f4-82111bc86046"). InnerVolumeSpecName "kube-api-access-qm7wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:00:42.695242 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.695124    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e383431-da15-4c63-a8c4-13217af0e721-plugins-config-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:00:42.695242 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.695164    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a7ee96-4642-405d-b9f4-82111bc86046-plugins-config-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:00:42.695242 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.695196    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bs5d\" (UniqueName: \"kubernetes.io/projected/6e383431-da15-4c63-a8c4-13217af0e721-kube-api-access-9bs5d\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:00:42.695242 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.695209    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm7wq\" (UniqueName: \"kubernetes.io/projected/96a7ee96-4642-405d-b9f4-82111bc86046-kube-api-access-qm7wq\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:00:42.754947 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.754908    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc15d138-31bc-4676-9994-04c0ac28a5cc" path="/var/lib/kubelet/pods/dc15d138-31bc-4676-9994-04c0ac28a5cc/volumes"
Sep 17 22:00:42.851897 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.851708    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj"]
Sep 17 22:00:42.858827 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.858791    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4bd0ef-769b59c79d-nhdmj"]
Sep 17 22:00:42.890840 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.890800    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf"]
Sep 17 22:00:42.911257 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:42.911204    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4bd0ef-5db6d44945-cb9wf"]
Sep 17 22:00:44.754404 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:44.754371    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e383431-da15-4c63-a8c4-13217af0e721" path="/var/lib/kubelet/pods/6e383431-da15-4c63-a8c4-13217af0e721/volumes"
Sep 17 22:00:44.754799 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:44.754737    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96a7ee96-4642-405d-b9f4-82111bc86046" path="/var/lib/kubelet/pods/96a7ee96-4642-405d-b9f4-82111bc86046/volumes"
Sep 17 22:00:47.806097 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:47.806014    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"]
Sep 17 22:00:50.416214 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:50.416163    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2"]
Sep 17 22:00:50.416631 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:50.416618    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e383431-da15-4c63-a8c4-13217af0e721" containerName="payload-pre-processing"
Sep 17 22:00:50.416674 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:50.416634    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e383431-da15-4c63-a8c4-13217af0e721" containerName="payload-pre-processing"
Sep 17 22:00:50.416674 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:50.416661    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96a7ee96-4642-405d-b9f4-82111bc86046" containerName="payload-processing"
Sep 17 22:00:50.416674 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:50.416666    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="96a7ee96-4642-405d-b9f4-82111bc86046" containerName="payload-processing"
Sep 17 22:00:50.416674 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:50.416672    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc15d138-31bc-4676-9994-04c0ac28a5cc" containerName="revoke-keys"
Sep 17 22:00:50.416810 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:50.416678    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc15d138-31bc-4676-9994-04c0ac28a5cc" containerName="revoke-keys"
Sep 17 22:00:50.416810 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:50.416748    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc15d138-31bc-4676-9994-04c0ac28a5cc" containerName="revoke-keys"
Sep 17 22:00:50.416810 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:50.416758    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e383431-da15-4c63-a8c4-13217af0e721" containerName="payload-pre-processing"
Sep 17 22:00:50.416810 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:50.416770    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="96a7ee96-4642-405d-b9f4-82111bc86046" containerName="payload-processing"
Sep 17 22:00:50.421479 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:50.421463    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2"
Sep 17 22:00:50.427363 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:50.427342    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-93af11bc\""
Sep 17 22:00:50.428445 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:50.428205    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-93af11bc-dockercfg-pllft\""
Sep 17 22:00:50.449579 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:50.449548    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2"]
Sep 17 22:00:50.575414 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:50.575377    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqvkq\" (UniqueName: \"kubernetes.io/projected/86fe5331-23a1-4082-99ad-ef55df5feae5-kube-api-access-dqvkq\") pod \"payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2\" (UID: \"86fe5331-23a1-4082-99ad-ef55df5feae5\") " pod="openshift-ingress/payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2"
Sep 17 22:00:50.575599 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:50.575435    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86fe5331-23a1-4082-99ad-ef55df5feae5-plugins-config-volume\") pod \"payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2\" (UID: \"86fe5331-23a1-4082-99ad-ef55df5feae5\") " pod="openshift-ingress/payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2"
Sep 17 22:00:50.677046 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:50.676945    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqvkq\" (UniqueName: \"kubernetes.io/projected/86fe5331-23a1-4082-99ad-ef55df5feae5-kube-api-access-dqvkq\") pod \"payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2\" (UID: \"86fe5331-23a1-4082-99ad-ef55df5feae5\") " pod="openshift-ingress/payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2"
Sep 17 22:00:50.677046 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:50.677013    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86fe5331-23a1-4082-99ad-ef55df5feae5-plugins-config-volume\") pod \"payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2\" (UID: \"86fe5331-23a1-4082-99ad-ef55df5feae5\") " pod="openshift-ingress/payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2"
Sep 17 22:00:50.677799 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:50.677777    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86fe5331-23a1-4082-99ad-ef55df5feae5-plugins-config-volume\") pod \"payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2\" (UID: \"86fe5331-23a1-4082-99ad-ef55df5feae5\") " pod="openshift-ingress/payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2"
Sep 17 22:00:50.691822 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:50.691789    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqvkq\" (UniqueName: \"kubernetes.io/projected/86fe5331-23a1-4082-99ad-ef55df5feae5-kube-api-access-dqvkq\") pod \"payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2\" (UID: \"86fe5331-23a1-4082-99ad-ef55df5feae5\") " pod="openshift-ingress/payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2"
Sep 17 22:00:50.732207 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:50.732137    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2"
Sep 17 22:00:51.020885 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:51.020800    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2"]
Sep 17 22:00:51.027838 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:00:51.027815    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86fe5331_23a1_4082_99ad_ef55df5feae5.slice/crio-377347ceb29a299b05f5aa65a7a77615135c957db80bf840f39150756511e098 WatchSource:0}: Error finding container 377347ceb29a299b05f5aa65a7a77615135c957db80bf840f39150756511e098: Status 404 returned error can't find the container with id 377347ceb29a299b05f5aa65a7a77615135c957db80bf840f39150756511e098
Sep 17 22:00:51.570273 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:51.570232    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2" event={"ID":"86fe5331-23a1-4082-99ad-ef55df5feae5","Type":"ContainerStarted","Data":"3790ead95fb2b47ba640a4b9c65e4b89064f7101c99a72b42d9c854a676cb32d"}
Sep 17 22:00:51.570727 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:51.570299    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2" event={"ID":"86fe5331-23a1-4082-99ad-ef55df5feae5","Type":"ContainerStarted","Data":"377347ceb29a299b05f5aa65a7a77615135c957db80bf840f39150756511e098"}
Sep 17 22:00:51.570727 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:51.570429    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2"
Sep 17 22:00:51.591417 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:00:51.591360    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2" podStartSLOduration=1.5913437830000001 podStartE2EDuration="1.591343783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:51.588966769 +0000 UTC m=+1167.407293083" watchObservedRunningTime="2026-09-17 22:00:51.591343783 +0000 UTC m=+1167.409670086"
Sep 17 22:01:02.578714 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:02.578684    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2"
Sep 17 22:01:02.827543 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:02.826163    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq" podUID="6284ed91-64e6-4acf-b78f-a1741bc6979b" containerName="main" containerID="cri-o://8198205b3275b23a94ccb720c794be035fc8cf9c7a6166ccf9aafc3191059203" gracePeriod=45
Sep 17 22:01:03.091489 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.091419    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 22:01:03.206779 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.206748    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6284ed91-64e6-4acf-b78f-a1741bc6979b-dshm\") pod \"6284ed91-64e6-4acf-b78f-a1741bc6979b\" (UID: \"6284ed91-64e6-4acf-b78f-a1741bc6979b\") "
Sep 17 22:01:03.206969 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.206797    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6284ed91-64e6-4acf-b78f-a1741bc6979b-model-cache\") pod \"6284ed91-64e6-4acf-b78f-a1741bc6979b\" (UID: \"6284ed91-64e6-4acf-b78f-a1741bc6979b\") "
Sep 17 22:01:03.206969 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.206850    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rvfm\" (UniqueName: \"kubernetes.io/projected/6284ed91-64e6-4acf-b78f-a1741bc6979b-kube-api-access-5rvfm\") pod \"6284ed91-64e6-4acf-b78f-a1741bc6979b\" (UID: \"6284ed91-64e6-4acf-b78f-a1741bc6979b\") "
Sep 17 22:01:03.206969 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.206874    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6284ed91-64e6-4acf-b78f-a1741bc6979b-home\") pod \"6284ed91-64e6-4acf-b78f-a1741bc6979b\" (UID: \"6284ed91-64e6-4acf-b78f-a1741bc6979b\") "
Sep 17 22:01:03.206969 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.206905    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6284ed91-64e6-4acf-b78f-a1741bc6979b-tls-certs\") pod \"6284ed91-64e6-4acf-b78f-a1741bc6979b\" (UID: \"6284ed91-64e6-4acf-b78f-a1741bc6979b\") "
Sep 17 22:01:03.206969 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.206958    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6284ed91-64e6-4acf-b78f-a1741bc6979b-tmp-dir\") pod \"6284ed91-64e6-4acf-b78f-a1741bc6979b\" (UID: \"6284ed91-64e6-4acf-b78f-a1741bc6979b\") "
Sep 17 22:01:03.207277 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.207115    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6284ed91-64e6-4acf-b78f-a1741bc6979b-model-cache" (OuterVolumeSpecName: "model-cache") pod "6284ed91-64e6-4acf-b78f-a1741bc6979b" (UID: "6284ed91-64e6-4acf-b78f-a1741bc6979b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:01:03.207277 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.207135    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6284ed91-64e6-4acf-b78f-a1741bc6979b-home" (OuterVolumeSpecName: "home") pod "6284ed91-64e6-4acf-b78f-a1741bc6979b" (UID: "6284ed91-64e6-4acf-b78f-a1741bc6979b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:01:03.207369 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.207325    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6284ed91-64e6-4acf-b78f-a1741bc6979b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6284ed91-64e6-4acf-b78f-a1741bc6979b" (UID: "6284ed91-64e6-4acf-b78f-a1741bc6979b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:01:03.207369 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.207354    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6284ed91-64e6-4acf-b78f-a1741bc6979b-model-cache\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:01:03.207476 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.207372    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6284ed91-64e6-4acf-b78f-a1741bc6979b-home\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:01:03.209299 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.209272    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6284ed91-64e6-4acf-b78f-a1741bc6979b-dshm" (OuterVolumeSpecName: "dshm") pod "6284ed91-64e6-4acf-b78f-a1741bc6979b" (UID: "6284ed91-64e6-4acf-b78f-a1741bc6979b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:01:03.209415 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.209307    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6284ed91-64e6-4acf-b78f-a1741bc6979b-kube-api-access-5rvfm" (OuterVolumeSpecName: "kube-api-access-5rvfm") pod "6284ed91-64e6-4acf-b78f-a1741bc6979b" (UID: "6284ed91-64e6-4acf-b78f-a1741bc6979b"). InnerVolumeSpecName "kube-api-access-5rvfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:01:03.209415 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.209380    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6284ed91-64e6-4acf-b78f-a1741bc6979b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6284ed91-64e6-4acf-b78f-a1741bc6979b" (UID: "6284ed91-64e6-4acf-b78f-a1741bc6979b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:01:03.308031 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.307994    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rvfm\" (UniqueName: \"kubernetes.io/projected/6284ed91-64e6-4acf-b78f-a1741bc6979b-kube-api-access-5rvfm\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:01:03.308031 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.308025    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6284ed91-64e6-4acf-b78f-a1741bc6979b-tls-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:01:03.308031 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.308035    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6284ed91-64e6-4acf-b78f-a1741bc6979b-tmp-dir\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:01:03.308031 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.308043    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6284ed91-64e6-4acf-b78f-a1741bc6979b-dshm\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:01:03.643304 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.643269    2564 generic.go:356] "Generic (PLEG): container finished" podID="6284ed91-64e6-4acf-b78f-a1741bc6979b" containerID="8198205b3275b23a94ccb720c794be035fc8cf9c7a6166ccf9aafc3191059203" exitCode=0
Sep 17 22:01:03.643693 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.643316    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"
Sep 17 22:01:03.643693 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.643349    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq" event={"ID":"6284ed91-64e6-4acf-b78f-a1741bc6979b","Type":"ContainerDied","Data":"8198205b3275b23a94ccb720c794be035fc8cf9c7a6166ccf9aafc3191059203"}
Sep 17 22:01:03.643693 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.643393    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq" event={"ID":"6284ed91-64e6-4acf-b78f-a1741bc6979b","Type":"ContainerDied","Data":"dc9c1d386425fa43420b210790844b9768099dcc63a96f5c014f422e57e00ea1"}
Sep 17 22:01:03.643693 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.643411    2564 scope.go:117] "RemoveContainer" containerID="8198205b3275b23a94ccb720c794be035fc8cf9c7a6166ccf9aafc3191059203"
Sep 17 22:01:03.653853 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.653834    2564 scope.go:117] "RemoveContainer" containerID="8198205b3275b23a94ccb720c794be035fc8cf9c7a6166ccf9aafc3191059203"
Sep 17 22:01:03.654203 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:01:03.654162    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8198205b3275b23a94ccb720c794be035fc8cf9c7a6166ccf9aafc3191059203\": container with ID starting with 8198205b3275b23a94ccb720c794be035fc8cf9c7a6166ccf9aafc3191059203 not found: ID does not exist" containerID="8198205b3275b23a94ccb720c794be035fc8cf9c7a6166ccf9aafc3191059203"
Sep 17 22:01:03.654314 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.654210    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8198205b3275b23a94ccb720c794be035fc8cf9c7a6166ccf9aafc3191059203"} err="failed to get container status \"8198205b3275b23a94ccb720c794be035fc8cf9c7a6166ccf9aafc3191059203\": rpc error: code = NotFound desc = could not find container \"8198205b3275b23a94ccb720c794be035fc8cf9c7a6166ccf9aafc3191059203\": container with ID starting with 8198205b3275b23a94ccb720c794be035fc8cf9c7a6166ccf9aafc3191059203 not found: ID does not exist"
Sep 17 22:01:03.672044 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.671984    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"]
Sep 17 22:01:03.674793 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:03.674768    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-4bd0ef/explicit-model-4bd0ef-kserve-6dfb75b9d7-rhldq"]
Sep 17 22:01:04.760455 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:04.760423    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6284ed91-64e6-4acf-b78f-a1741bc6979b" path="/var/lib/kubelet/pods/6284ed91-64e6-4acf-b78f-a1741bc6979b/volumes"
Sep 17 22:01:23.208101 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:23.208063    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93af11bc-4mv6l"]
Sep 17 22:01:23.210418 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:23.208531    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6284ed91-64e6-4acf-b78f-a1741bc6979b" containerName="main"
Sep 17 22:01:23.210418 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:23.208544    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6284ed91-64e6-4acf-b78f-a1741bc6979b" containerName="main"
Sep 17 22:01:23.210418 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:23.208610    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6284ed91-64e6-4acf-b78f-a1741bc6979b" containerName="main"
Sep 17 22:01:23.211541 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:23.211525    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93af11bc-4mv6l"
Sep 17 22:01:23.214381 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:23.214353    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 22:01:23.214528 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:23.214512    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmdbp\""
Sep 17 22:01:23.227764 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:23.227733    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93af11bc-4mv6l"]
Sep 17 22:01:23.307915 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:23.307879    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4309da46-e5ce-4bd4-bc0d-7fa6767f847c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-93af11bc-4mv6l\" (UID: \"4309da46-e5ce-4bd4-bc0d-7fa6767f847c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93af11bc-4mv6l"
Sep 17 22:01:23.408954 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:23.408895    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4309da46-e5ce-4bd4-bc0d-7fa6767f847c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-93af11bc-4mv6l\" (UID: \"4309da46-e5ce-4bd4-bc0d-7fa6767f847c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93af11bc-4mv6l"
Sep 17 22:01:23.409577 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:23.409555    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4309da46-e5ce-4bd4-bc0d-7fa6767f847c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-93af11bc-4mv6l\" (UID: \"4309da46-e5ce-4bd4-bc0d-7fa6767f847c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93af11bc-4mv6l"
Sep 17 22:01:23.521621 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:23.521536    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93af11bc-4mv6l"
Sep 17 22:01:23.808836 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:23.808803    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93af11bc-4mv6l"]
Sep 17 22:01:23.810563 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:01:23.810537    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4309da46_e5ce_4bd4_bc0d_7fa6767f847c.slice/crio-5769c96d026b7b9f368daafb45c9c5bb8d042977d376f57a364336a93e51c109 WatchSource:0}: Error finding container 5769c96d026b7b9f368daafb45c9c5bb8d042977d376f57a364336a93e51c109: Status 404 returned error can't find the container with id 5769c96d026b7b9f368daafb45c9c5bb8d042977d376f57a364336a93e51c109
Sep 17 22:01:24.720352 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:24.720325    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdcnw_6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2/ovn-acl-logging/0.log"
Sep 17 22:01:24.722975 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:24.722952    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdcnw_6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2/ovn-acl-logging/0.log"
Sep 17 22:01:24.782049 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:24.782008    2564 generic.go:356] "Generic (PLEG): container finished" podID="4309da46-e5ce-4bd4-bc0d-7fa6767f847c" containerID="83673a4ad1bd3a298cac7b606483202cffd86e97aff57ee2961e52c5ddd8e23e" exitCode=0
Sep 17 22:01:24.782164 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:24.782093    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93af11bc-4mv6l" event={"ID":"4309da46-e5ce-4bd4-bc0d-7fa6767f847c","Type":"ContainerDied","Data":"83673a4ad1bd3a298cac7b606483202cffd86e97aff57ee2961e52c5ddd8e23e"}
Sep 17 22:01:24.782253 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:24.782214    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93af11bc-4mv6l" event={"ID":"4309da46-e5ce-4bd4-bc0d-7fa6767f847c","Type":"ContainerStarted","Data":"5769c96d026b7b9f368daafb45c9c5bb8d042977d376f57a364336a93e51c109"}
Sep 17 22:01:25.932064 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:25.932036    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93af11bc-4mv6l"
Sep 17 22:01:26.030641 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:26.030609    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4309da46-e5ce-4bd4-bc0d-7fa6767f847c-maas-api-service-ca\") pod \"4309da46-e5ce-4bd4-bc0d-7fa6767f847c\" (UID: \"4309da46-e5ce-4bd4-bc0d-7fa6767f847c\") "
Sep 17 22:01:26.031003 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:26.030979    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4309da46-e5ce-4bd4-bc0d-7fa6767f847c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4309da46-e5ce-4bd4-bc0d-7fa6767f847c" (UID: "4309da46-e5ce-4bd4-bc0d-7fa6767f847c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:01:26.131816 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:26.131726    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4309da46-e5ce-4bd4-bc0d-7fa6767f847c-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:01:26.799034 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:26.798984    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93af11bc-4mv6l" event={"ID":"4309da46-e5ce-4bd4-bc0d-7fa6767f847c","Type":"ContainerDied","Data":"5769c96d026b7b9f368daafb45c9c5bb8d042977d376f57a364336a93e51c109"}
Sep 17 22:01:26.799276 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:26.799057    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5769c96d026b7b9f368daafb45c9c5bb8d042977d376f57a364336a93e51c109"
Sep 17 22:01:26.799276 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:26.799094    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93af11bc-4mv6l"
Sep 17 22:01:34.052232 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:34.052157    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2"]
Sep 17 22:01:34.052762 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:34.052462    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2" podUID="86fe5331-23a1-4082-99ad-ef55df5feae5" containerName="payload-processing" containerID="cri-o://3790ead95fb2b47ba640a4b9c65e4b89064f7101c99a72b42d9c854a676cb32d" gracePeriod=30
Sep 17 22:01:34.321971 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:34.321949    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2"
Sep 17 22:01:34.409763 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:34.409727    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqvkq\" (UniqueName: \"kubernetes.io/projected/86fe5331-23a1-4082-99ad-ef55df5feae5-kube-api-access-dqvkq\") pod \"86fe5331-23a1-4082-99ad-ef55df5feae5\" (UID: \"86fe5331-23a1-4082-99ad-ef55df5feae5\") "
Sep 17 22:01:34.409932 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:34.409830    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86fe5331-23a1-4082-99ad-ef55df5feae5-plugins-config-volume\") pod \"86fe5331-23a1-4082-99ad-ef55df5feae5\" (UID: \"86fe5331-23a1-4082-99ad-ef55df5feae5\") "
Sep 17 22:01:34.410253 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:34.410222    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86fe5331-23a1-4082-99ad-ef55df5feae5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "86fe5331-23a1-4082-99ad-ef55df5feae5" (UID: "86fe5331-23a1-4082-99ad-ef55df5feae5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:01:34.412031 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:34.411995    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86fe5331-23a1-4082-99ad-ef55df5feae5-kube-api-access-dqvkq" (OuterVolumeSpecName: "kube-api-access-dqvkq") pod "86fe5331-23a1-4082-99ad-ef55df5feae5" (UID: "86fe5331-23a1-4082-99ad-ef55df5feae5"). InnerVolumeSpecName "kube-api-access-dqvkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:01:34.416100 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:34.416073    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93af11bc-4mv6l"]
Sep 17 22:01:34.421213 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:34.421189    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93af11bc-4mv6l"]
Sep 17 22:01:34.511220 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:34.511151    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86fe5331-23a1-4082-99ad-ef55df5feae5-plugins-config-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:01:34.511220 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:34.511214    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqvkq\" (UniqueName: \"kubernetes.io/projected/86fe5331-23a1-4082-99ad-ef55df5feae5-kube-api-access-dqvkq\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:01:34.757198 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:34.756110    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4309da46-e5ce-4bd4-bc0d-7fa6767f847c" path="/var/lib/kubelet/pods/4309da46-e5ce-4bd4-bc0d-7fa6767f847c/volumes"
Sep 17 22:01:34.845684 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:34.845649    2564 generic.go:356] "Generic (PLEG): container finished" podID="86fe5331-23a1-4082-99ad-ef55df5feae5" containerID="3790ead95fb2b47ba640a4b9c65e4b89064f7101c99a72b42d9c854a676cb32d" exitCode=0
Sep 17 22:01:34.845865 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:34.845712    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2" event={"ID":"86fe5331-23a1-4082-99ad-ef55df5feae5","Type":"ContainerDied","Data":"3790ead95fb2b47ba640a4b9c65e4b89064f7101c99a72b42d9c854a676cb32d"}
Sep 17 22:01:34.845865 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:34.845756    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2" event={"ID":"86fe5331-23a1-4082-99ad-ef55df5feae5","Type":"ContainerDied","Data":"377347ceb29a299b05f5aa65a7a77615135c957db80bf840f39150756511e098"}
Sep 17 22:01:34.845865 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:34.845758    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2"
Sep 17 22:01:34.845865 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:34.845776    2564 scope.go:117] "RemoveContainer" containerID="3790ead95fb2b47ba640a4b9c65e4b89064f7101c99a72b42d9c854a676cb32d"
Sep 17 22:01:34.855514 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:34.855491    2564 scope.go:117] "RemoveContainer" containerID="3790ead95fb2b47ba640a4b9c65e4b89064f7101c99a72b42d9c854a676cb32d"
Sep 17 22:01:34.855806 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:01:34.855786    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3790ead95fb2b47ba640a4b9c65e4b89064f7101c99a72b42d9c854a676cb32d\": container with ID starting with 3790ead95fb2b47ba640a4b9c65e4b89064f7101c99a72b42d9c854a676cb32d not found: ID does not exist" containerID="3790ead95fb2b47ba640a4b9c65e4b89064f7101c99a72b42d9c854a676cb32d"
Sep 17 22:01:34.855866 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:34.855814    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3790ead95fb2b47ba640a4b9c65e4b89064f7101c99a72b42d9c854a676cb32d"} err="failed to get container status \"3790ead95fb2b47ba640a4b9c65e4b89064f7101c99a72b42d9c854a676cb32d\": rpc error: code = NotFound desc = could not find container \"3790ead95fb2b47ba640a4b9c65e4b89064f7101c99a72b42d9c854a676cb32d\": container with ID starting with 3790ead95fb2b47ba640a4b9c65e4b89064f7101c99a72b42d9c854a676cb32d not found: ID does not exist"
Sep 17 22:01:34.881313 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:34.881256    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2"]
Sep 17 22:01:34.883729 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:34.883704    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-93af11bc-588fddb4d9-jv2h2"]
Sep 17 22:01:36.754590 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:01:36.754553    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86fe5331-23a1-4082-99ad-ef55df5feae5" path="/var/lib/kubelet/pods/86fe5331-23a1-4082-99ad-ef55df5feae5/volumes"
Sep 17 22:02:05.527668 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:05.527618    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz"]
Sep 17 22:02:05.528236 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:05.528056    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4309da46-e5ce-4bd4-bc0d-7fa6767f847c" containerName="revoke-keys"
Sep 17 22:02:05.528236 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:05.528067    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4309da46-e5ce-4bd4-bc0d-7fa6767f847c" containerName="revoke-keys"
Sep 17 22:02:05.528236 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:05.528092    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86fe5331-23a1-4082-99ad-ef55df5feae5" containerName="payload-processing"
Sep 17 22:02:05.528236 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:05.528097    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="86fe5331-23a1-4082-99ad-ef55df5feae5" containerName="payload-processing"
Sep 17 22:02:05.528236 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:05.528157    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4309da46-e5ce-4bd4-bc0d-7fa6767f847c" containerName="revoke-keys"
Sep 17 22:02:05.528236 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:05.528166    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="86fe5331-23a1-4082-99ad-ef55df5feae5" containerName="payload-processing"
Sep 17 22:02:05.531237 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:05.531218    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz"
Sep 17 22:02:05.534417 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:05.534394    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-a23afc-dockercfg-x7qlf\""
Sep 17 22:02:05.534842 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:05.534816    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-a23afc\""
Sep 17 22:02:05.548367 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:05.548340    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz"]
Sep 17 22:02:05.627064 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:05.627030    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5835d2b2-3580-4255-8ccc-27812592b696-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz\" (UID: \"5835d2b2-3580-4255-8ccc-27812592b696\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz"
Sep 17 22:02:05.627285 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:05.627095    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-756mk\" (UniqueName: \"kubernetes.io/projected/5835d2b2-3580-4255-8ccc-27812592b696-kube-api-access-756mk\") pod \"payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz\" (UID: \"5835d2b2-3580-4255-8ccc-27812592b696\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz"
Sep 17 22:02:05.728450 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:05.728407    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-756mk\" (UniqueName: \"kubernetes.io/projected/5835d2b2-3580-4255-8ccc-27812592b696-kube-api-access-756mk\") pod \"payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz\" (UID: \"5835d2b2-3580-4255-8ccc-27812592b696\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz"
Sep 17 22:02:05.728629 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:05.728533    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5835d2b2-3580-4255-8ccc-27812592b696-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz\" (UID: \"5835d2b2-3580-4255-8ccc-27812592b696\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz"
Sep 17 22:02:05.729306 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:05.729285    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5835d2b2-3580-4255-8ccc-27812592b696-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz\" (UID: \"5835d2b2-3580-4255-8ccc-27812592b696\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz"
Sep 17 22:02:05.741302 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:05.741275    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-756mk\" (UniqueName: \"kubernetes.io/projected/5835d2b2-3580-4255-8ccc-27812592b696-kube-api-access-756mk\") pod \"payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz\" (UID: \"5835d2b2-3580-4255-8ccc-27812592b696\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz"
Sep 17 22:02:05.842206 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:05.842092    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz"
Sep 17 22:02:06.139867 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:06.139840    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz"]
Sep 17 22:02:06.141624 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:02:06.141597    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5835d2b2_3580_4255_8ccc_27812592b696.slice/crio-d1f51e0e246978078513bf6a8624a37c30292a66450101f33b3bf95cbcd3947b WatchSource:0}: Error finding container d1f51e0e246978078513bf6a8624a37c30292a66450101f33b3bf95cbcd3947b: Status 404 returned error can't find the container with id d1f51e0e246978078513bf6a8624a37c30292a66450101f33b3bf95cbcd3947b
Sep 17 22:02:07.038973 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:07.038936    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz" event={"ID":"5835d2b2-3580-4255-8ccc-27812592b696","Type":"ContainerStarted","Data":"287e244af6a6fba15e742e50ae3f76858f66a2900b6ada7accca33131e92b93c"}
Sep 17 22:02:07.039404 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:07.038990    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz" event={"ID":"5835d2b2-3580-4255-8ccc-27812592b696","Type":"ContainerStarted","Data":"d1f51e0e246978078513bf6a8624a37c30292a66450101f33b3bf95cbcd3947b"}
Sep 17 22:02:07.039404 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:07.039079    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz"
Sep 17 22:02:07.063411 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:07.063353    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz" podStartSLOduration=2.063336762 podStartE2EDuration="2.063336762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:02:07.062021654 +0000 UTC m=+1242.880347967" watchObservedRunningTime="2026-09-17 22:02:07.063336762 +0000 UTC m=+1242.881663107"
Sep 17 22:02:18.046904 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:18.046822    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz"
Sep 17 22:02:22.378162 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:22.378119    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"]
Sep 17 22:02:22.381819 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:22.381793    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:22.387020 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:02:22.386985    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-k66sq\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-noten-2d1bcf\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-noten-2d1bcf\"/\"default-dockercfg-k66sq\"" type="*v1.Secret"
Sep 17 22:02:22.387205 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:02:22.387020    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"noten-model-2d1bcf-kserve-59ff6798bc-mkcc8\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"ai-tenant-e2e-noten-2d1bcf\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="17b98d06-7b3f-4910-971c-7acbc508ada8" pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:22.387205 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:02:22.387049    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"ai-tenant-e2e-noten-2d1bcf\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-noten-2d1bcf\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 17 22:02:22.387334 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:02:22.387233    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"noten-model-2d1bcf-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-noten-2d1bcf\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-noten-2d1bcf\"/\"noten-model-2d1bcf-kserve-self-signed-certs\"" type="*v1.Secret"
Sep 17 22:02:22.388330 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:02:22.388301    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"ai-tenant-e2e-noten-2d1bcf\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-noten-2d1bcf\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 17 22:02:22.411712 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:22.411683    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"]
Sep 17 22:02:22.496983 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:22.496946    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b98d06-7b3f-4910-971c-7acbc508ada8-tmp-dir\") pod \"noten-model-2d1bcf-kserve-59ff6798bc-mkcc8\" (UID: \"17b98d06-7b3f-4910-971c-7acbc508ada8\") " pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:22.497165 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:22.497007    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfcjp\" (UniqueName: \"kubernetes.io/projected/17b98d06-7b3f-4910-971c-7acbc508ada8-kube-api-access-wfcjp\") pod \"noten-model-2d1bcf-kserve-59ff6798bc-mkcc8\" (UID: \"17b98d06-7b3f-4910-971c-7acbc508ada8\") " pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:22.497165 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:22.497115    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b98d06-7b3f-4910-971c-7acbc508ada8-dshm\") pod \"noten-model-2d1bcf-kserve-59ff6798bc-mkcc8\" (UID: \"17b98d06-7b3f-4910-971c-7acbc508ada8\") " pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:22.497165 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:22.497151    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b98d06-7b3f-4910-971c-7acbc508ada8-home\") pod \"noten-model-2d1bcf-kserve-59ff6798bc-mkcc8\" (UID: \"17b98d06-7b3f-4910-971c-7acbc508ada8\") " pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:22.497330 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:22.497261    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b98d06-7b3f-4910-971c-7acbc508ada8-model-cache\") pod \"noten-model-2d1bcf-kserve-59ff6798bc-mkcc8\" (UID: \"17b98d06-7b3f-4910-971c-7acbc508ada8\") " pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:22.497330 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:22.497315    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b98d06-7b3f-4910-971c-7acbc508ada8-tls-certs\") pod \"noten-model-2d1bcf-kserve-59ff6798bc-mkcc8\" (UID: \"17b98d06-7b3f-4910-971c-7acbc508ada8\") " pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:22.598774 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:22.598733    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b98d06-7b3f-4910-971c-7acbc508ada8-dshm\") pod \"noten-model-2d1bcf-kserve-59ff6798bc-mkcc8\" (UID: \"17b98d06-7b3f-4910-971c-7acbc508ada8\") " pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:22.598774 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:22.598779    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b98d06-7b3f-4910-971c-7acbc508ada8-home\") pod \"noten-model-2d1bcf-kserve-59ff6798bc-mkcc8\" (UID: \"17b98d06-7b3f-4910-971c-7acbc508ada8\") " pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:22.599094 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:22.598831    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b98d06-7b3f-4910-971c-7acbc508ada8-model-cache\") pod \"noten-model-2d1bcf-kserve-59ff6798bc-mkcc8\" (UID: \"17b98d06-7b3f-4910-971c-7acbc508ada8\") " pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:22.599094 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:22.598872    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b98d06-7b3f-4910-971c-7acbc508ada8-tls-certs\") pod \"noten-model-2d1bcf-kserve-59ff6798bc-mkcc8\" (UID: \"17b98d06-7b3f-4910-971c-7acbc508ada8\") " pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:22.599094 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:22.598930    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b98d06-7b3f-4910-971c-7acbc508ada8-tmp-dir\") pod \"noten-model-2d1bcf-kserve-59ff6798bc-mkcc8\" (UID: \"17b98d06-7b3f-4910-971c-7acbc508ada8\") " pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:22.599094 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:22.598957    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfcjp\" (UniqueName: \"kubernetes.io/projected/17b98d06-7b3f-4910-971c-7acbc508ada8-kube-api-access-wfcjp\") pod \"noten-model-2d1bcf-kserve-59ff6798bc-mkcc8\" (UID: \"17b98d06-7b3f-4910-971c-7acbc508ada8\") " pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:22.599340 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:22.599302    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b98d06-7b3f-4910-971c-7acbc508ada8-home\") pod \"noten-model-2d1bcf-kserve-59ff6798bc-mkcc8\" (UID: \"17b98d06-7b3f-4910-971c-7acbc508ada8\") " pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:22.599396 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:22.599354    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b98d06-7b3f-4910-971c-7acbc508ada8-model-cache\") pod \"noten-model-2d1bcf-kserve-59ff6798bc-mkcc8\" (UID: \"17b98d06-7b3f-4910-971c-7acbc508ada8\") " pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:22.599396 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:22.599359    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b98d06-7b3f-4910-971c-7acbc508ada8-tmp-dir\") pod \"noten-model-2d1bcf-kserve-59ff6798bc-mkcc8\" (UID: \"17b98d06-7b3f-4910-971c-7acbc508ada8\") " pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:22.601163 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:22.601141    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b98d06-7b3f-4910-971c-7acbc508ada8-dshm\") pod \"noten-model-2d1bcf-kserve-59ff6798bc-mkcc8\" (UID: \"17b98d06-7b3f-4910-971c-7acbc508ada8\") " pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:23.473385 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:23.473352    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-2d1bcf\"/\"kube-root-ca.crt\""
Sep 17 22:02:23.531364 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:23.531317    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-2d1bcf\"/\"default-dockercfg-k66sq\""
Sep 17 22:02:23.532342 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:23.532324    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-2d1bcf\"/\"noten-model-2d1bcf-kserve-self-signed-certs\""
Sep 17 22:02:23.542092 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:23.542072    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b98d06-7b3f-4910-971c-7acbc508ada8-tls-certs\") pod \"noten-model-2d1bcf-kserve-59ff6798bc-mkcc8\" (UID: \"17b98d06-7b3f-4910-971c-7acbc508ada8\") " pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:23.579373 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:23.579339    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-2d1bcf\"/\"openshift-service-ca.crt\""
Sep 17 22:02:23.590610 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:23.590577    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfcjp\" (UniqueName: \"kubernetes.io/projected/17b98d06-7b3f-4910-971c-7acbc508ada8-kube-api-access-wfcjp\") pod \"noten-model-2d1bcf-kserve-59ff6798bc-mkcc8\" (UID: \"17b98d06-7b3f-4910-971c-7acbc508ada8\") " pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:23.592301 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:23.592270    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:23.899011 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:23.898987    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"]
Sep 17 22:02:23.900658 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:02:23.900628    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17b98d06_7b3f_4910_971c_7acbc508ada8.slice/crio-bc013374046d1a3d495f78cc00e36fce5f3f74683b6be21751432c430b5ecc29 WatchSource:0}: Error finding container bc013374046d1a3d495f78cc00e36fce5f3f74683b6be21751432c430b5ecc29: Status 404 returned error can't find the container with id bc013374046d1a3d495f78cc00e36fce5f3f74683b6be21751432c430b5ecc29
Sep 17 22:02:24.143095 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:24.143058    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8" event={"ID":"17b98d06-7b3f-4910-971c-7acbc508ada8","Type":"ContainerStarted","Data":"f3591fa42854aa99a30cb3da760d9e0425c9bcc4fdc478035a82971b4debecf3"}
Sep 17 22:02:24.143095 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:24.143101    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8" event={"ID":"17b98d06-7b3f-4910-971c-7acbc508ada8","Type":"ContainerStarted","Data":"bc013374046d1a3d495f78cc00e36fce5f3f74683b6be21751432c430b5ecc29"}
Sep 17 22:02:24.184874 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:24.184776    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8" podStartSLOduration=2.184760748 podStartE2EDuration="2.184760748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:02:24.184438944 +0000 UTC m=+1260.002765257" watchObservedRunningTime="2026-09-17 22:02:24.184760748 +0000 UTC m=+1260.003087059"
Sep 17 22:02:24.794462 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:24.794432    2564 scope.go:117] "RemoveContainer" containerID="8872575afde23a6aa02d7ab2e88e64f259572e94b05b5e2a83250eb1bc57d73e"
Sep 17 22:02:33.592528 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:33.592481    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:33.592528 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:33.592536    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:33.605952 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:33.605919    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:34.212052 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:34.212014    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:02:38.722891 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:38.722846    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8"]
Sep 17 22:02:38.727811 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:38.727788    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8"
Sep 17 22:02:38.731020 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:38.730989    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-12d586-dockercfg-2dxr8\""
Sep 17 22:02:38.731158 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:38.731033    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-12d586\""
Sep 17 22:02:38.742652 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:38.742624    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8"]
Sep 17 22:02:38.798395 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:38.798355    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg"]
Sep 17 22:02:38.803444 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:38.803413    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg"
Sep 17 22:02:38.812526 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:38.812494    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg"]
Sep 17 22:02:38.875195 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:38.875146    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2x2g\" (UniqueName: \"kubernetes.io/projected/d73be6ba-2cd5-436e-a330-10dde63bcaa7-kube-api-access-l2x2g\") pod \"payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8\" (UID: \"d73be6ba-2cd5-436e-a330-10dde63bcaa7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8"
Sep 17 22:02:38.875407 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:38.875267    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73be6ba-2cd5-436e-a330-10dde63bcaa7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8\" (UID: \"d73be6ba-2cd5-436e-a330-10dde63bcaa7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8"
Sep 17 22:02:38.976419 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:38.976325    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2x2g\" (UniqueName: \"kubernetes.io/projected/d73be6ba-2cd5-436e-a330-10dde63bcaa7-kube-api-access-l2x2g\") pod \"payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8\" (UID: \"d73be6ba-2cd5-436e-a330-10dde63bcaa7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8"
Sep 17 22:02:38.976419 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:38.976372    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7crp7\" (UniqueName: \"kubernetes.io/projected/f8a94467-affb-4833-a0e7-d916a4f71f06-kube-api-access-7crp7\") pod \"payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg\" (UID: \"f8a94467-affb-4833-a0e7-d916a4f71f06\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg"
Sep 17 22:02:38.976419 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:38.976395    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73be6ba-2cd5-436e-a330-10dde63bcaa7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8\" (UID: \"d73be6ba-2cd5-436e-a330-10dde63bcaa7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8"
Sep 17 22:02:38.976644 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:38.976492    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a94467-affb-4833-a0e7-d916a4f71f06-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg\" (UID: \"f8a94467-affb-4833-a0e7-d916a4f71f06\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg"
Sep 17 22:02:38.977070 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:38.977052    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73be6ba-2cd5-436e-a330-10dde63bcaa7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8\" (UID: \"d73be6ba-2cd5-436e-a330-10dde63bcaa7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8"
Sep 17 22:02:38.986391 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:38.986363    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2x2g\" (UniqueName: \"kubernetes.io/projected/d73be6ba-2cd5-436e-a330-10dde63bcaa7-kube-api-access-l2x2g\") pod \"payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8\" (UID: \"d73be6ba-2cd5-436e-a330-10dde63bcaa7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8"
Sep 17 22:02:39.038351 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:39.038315    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8"
Sep 17 22:02:39.077707 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:39.077671    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7crp7\" (UniqueName: \"kubernetes.io/projected/f8a94467-affb-4833-a0e7-d916a4f71f06-kube-api-access-7crp7\") pod \"payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg\" (UID: \"f8a94467-affb-4833-a0e7-d916a4f71f06\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg"
Sep 17 22:02:39.077878 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:39.077731    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a94467-affb-4833-a0e7-d916a4f71f06-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg\" (UID: \"f8a94467-affb-4833-a0e7-d916a4f71f06\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg"
Sep 17 22:02:39.078526 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:39.078501    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a94467-affb-4833-a0e7-d916a4f71f06-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg\" (UID: \"f8a94467-affb-4833-a0e7-d916a4f71f06\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg"
Sep 17 22:02:39.097107 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:39.097048    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7crp7\" (UniqueName: \"kubernetes.io/projected/f8a94467-affb-4833-a0e7-d916a4f71f06-kube-api-access-7crp7\") pod \"payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg\" (UID: \"f8a94467-affb-4833-a0e7-d916a4f71f06\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg"
Sep 17 22:02:39.119078 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:39.119049    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg"
Sep 17 22:02:39.340485 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:39.340462    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8"]
Sep 17 22:02:39.341670 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:02:39.341644    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73be6ba_2cd5_436e_a330_10dde63bcaa7.slice/crio-c9739b5a779cea87592d72393fd63c96f259322536c4e05b6aeb37f9e2d33e6e WatchSource:0}: Error finding container c9739b5a779cea87592d72393fd63c96f259322536c4e05b6aeb37f9e2d33e6e: Status 404 returned error can't find the container with id c9739b5a779cea87592d72393fd63c96f259322536c4e05b6aeb37f9e2d33e6e
Sep 17 22:02:39.412820 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:39.412795    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg"]
Sep 17 22:02:39.415244 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:02:39.415142    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a94467_affb_4833_a0e7_d916a4f71f06.slice/crio-3d2605cca450681eb4542ff99328749264e954a625fccdd5b12cd475c30192f7 WatchSource:0}: Error finding container 3d2605cca450681eb4542ff99328749264e954a625fccdd5b12cd475c30192f7: Status 404 returned error can't find the container with id 3d2605cca450681eb4542ff99328749264e954a625fccdd5b12cd475c30192f7
Sep 17 22:02:40.239132 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:40.239092    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8" event={"ID":"d73be6ba-2cd5-436e-a330-10dde63bcaa7","Type":"ContainerStarted","Data":"7136aa5ad6fcf0b14b5aaee0aed6ae45bec95301a021c88f6894412e610cfa14"}
Sep 17 22:02:40.239590 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:40.239147    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8" event={"ID":"d73be6ba-2cd5-436e-a330-10dde63bcaa7","Type":"ContainerStarted","Data":"c9739b5a779cea87592d72393fd63c96f259322536c4e05b6aeb37f9e2d33e6e"}
Sep 17 22:02:40.239590 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:40.239315    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8"
Sep 17 22:02:40.240869 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:40.240844    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg" event={"ID":"f8a94467-affb-4833-a0e7-d916a4f71f06","Type":"ContainerStarted","Data":"50ec18670a436a6b1cff622a182a8ca420cba389c4f352b80f423669415e1ddb"}
Sep 17 22:02:40.240995 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:40.240885    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg" event={"ID":"f8a94467-affb-4833-a0e7-d916a4f71f06","Type":"ContainerStarted","Data":"3d2605cca450681eb4542ff99328749264e954a625fccdd5b12cd475c30192f7"}
Sep 17 22:02:40.240995 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:40.240912    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg"
Sep 17 22:02:40.269698 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:40.269647    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8" podStartSLOduration=2.269633333 podStartE2EDuration="2.269633333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:02:40.268743976 +0000 UTC m=+1276.087070314" watchObservedRunningTime="2026-09-17 22:02:40.269633333 +0000 UTC m=+1276.087959646"
Sep 17 22:02:40.292135 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:40.292071    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg" podStartSLOduration=2.292050531 podStartE2EDuration="2.292050531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:02:40.290833664 +0000 UTC m=+1276.109159978" watchObservedRunningTime="2026-09-17 22:02:40.292050531 +0000 UTC m=+1276.110376844"
Sep 17 22:02:42.674339 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:42.674303    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a6c05-ttw9s"]
Sep 17 22:02:42.677949 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:42.677933    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a6c05-ttw9s"
Sep 17 22:02:42.680918 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:42.680896    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 22:02:42.681060 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:42.680966    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmdbp\""
Sep 17 22:02:42.688195 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:42.688152    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a6c05-ttw9s"]
Sep 17 22:02:42.819558 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:42.819515    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b109652d-2b05-468a-99ba-a8611107b165-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6a6c05-ttw9s\" (UID: \"b109652d-2b05-468a-99ba-a8611107b165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a6c05-ttw9s"
Sep 17 22:02:42.920958 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:42.920915    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b109652d-2b05-468a-99ba-a8611107b165-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6a6c05-ttw9s\" (UID: \"b109652d-2b05-468a-99ba-a8611107b165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a6c05-ttw9s"
Sep 17 22:02:42.921592 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:42.921570    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b109652d-2b05-468a-99ba-a8611107b165-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6a6c05-ttw9s\" (UID: \"b109652d-2b05-468a-99ba-a8611107b165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a6c05-ttw9s"
Sep 17 22:02:42.990377 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:42.990277    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a6c05-ttw9s"
Sep 17 22:02:43.273232 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:43.273207    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a6c05-ttw9s"]
Sep 17 22:02:43.274970 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:02:43.274933    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb109652d_2b05_468a_99ba_a8611107b165.slice/crio-5feb5705aa90da43512d8f70e4b6b46128c9e60a079dcc0e6d5550f5a9474a29 WatchSource:0}: Error finding container 5feb5705aa90da43512d8f70e4b6b46128c9e60a079dcc0e6d5550f5a9474a29: Status 404 returned error can't find the container with id 5feb5705aa90da43512d8f70e4b6b46128c9e60a079dcc0e6d5550f5a9474a29
Sep 17 22:02:44.267235 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:44.267114    2564 generic.go:356] "Generic (PLEG): container finished" podID="b109652d-2b05-468a-99ba-a8611107b165" containerID="1df7f6675730fc35cb5d5d38cf9437666009cdcb0729d0b002460db09058fba6" exitCode=0
Sep 17 22:02:44.267634 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:44.267225    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a6c05-ttw9s" event={"ID":"b109652d-2b05-468a-99ba-a8611107b165","Type":"ContainerDied","Data":"1df7f6675730fc35cb5d5d38cf9437666009cdcb0729d0b002460db09058fba6"}
Sep 17 22:02:44.267634 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:44.267278    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a6c05-ttw9s" event={"ID":"b109652d-2b05-468a-99ba-a8611107b165","Type":"ContainerStarted","Data":"5feb5705aa90da43512d8f70e4b6b46128c9e60a079dcc0e6d5550f5a9474a29"}
Sep 17 22:02:45.424296 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:45.424269    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a6c05-ttw9s"
Sep 17 22:02:45.441829 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:45.441805    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b109652d-2b05-468a-99ba-a8611107b165-maas-api-service-ca\") pod \"b109652d-2b05-468a-99ba-a8611107b165\" (UID: \"b109652d-2b05-468a-99ba-a8611107b165\") "
Sep 17 22:02:45.442136 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:45.442114    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b109652d-2b05-468a-99ba-a8611107b165-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b109652d-2b05-468a-99ba-a8611107b165" (UID: "b109652d-2b05-468a-99ba-a8611107b165"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:02:45.542813 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:45.542722    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b109652d-2b05-468a-99ba-a8611107b165-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:02:46.280712 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:46.280684    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a6c05-ttw9s"
Sep 17 22:02:46.280879 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:46.280688    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a6c05-ttw9s" event={"ID":"b109652d-2b05-468a-99ba-a8611107b165","Type":"ContainerDied","Data":"5feb5705aa90da43512d8f70e4b6b46128c9e60a079dcc0e6d5550f5a9474a29"}
Sep 17 22:02:46.280879 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:46.280810    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5feb5705aa90da43512d8f70e4b6b46128c9e60a079dcc0e6d5550f5a9474a29"
Sep 17 22:02:47.769367 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:47.769332    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"]
Sep 17 22:02:51.249580 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:51.249548    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg"
Sep 17 22:02:51.249970 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:51.249727    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8"
Sep 17 22:02:55.927623 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:55.927588    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a6c05-29828040-n57qk"]
Sep 17 22:02:55.941692 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:55.941650    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a6c05-29828040-n57qk"]
Sep 17 22:02:55.995125 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:55.995085    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l"]
Sep 17 22:02:55.995383 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:55.995341    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l" podUID="95f8375e-116e-478d-b3d6-f634936aa0f0" containerName="payload-pre-processing" containerID="cri-o://a2a591c21d8414800c95e3ddefdab5f2ab77c2582ac76dd53c5d1eff276bea51" gracePeriod=30
Sep 17 22:02:56.004045 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.004021    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf"]
Sep 17 22:02:56.004341 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.004317    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf" podUID="dcfb0f22-9ccc-4012-a92d-7875c900d3da" containerName="payload-processing" containerID="cri-o://22ea0452fae25218746738dc8a1c275d86ef939bd6680dc91bdfe23aec345284" gracePeriod=30
Sep 17 22:02:56.294484 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.294458    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l"
Sep 17 22:02:56.297874 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.297851    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf"
Sep 17 22:02:56.347941 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.347902    2564 generic.go:356] "Generic (PLEG): container finished" podID="95f8375e-116e-478d-b3d6-f634936aa0f0" containerID="a2a591c21d8414800c95e3ddefdab5f2ab77c2582ac76dd53c5d1eff276bea51" exitCode=0
Sep 17 22:02:56.348139 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.347943    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l" event={"ID":"95f8375e-116e-478d-b3d6-f634936aa0f0","Type":"ContainerDied","Data":"a2a591c21d8414800c95e3ddefdab5f2ab77c2582ac76dd53c5d1eff276bea51"}
Sep 17 22:02:56.348139 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.347963    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l"
Sep 17 22:02:56.348139 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.347999    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l" event={"ID":"95f8375e-116e-478d-b3d6-f634936aa0f0","Type":"ContainerDied","Data":"235f8a67fabae9a87021b72940c85b1e54ff84d46fd9c8b2f888ed7faf6daf64"}
Sep 17 22:02:56.348139 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.348026    2564 scope.go:117] "RemoveContainer" containerID="a2a591c21d8414800c95e3ddefdab5f2ab77c2582ac76dd53c5d1eff276bea51"
Sep 17 22:02:56.348773 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.348738    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pd6h\" (UniqueName: \"kubernetes.io/projected/dcfb0f22-9ccc-4012-a92d-7875c900d3da-kube-api-access-8pd6h\") pod \"dcfb0f22-9ccc-4012-a92d-7875c900d3da\" (UID: \"dcfb0f22-9ccc-4012-a92d-7875c900d3da\") "
Sep 17 22:02:56.348900 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.348890    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95f8375e-116e-478d-b3d6-f634936aa0f0-plugins-config-volume\") pod \"95f8375e-116e-478d-b3d6-f634936aa0f0\" (UID: \"95f8375e-116e-478d-b3d6-f634936aa0f0\") "
Sep 17 22:02:56.348977 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.348917    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcfb0f22-9ccc-4012-a92d-7875c900d3da-plugins-config-volume\") pod \"dcfb0f22-9ccc-4012-a92d-7875c900d3da\" (UID: \"dcfb0f22-9ccc-4012-a92d-7875c900d3da\") "
Sep 17 22:02:56.348977 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.348947    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pccbj\" (UniqueName: \"kubernetes.io/projected/95f8375e-116e-478d-b3d6-f634936aa0f0-kube-api-access-pccbj\") pod \"95f8375e-116e-478d-b3d6-f634936aa0f0\" (UID: \"95f8375e-116e-478d-b3d6-f634936aa0f0\") "
Sep 17 22:02:56.349381 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.349341    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcfb0f22-9ccc-4012-a92d-7875c900d3da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dcfb0f22-9ccc-4012-a92d-7875c900d3da" (UID: "dcfb0f22-9ccc-4012-a92d-7875c900d3da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:02:56.349381 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.349362    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95f8375e-116e-478d-b3d6-f634936aa0f0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "95f8375e-116e-478d-b3d6-f634936aa0f0" (UID: "95f8375e-116e-478d-b3d6-f634936aa0f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:02:56.350011 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.349979    2564 generic.go:356] "Generic (PLEG): container finished" podID="dcfb0f22-9ccc-4012-a92d-7875c900d3da" containerID="22ea0452fae25218746738dc8a1c275d86ef939bd6680dc91bdfe23aec345284" exitCode=0
Sep 17 22:02:56.350099 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.350052    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf" event={"ID":"dcfb0f22-9ccc-4012-a92d-7875c900d3da","Type":"ContainerDied","Data":"22ea0452fae25218746738dc8a1c275d86ef939bd6680dc91bdfe23aec345284"}
Sep 17 22:02:56.350099 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.350095    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf" event={"ID":"dcfb0f22-9ccc-4012-a92d-7875c900d3da","Type":"ContainerDied","Data":"6e611a5868855f811616a28d877f103507107d8d40cb33cf71edacff824aaf13"}
Sep 17 22:02:56.350277 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.350191    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf"
Sep 17 22:02:56.351410 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.351388    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a6c05-ttw9s"]
Sep 17 22:02:56.351516 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.351443    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcfb0f22-9ccc-4012-a92d-7875c900d3da-kube-api-access-8pd6h" (OuterVolumeSpecName: "kube-api-access-8pd6h") pod "dcfb0f22-9ccc-4012-a92d-7875c900d3da" (UID: "dcfb0f22-9ccc-4012-a92d-7875c900d3da"). InnerVolumeSpecName "kube-api-access-8pd6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:02:56.351872 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.351795    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f8375e-116e-478d-b3d6-f634936aa0f0-kube-api-access-pccbj" (OuterVolumeSpecName: "kube-api-access-pccbj") pod "95f8375e-116e-478d-b3d6-f634936aa0f0" (UID: "95f8375e-116e-478d-b3d6-f634936aa0f0"). InnerVolumeSpecName "kube-api-access-pccbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:02:56.365456 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.365431    2564 scope.go:117] "RemoveContainer" containerID="a2a591c21d8414800c95e3ddefdab5f2ab77c2582ac76dd53c5d1eff276bea51"
Sep 17 22:02:56.365860 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:02:56.365827    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2a591c21d8414800c95e3ddefdab5f2ab77c2582ac76dd53c5d1eff276bea51\": container with ID starting with a2a591c21d8414800c95e3ddefdab5f2ab77c2582ac76dd53c5d1eff276bea51 not found: ID does not exist" containerID="a2a591c21d8414800c95e3ddefdab5f2ab77c2582ac76dd53c5d1eff276bea51"
Sep 17 22:02:56.365975 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.365871    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2a591c21d8414800c95e3ddefdab5f2ab77c2582ac76dd53c5d1eff276bea51"} err="failed to get container status \"a2a591c21d8414800c95e3ddefdab5f2ab77c2582ac76dd53c5d1eff276bea51\": rpc error: code = NotFound desc = could not find container \"a2a591c21d8414800c95e3ddefdab5f2ab77c2582ac76dd53c5d1eff276bea51\": container with ID starting with a2a591c21d8414800c95e3ddefdab5f2ab77c2582ac76dd53c5d1eff276bea51 not found: ID does not exist"
Sep 17 22:02:56.365975 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.365896    2564 scope.go:117] "RemoveContainer" containerID="22ea0452fae25218746738dc8a1c275d86ef939bd6680dc91bdfe23aec345284"
Sep 17 22:02:56.373596 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.373572    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a6c05-ttw9s"]
Sep 17 22:02:56.377446 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.377427    2564 scope.go:117] "RemoveContainer" containerID="22ea0452fae25218746738dc8a1c275d86ef939bd6680dc91bdfe23aec345284"
Sep 17 22:02:56.377809 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:02:56.377784    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22ea0452fae25218746738dc8a1c275d86ef939bd6680dc91bdfe23aec345284\": container with ID starting with 22ea0452fae25218746738dc8a1c275d86ef939bd6680dc91bdfe23aec345284 not found: ID does not exist" containerID="22ea0452fae25218746738dc8a1c275d86ef939bd6680dc91bdfe23aec345284"
Sep 17 22:02:56.377895 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.377815    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ea0452fae25218746738dc8a1c275d86ef939bd6680dc91bdfe23aec345284"} err="failed to get container status \"22ea0452fae25218746738dc8a1c275d86ef939bd6680dc91bdfe23aec345284\": rpc error: code = NotFound desc = could not find container \"22ea0452fae25218746738dc8a1c275d86ef939bd6680dc91bdfe23aec345284\": container with ID starting with 22ea0452fae25218746738dc8a1c275d86ef939bd6680dc91bdfe23aec345284 not found: ID does not exist"
Sep 17 22:02:56.450598 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.450556    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pccbj\" (UniqueName: \"kubernetes.io/projected/95f8375e-116e-478d-b3d6-f634936aa0f0-kube-api-access-pccbj\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:02:56.450598 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.450592    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pd6h\" (UniqueName: \"kubernetes.io/projected/dcfb0f22-9ccc-4012-a92d-7875c900d3da-kube-api-access-8pd6h\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:02:56.450598 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.450607    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95f8375e-116e-478d-b3d6-f634936aa0f0-plugins-config-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:02:56.450858 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.450619    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcfb0f22-9ccc-4012-a92d-7875c900d3da-plugins-config-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:02:56.681552 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.681526    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l"]
Sep 17 22:02:56.685737 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.685715    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6a6c05-5f99497d46-24d6l"]
Sep 17 22:02:56.700726 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.700697    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf"]
Sep 17 22:02:56.703943 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.703919    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6a6c05-65f65cd7d7-xhhwf"]
Sep 17 22:02:56.754277 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.754240    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e0b9d01-6e7d-43c5-bde8-ddefe4313d4f" path="/var/lib/kubelet/pods/3e0b9d01-6e7d-43c5-bde8-ddefe4313d4f/volumes"
Sep 17 22:02:56.754561 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.754548    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95f8375e-116e-478d-b3d6-f634936aa0f0" path="/var/lib/kubelet/pods/95f8375e-116e-478d-b3d6-f634936aa0f0/volumes"
Sep 17 22:02:56.754837 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.754826    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b109652d-2b05-468a-99ba-a8611107b165" path="/var/lib/kubelet/pods/b109652d-2b05-468a-99ba-a8611107b165/volumes"
Sep 17 22:02:56.755157 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:56.755145    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcfb0f22-9ccc-4012-a92d-7875c900d3da" path="/var/lib/kubelet/pods/dcfb0f22-9ccc-4012-a92d-7875c900d3da/volumes"
Sep 17 22:02:59.185482 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:59.185446    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7820-d4dd29"]
Sep 17 22:02:59.185880 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:59.185868    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcfb0f22-9ccc-4012-a92d-7875c900d3da" containerName="payload-processing"
Sep 17 22:02:59.185919 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:59.185881    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcfb0f22-9ccc-4012-a92d-7875c900d3da" containerName="payload-processing"
Sep 17 22:02:59.185919 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:59.185894    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b109652d-2b05-468a-99ba-a8611107b165" containerName="revoke-keys"
Sep 17 22:02:59.185919 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:59.185899    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b109652d-2b05-468a-99ba-a8611107b165" containerName="revoke-keys"
Sep 17 22:02:59.185919 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:59.185907    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95f8375e-116e-478d-b3d6-f634936aa0f0" containerName="payload-pre-processing"
Sep 17 22:02:59.185919 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:59.185913    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="95f8375e-116e-478d-b3d6-f634936aa0f0" containerName="payload-pre-processing"
Sep 17 22:02:59.186086 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:59.185975    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="95f8375e-116e-478d-b3d6-f634936aa0f0" containerName="payload-pre-processing"
Sep 17 22:02:59.186086 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:59.185981    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcfb0f22-9ccc-4012-a92d-7875c900d3da" containerName="payload-processing"
Sep 17 22:02:59.186086 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:59.185991    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b109652d-2b05-468a-99ba-a8611107b165" containerName="revoke-keys"
Sep 17 22:02:59.190854 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:59.190830    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7820-d4dd29"
Sep 17 22:02:59.194655 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:59.194629    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lbp7g\""
Sep 17 22:02:59.196391 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:59.196367    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7820-d4dd29"]
Sep 17 22:02:59.278056 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:59.278022    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zhl6\" (UniqueName: \"kubernetes.io/projected/cb05e0b0-5ade-4048-abb9-3d34331027b8-kube-api-access-5zhl6\") pod \"test-curl-7820-d4dd29\" (UID: \"cb05e0b0-5ade-4048-abb9-3d34331027b8\") " pod="opendatahub/test-curl-7820-d4dd29"
Sep 17 22:02:59.379115 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:59.379081    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zhl6\" (UniqueName: \"kubernetes.io/projected/cb05e0b0-5ade-4048-abb9-3d34331027b8-kube-api-access-5zhl6\") pod \"test-curl-7820-d4dd29\" (UID: \"cb05e0b0-5ade-4048-abb9-3d34331027b8\") " pod="opendatahub/test-curl-7820-d4dd29"
Sep 17 22:02:59.388519 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:59.388485    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zhl6\" (UniqueName: \"kubernetes.io/projected/cb05e0b0-5ade-4048-abb9-3d34331027b8-kube-api-access-5zhl6\") pod \"test-curl-7820-d4dd29\" (UID: \"cb05e0b0-5ade-4048-abb9-3d34331027b8\") " pod="opendatahub/test-curl-7820-d4dd29"
Sep 17 22:02:59.504349 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:59.504240    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7820-d4dd29"
Sep 17 22:02:59.805953 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:59.805927    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7820-d4dd29"]
Sep 17 22:02:59.808078 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:02:59.808050    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb05e0b0_5ade_4048_abb9_3d34331027b8.slice/crio-643c40a7fed642377c7a783c2acbfaa1846a9624f8d8deb0b1c37e2e8d8dbac5 WatchSource:0}: Error finding container 643c40a7fed642377c7a783c2acbfaa1846a9624f8d8deb0b1c37e2e8d8dbac5: Status 404 returned error can't find the container with id 643c40a7fed642377c7a783c2acbfaa1846a9624f8d8deb0b1c37e2e8d8dbac5
Sep 17 22:02:59.810051 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:02:59.810034    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:03:00.377095 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:00.377048    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7820-d4dd29" event={"ID":"cb05e0b0-5ade-4048-abb9-3d34331027b8","Type":"ContainerStarted","Data":"643c40a7fed642377c7a783c2acbfaa1846a9624f8d8deb0b1c37e2e8d8dbac5"}
Sep 17 22:03:02.391424 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:02.391389    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7820-d4dd29" event={"ID":"cb05e0b0-5ade-4048-abb9-3d34331027b8","Type":"ContainerStarted","Data":"aa2ca380627699fceece4dcf8255ebaabe66d449a4ea027777570f41b00f34b1"}
Sep 17 22:03:02.417773 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:02.417720    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7820-d4dd29" podStartSLOduration=1.5669178289999999 podStartE2EDuration="3.417701053s" totalImagesPullingTime="1.850783224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:03:02.41509275 +0000 UTC m=+1298.233419062" watchObservedRunningTime="2026-09-17 22:03:02.417701053 +0000 UTC m=+1298.236027366"
Sep 17 22:03:02.505551 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:02.505509    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"]
Sep 17 22:03:02.789352 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:02.789305    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns" podUID="12ae6330-0fbf-41ff-bc85-ff4e2322c76e" containerName="main" containerID="cri-o://0008b2ac017d0c13a37e4e4434241fd92b999bcaee104af8526d91d77c3f174d" gracePeriod=45
Sep 17 22:03:03.053271 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.053203    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 22:03:03.123670 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.123639    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-home\") pod \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\" (UID: \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\") "
Sep 17 22:03:03.123815 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.123690    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-dshm\") pod \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\" (UID: \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\") "
Sep 17 22:03:03.123815 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.123711    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lt7g\" (UniqueName: \"kubernetes.io/projected/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-kube-api-access-8lt7g\") pod \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\" (UID: \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\") "
Sep 17 22:03:03.123815 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.123747    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-tls-certs\") pod \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\" (UID: \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\") "
Sep 17 22:03:03.123815 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.123767    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-model-cache\") pod \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\" (UID: \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\") "
Sep 17 22:03:03.124021 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.123858    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-tmp-dir\") pod \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\" (UID: \"12ae6330-0fbf-41ff-bc85-ff4e2322c76e\") "
Sep 17 22:03:03.124021 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.123927    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-home" (OuterVolumeSpecName: "home") pod "12ae6330-0fbf-41ff-bc85-ff4e2322c76e" (UID: "12ae6330-0fbf-41ff-bc85-ff4e2322c76e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:03:03.124143 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.124110    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-model-cache" (OuterVolumeSpecName: "model-cache") pod "12ae6330-0fbf-41ff-bc85-ff4e2322c76e" (UID: "12ae6330-0fbf-41ff-bc85-ff4e2322c76e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:03:03.124233 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.124205    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "12ae6330-0fbf-41ff-bc85-ff4e2322c76e" (UID: "12ae6330-0fbf-41ff-bc85-ff4e2322c76e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:03:03.124294 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.124220    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-home\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:03.124294 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.124263    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-model-cache\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:03.126241 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.126211    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12ae6330-0fbf-41ff-bc85-ff4e2322c76e" (UID: "12ae6330-0fbf-41ff-bc85-ff4e2322c76e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:03:03.126241 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.126226    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-kube-api-access-8lt7g" (OuterVolumeSpecName: "kube-api-access-8lt7g") pod "12ae6330-0fbf-41ff-bc85-ff4e2322c76e" (UID: "12ae6330-0fbf-41ff-bc85-ff4e2322c76e"). InnerVolumeSpecName "kube-api-access-8lt7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:03:03.126384 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.126268    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-dshm" (OuterVolumeSpecName: "dshm") pod "12ae6330-0fbf-41ff-bc85-ff4e2322c76e" (UID: "12ae6330-0fbf-41ff-bc85-ff4e2322c76e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:03:03.181641 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.181598    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7820-d4dd29"]
Sep 17 22:03:03.188651 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.188615    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7820-d4dd29"]
Sep 17 22:03:03.225453 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.225422    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-dshm\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:03.225453 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.225453    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lt7g\" (UniqueName: \"kubernetes.io/projected/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-kube-api-access-8lt7g\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:03.225659 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.225468    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-tls-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:03.225659 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.225482    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12ae6330-0fbf-41ff-bc85-ff4e2322c76e-tmp-dir\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:03.306783 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.306700    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7820-bdb6db"]
Sep 17 22:03:03.307339 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.307320    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12ae6330-0fbf-41ff-bc85-ff4e2322c76e" containerName="main"
Sep 17 22:03:03.307406 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.307342    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="12ae6330-0fbf-41ff-bc85-ff4e2322c76e" containerName="main"
Sep 17 22:03:03.307406 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.307371    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb05e0b0-5ade-4048-abb9-3d34331027b8" containerName="test-curl-7820-d4dd29"
Sep 17 22:03:03.307406 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.307380    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb05e0b0-5ade-4048-abb9-3d34331027b8" containerName="test-curl-7820-d4dd29"
Sep 17 22:03:03.307504 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.307482    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb05e0b0-5ade-4048-abb9-3d34331027b8" containerName="test-curl-7820-d4dd29"
Sep 17 22:03:03.307504 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.307495    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="12ae6330-0fbf-41ff-bc85-ff4e2322c76e" containerName="main"
Sep 17 22:03:03.310899 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.310874    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7820-bdb6db"
Sep 17 22:03:03.318967 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.318944    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7820-bdb6db"]
Sep 17 22:03:03.400061 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.400025    2564 generic.go:356] "Generic (PLEG): container finished" podID="12ae6330-0fbf-41ff-bc85-ff4e2322c76e" containerID="0008b2ac017d0c13a37e4e4434241fd92b999bcaee104af8526d91d77c3f174d" exitCode=0
Sep 17 22:03:03.400535 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.400081    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"
Sep 17 22:03:03.400535 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.400112    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns" event={"ID":"12ae6330-0fbf-41ff-bc85-ff4e2322c76e","Type":"ContainerDied","Data":"0008b2ac017d0c13a37e4e4434241fd92b999bcaee104af8526d91d77c3f174d"}
Sep 17 22:03:03.400535 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.400200    2564 scope.go:117] "RemoveContainer" containerID="0008b2ac017d0c13a37e4e4434241fd92b999bcaee104af8526d91d77c3f174d"
Sep 17 22:03:03.400535 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.400197    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns" event={"ID":"12ae6330-0fbf-41ff-bc85-ff4e2322c76e","Type":"ContainerDied","Data":"152b43a456008120bf556d9ddcbd697d821ef9a846dbc0d8763ac798815f36e2"}
Sep 17 22:03:03.400535 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:03:03.400509    2564 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7820-d4dd29" container="test-curl-7820-d4dd29"
Sep 17 22:03:03.416600 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.416578    2564 scope.go:117] "RemoveContainer" containerID="0008b2ac017d0c13a37e4e4434241fd92b999bcaee104af8526d91d77c3f174d"
Sep 17 22:03:03.416923 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:03:03.416896    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0008b2ac017d0c13a37e4e4434241fd92b999bcaee104af8526d91d77c3f174d\": container with ID starting with 0008b2ac017d0c13a37e4e4434241fd92b999bcaee104af8526d91d77c3f174d not found: ID does not exist" containerID="0008b2ac017d0c13a37e4e4434241fd92b999bcaee104af8526d91d77c3f174d"
Sep 17 22:03:03.416992 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.416931    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0008b2ac017d0c13a37e4e4434241fd92b999bcaee104af8526d91d77c3f174d"} err="failed to get container status \"0008b2ac017d0c13a37e4e4434241fd92b999bcaee104af8526d91d77c3f174d\": rpc error: code = NotFound desc = could not find container \"0008b2ac017d0c13a37e4e4434241fd92b999bcaee104af8526d91d77c3f174d\": container with ID starting with 0008b2ac017d0c13a37e4e4434241fd92b999bcaee104af8526d91d77c3f174d not found: ID does not exist"
Sep 17 22:03:03.427654 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.427616    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxgpj\" (UniqueName: \"kubernetes.io/projected/ef2d5ed5-11e0-4edb-9cf3-b141ed2eabc5-kube-api-access-qxgpj\") pod \"test-curl-7820-bdb6db\" (UID: \"ef2d5ed5-11e0-4edb-9cf3-b141ed2eabc5\") " pod="opendatahub/test-curl-7820-bdb6db"
Sep 17 22:03:03.432357 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.432318    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"]
Sep 17 22:03:03.442421 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.442389    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6a6c05/test-model-6a6c05-kserve-5966cfccf5-lc5ns"]
Sep 17 22:03:03.528584 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.528527    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxgpj\" (UniqueName: \"kubernetes.io/projected/ef2d5ed5-11e0-4edb-9cf3-b141ed2eabc5-kube-api-access-qxgpj\") pod \"test-curl-7820-bdb6db\" (UID: \"ef2d5ed5-11e0-4edb-9cf3-b141ed2eabc5\") " pod="opendatahub/test-curl-7820-bdb6db"
Sep 17 22:03:03.539161 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.539135    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxgpj\" (UniqueName: \"kubernetes.io/projected/ef2d5ed5-11e0-4edb-9cf3-b141ed2eabc5-kube-api-access-qxgpj\") pod \"test-curl-7820-bdb6db\" (UID: \"ef2d5ed5-11e0-4edb-9cf3-b141ed2eabc5\") " pod="opendatahub/test-curl-7820-bdb6db"
Sep 17 22:03:03.621146 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.621057    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7820-bdb6db"
Sep 17 22:03:03.914620 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:03.914590    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7820-bdb6db"]
Sep 17 22:03:03.916562 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:03:03.916535    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef2d5ed5_11e0_4edb_9cf3_b141ed2eabc5.slice/crio-311b36232d74335c4d4c38c70a9b0f4cd7bbc664456e22909b38e15d6d6ccbc1 WatchSource:0}: Error finding container 311b36232d74335c4d4c38c70a9b0f4cd7bbc664456e22909b38e15d6d6ccbc1: Status 404 returned error can't find the container with id 311b36232d74335c4d4c38c70a9b0f4cd7bbc664456e22909b38e15d6d6ccbc1
Sep 17 22:03:04.409943 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:04.409899    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7820-bdb6db" event={"ID":"ef2d5ed5-11e0-4edb-9cf3-b141ed2eabc5","Type":"ContainerStarted","Data":"1db788b8fd913d9c3c76dffad1fe03f068960d0f1623e642cfa018dd497d69a3"}
Sep 17 22:03:04.410440 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:04.409954    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7820-bdb6db" event={"ID":"ef2d5ed5-11e0-4edb-9cf3-b141ed2eabc5","Type":"ContainerStarted","Data":"311b36232d74335c4d4c38c70a9b0f4cd7bbc664456e22909b38e15d6d6ccbc1"}
Sep 17 22:03:04.410440 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:04.409998    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7820-d4dd29" podUID="cb05e0b0-5ade-4048-abb9-3d34331027b8" containerName="test-curl-7820-d4dd29" containerID="cri-o://aa2ca380627699fceece4dcf8255ebaabe66d449a4ea027777570f41b00f34b1" gracePeriod=2
Sep 17 22:03:04.413970 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:03:04.413928    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7820-d4dd29\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="cb05e0b0-5ade-4048-abb9-3d34331027b8" pod="opendatahub/test-curl-7820-d4dd29"
Sep 17 22:03:04.443145 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:03:04.443104    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7820-d4dd29\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="cb05e0b0-5ade-4048-abb9-3d34331027b8" pod="opendatahub/test-curl-7820-d4dd29"
Sep 17 22:03:04.443145 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:04.443110    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7820-bdb6db" podStartSLOduration=1.443094256 podStartE2EDuration="1.443094256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:03:04.440759272 +0000 UTC m=+1300.259085586" watchObservedRunningTime="2026-09-17 22:03:04.443094256 +0000 UTC m=+1300.261420571"
Sep 17 22:03:04.752970 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:03:04.752878    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7820-d4dd29\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="cb05e0b0-5ade-4048-abb9-3d34331027b8" pod="opendatahub/test-curl-7820-d4dd29"
Sep 17 22:03:04.754626 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:04.754600    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12ae6330-0fbf-41ff-bc85-ff4e2322c76e" path="/var/lib/kubelet/pods/12ae6330-0fbf-41ff-bc85-ff4e2322c76e/volumes"
Sep 17 22:03:05.062945 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:05.062844    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7820-bdb6db"]
Sep 17 22:03:05.068192 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:05.068148    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7820-bdb6db"]
Sep 17 22:03:05.191033 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:05.190994    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7820-c6d2c7"]
Sep 17 22:03:05.191640 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:05.191624    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef2d5ed5-11e0-4edb-9cf3-b141ed2eabc5" containerName="test-curl-7820-bdb6db"
Sep 17 22:03:05.191700 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:05.191643    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef2d5ed5-11e0-4edb-9cf3-b141ed2eabc5" containerName="test-curl-7820-bdb6db"
Sep 17 22:03:05.191750 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:05.191739    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef2d5ed5-11e0-4edb-9cf3-b141ed2eabc5" containerName="test-curl-7820-bdb6db"
Sep 17 22:03:05.195469 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:05.195443    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7820-c6d2c7"
Sep 17 22:03:05.197404 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:05.197380    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7820-c6d2c7"]
Sep 17 22:03:05.244632 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:05.244599    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nphcn\" (UniqueName: \"kubernetes.io/projected/a10ebd5e-3da5-4951-9a35-41b4d40fc930-kube-api-access-nphcn\") pod \"test-curl-7820-c6d2c7\" (UID: \"a10ebd5e-3da5-4951-9a35-41b4d40fc930\") " pod="opendatahub/test-curl-7820-c6d2c7"
Sep 17 22:03:05.345726 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:05.345628    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nphcn\" (UniqueName: \"kubernetes.io/projected/a10ebd5e-3da5-4951-9a35-41b4d40fc930-kube-api-access-nphcn\") pod \"test-curl-7820-c6d2c7\" (UID: \"a10ebd5e-3da5-4951-9a35-41b4d40fc930\") " pod="opendatahub/test-curl-7820-c6d2c7"
Sep 17 22:03:05.355320 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:05.355289    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nphcn\" (UniqueName: \"kubernetes.io/projected/a10ebd5e-3da5-4951-9a35-41b4d40fc930-kube-api-access-nphcn\") pod \"test-curl-7820-c6d2c7\" (UID: \"a10ebd5e-3da5-4951-9a35-41b4d40fc930\") " pod="opendatahub/test-curl-7820-c6d2c7"
Sep 17 22:03:05.415807 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:03:05.415780    2564 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7820-bdb6db" container="test-curl-7820-bdb6db"
Sep 17 22:03:05.507748 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:05.507706    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7820-c6d2c7"
Sep 17 22:03:05.781702 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:05.781676    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7820-c6d2c7"]
Sep 17 22:03:05.783850 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:03:05.783819    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda10ebd5e_3da5_4951_9a35_41b4d40fc930.slice/crio-5b5656d28359bdf11f2cdbacbd76e5f2ade011da4947d5357a234aa4e450efc4 WatchSource:0}: Error finding container 5b5656d28359bdf11f2cdbacbd76e5f2ade011da4947d5357a234aa4e450efc4: Status 404 returned error can't find the container with id 5b5656d28359bdf11f2cdbacbd76e5f2ade011da4947d5357a234aa4e450efc4
Sep 17 22:03:06.424648 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:06.424610    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7820-c6d2c7" event={"ID":"a10ebd5e-3da5-4951-9a35-41b4d40fc930","Type":"ContainerStarted","Data":"23addd9c3a4577e782220735fdc6a60508e1e260ddb85ecd54a8fec023818c51"}
Sep 17 22:03:06.425020 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:06.424663    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7820-c6d2c7" event={"ID":"a10ebd5e-3da5-4951-9a35-41b4d40fc930","Type":"ContainerStarted","Data":"5b5656d28359bdf11f2cdbacbd76e5f2ade011da4947d5357a234aa4e450efc4"}
Sep 17 22:03:06.425020 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:06.424790    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7820-bdb6db" podUID="ef2d5ed5-11e0-4edb-9cf3-b141ed2eabc5" containerName="test-curl-7820-bdb6db" containerID="cri-o://1db788b8fd913d9c3c76dffad1fe03f068960d0f1623e642cfa018dd497d69a3" gracePeriod=2
Sep 17 22:03:06.443225 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:03:06.443158    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7820-bdb6db\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="ef2d5ed5-11e0-4edb-9cf3-b141ed2eabc5" pod="opendatahub/test-curl-7820-bdb6db"
Sep 17 22:03:06.443433 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:06.443392    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7820-c6d2c7" podStartSLOduration=1.443373505 podStartE2EDuration="1.443373505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:03:06.440651645 +0000 UTC m=+1302.258977958" watchObservedRunningTime="2026-09-17 22:03:06.443373505 +0000 UTC m=+1302.261699819"
Sep 17 22:03:06.614609 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:06.614581    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7820-d4dd29"
Sep 17 22:03:06.617240 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:03:06.617215    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7820-bdb6db\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="ef2d5ed5-11e0-4edb-9cf3-b141ed2eabc5" pod="opendatahub/test-curl-7820-bdb6db"
Sep 17 22:03:06.760346 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:06.760255    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zhl6\" (UniqueName: \"kubernetes.io/projected/cb05e0b0-5ade-4048-abb9-3d34331027b8-kube-api-access-5zhl6\") pod \"cb05e0b0-5ade-4048-abb9-3d34331027b8\" (UID: \"cb05e0b0-5ade-4048-abb9-3d34331027b8\") "
Sep 17 22:03:06.762650 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:06.762617    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb05e0b0-5ade-4048-abb9-3d34331027b8-kube-api-access-5zhl6" (OuterVolumeSpecName: "kube-api-access-5zhl6") pod "cb05e0b0-5ade-4048-abb9-3d34331027b8" (UID: "cb05e0b0-5ade-4048-abb9-3d34331027b8"). InnerVolumeSpecName "kube-api-access-5zhl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:03:06.861814 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:06.861786    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zhl6\" (UniqueName: \"kubernetes.io/projected/cb05e0b0-5ade-4048-abb9-3d34331027b8-kube-api-access-5zhl6\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:07.058436 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:07.058346    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7820-c6d2c7"]
Sep 17 22:03:07.063482 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:07.063454    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7820-c6d2c7"]
Sep 17 22:03:07.432540 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:07.432496    2564 generic.go:356] "Generic (PLEG): container finished" podID="cb05e0b0-5ade-4048-abb9-3d34331027b8" containerID="aa2ca380627699fceece4dcf8255ebaabe66d449a4ea027777570f41b00f34b1" exitCode=137
Sep 17 22:03:07.433009 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:07.432554    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7820-d4dd29"
Sep 17 22:03:07.433009 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:07.432603    2564 scope.go:117] "RemoveContainer" containerID="aa2ca380627699fceece4dcf8255ebaabe66d449a4ea027777570f41b00f34b1"
Sep 17 22:03:07.433009 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:03:07.432895    2564 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7820-c6d2c7" container="test-curl-7820-c6d2c7"
Sep 17 22:03:07.442316 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:07.442293    2564 scope.go:117] "RemoveContainer" containerID="aa2ca380627699fceece4dcf8255ebaabe66d449a4ea027777570f41b00f34b1"
Sep 17 22:03:07.442602 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:03:07.442583    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa2ca380627699fceece4dcf8255ebaabe66d449a4ea027777570f41b00f34b1\": container with ID starting with aa2ca380627699fceece4dcf8255ebaabe66d449a4ea027777570f41b00f34b1 not found: ID does not exist" containerID="aa2ca380627699fceece4dcf8255ebaabe66d449a4ea027777570f41b00f34b1"
Sep 17 22:03:07.442666 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:07.442616    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa2ca380627699fceece4dcf8255ebaabe66d449a4ea027777570f41b00f34b1"} err="failed to get container status \"aa2ca380627699fceece4dcf8255ebaabe66d449a4ea027777570f41b00f34b1\": rpc error: code = NotFound desc = could not find container \"aa2ca380627699fceece4dcf8255ebaabe66d449a4ea027777570f41b00f34b1\": container with ID starting with aa2ca380627699fceece4dcf8255ebaabe66d449a4ea027777570f41b00f34b1 not found: ID does not exist"
Sep 17 22:03:08.301821 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:08.301782    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"]
Sep 17 22:03:08.302121 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:08.302091    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv" podUID="b112866e-0590-411f-bc6f-8f5fe79ed928" containerName="istio-proxy" containerID="cri-o://9d756e8853db6437f6f042ec036f96ec0313e549aa865c4a0b78bd9555ee952a" gracePeriod=30
Sep 17 22:03:08.440093 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:08.440017    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7820-c6d2c7" podUID="a10ebd5e-3da5-4951-9a35-41b4d40fc930" containerName="test-curl-7820-c6d2c7" containerID="cri-o://23addd9c3a4577e782220735fdc6a60508e1e260ddb85ecd54a8fec023818c51" gracePeriod=2
Sep 17 22:03:08.442860 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:03:08.442831    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7820-c6d2c7\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="a10ebd5e-3da5-4951-9a35-41b4d40fc930" pod="opendatahub/test-curl-7820-c6d2c7"
Sep 17 22:03:08.633010 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:08.632987    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7820-bdb6db"
Sep 17 22:03:08.635706 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:03:08.635678    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7820-c6d2c7\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="a10ebd5e-3da5-4951-9a35-41b4d40fc930" pod="opendatahub/test-curl-7820-c6d2c7"
Sep 17 22:03:08.646214 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:08.646191    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d5c73-4872g"]
Sep 17 22:03:08.646625 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:08.646614    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a10ebd5e-3da5-4951-9a35-41b4d40fc930" containerName="test-curl-7820-c6d2c7"
Sep 17 22:03:08.646674 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:08.646628    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a10ebd5e-3da5-4951-9a35-41b4d40fc930" containerName="test-curl-7820-c6d2c7"
Sep 17 22:03:08.646706 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:08.646691    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a10ebd5e-3da5-4951-9a35-41b4d40fc930" containerName="test-curl-7820-c6d2c7"
Sep 17 22:03:08.649640 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:08.649620    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d5c73-4872g"
Sep 17 22:03:08.653679 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:03:08.653637    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7820-c6d2c7\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="a10ebd5e-3da5-4951-9a35-41b4d40fc930" pod="opendatahub/test-curl-7820-c6d2c7"
Sep 17 22:03:08.653957 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:08.653938    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmdbp\""
Sep 17 22:03:08.654090 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:08.653956    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 22:03:08.661891 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:08.661869    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d5c73-4872g"]
Sep 17 22:03:08.754372 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:08.754340    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb05e0b0-5ade-4048-abb9-3d34331027b8" path="/var/lib/kubelet/pods/cb05e0b0-5ade-4048-abb9-3d34331027b8/volumes"
Sep 17 22:03:08.781810 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:08.781782    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxgpj\" (UniqueName: \"kubernetes.io/projected/ef2d5ed5-11e0-4edb-9cf3-b141ed2eabc5-kube-api-access-qxgpj\") pod \"ef2d5ed5-11e0-4edb-9cf3-b141ed2eabc5\" (UID: \"ef2d5ed5-11e0-4edb-9cf3-b141ed2eabc5\") "
Sep 17 22:03:08.781960 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:08.781929    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a33d84f6-937e-44b5-9d4d-a219fe650e19-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3d5c73-4872g\" (UID: \"a33d84f6-937e-44b5-9d4d-a219fe650e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d5c73-4872g"
Sep 17 22:03:08.783999 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:08.783975    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef2d5ed5-11e0-4edb-9cf3-b141ed2eabc5-kube-api-access-qxgpj" (OuterVolumeSpecName: "kube-api-access-qxgpj") pod "ef2d5ed5-11e0-4edb-9cf3-b141ed2eabc5" (UID: "ef2d5ed5-11e0-4edb-9cf3-b141ed2eabc5"). InnerVolumeSpecName "kube-api-access-qxgpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:03:08.882874 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:08.882776    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a33d84f6-937e-44b5-9d4d-a219fe650e19-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3d5c73-4872g\" (UID: \"a33d84f6-937e-44b5-9d4d-a219fe650e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d5c73-4872g"
Sep 17 22:03:08.882874 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:08.882838    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxgpj\" (UniqueName: \"kubernetes.io/projected/ef2d5ed5-11e0-4edb-9cf3-b141ed2eabc5-kube-api-access-qxgpj\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:08.883486 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:08.883461    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a33d84f6-937e-44b5-9d4d-a219fe650e19-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3d5c73-4872g\" (UID: \"a33d84f6-937e-44b5-9d4d-a219fe650e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d5c73-4872g"
Sep 17 22:03:08.959693 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:08.959658    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d5c73-4872g"
Sep 17 22:03:09.241265 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:09.241230    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d5c73-4872g"]
Sep 17 22:03:09.242072 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:03:09.242040    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33d84f6_937e_44b5_9d4d_a219fe650e19.slice/crio-6bb5766b719d57c613a2cac07388b885f67ad7746386dbc29932180d45b255fe WatchSource:0}: Error finding container 6bb5766b719d57c613a2cac07388b885f67ad7746386dbc29932180d45b255fe: Status 404 returned error can't find the container with id 6bb5766b719d57c613a2cac07388b885f67ad7746386dbc29932180d45b255fe
Sep 17 22:03:09.447112 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:09.447077    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d5c73-4872g" event={"ID":"a33d84f6-937e-44b5-9d4d-a219fe650e19","Type":"ContainerStarted","Data":"6bb5766b719d57c613a2cac07388b885f67ad7746386dbc29932180d45b255fe"}
Sep 17 22:03:09.448394 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:09.448371    2564 generic.go:356] "Generic (PLEG): container finished" podID="ef2d5ed5-11e0-4edb-9cf3-b141ed2eabc5" containerID="1db788b8fd913d9c3c76dffad1fe03f068960d0f1623e642cfa018dd497d69a3" exitCode=137
Sep 17 22:03:09.448514 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:09.448412    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7820-bdb6db"
Sep 17 22:03:09.448514 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:09.448455    2564 scope.go:117] "RemoveContainer" containerID="1db788b8fd913d9c3c76dffad1fe03f068960d0f1623e642cfa018dd497d69a3"
Sep 17 22:03:09.458773 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:09.458751    2564 scope.go:117] "RemoveContainer" containerID="1db788b8fd913d9c3c76dffad1fe03f068960d0f1623e642cfa018dd497d69a3"
Sep 17 22:03:09.459121 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:03:09.459102    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1db788b8fd913d9c3c76dffad1fe03f068960d0f1623e642cfa018dd497d69a3\": container with ID starting with 1db788b8fd913d9c3c76dffad1fe03f068960d0f1623e642cfa018dd497d69a3 not found: ID does not exist" containerID="1db788b8fd913d9c3c76dffad1fe03f068960d0f1623e642cfa018dd497d69a3"
Sep 17 22:03:09.459202 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:09.459130    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db788b8fd913d9c3c76dffad1fe03f068960d0f1623e642cfa018dd497d69a3"} err="failed to get container status \"1db788b8fd913d9c3c76dffad1fe03f068960d0f1623e642cfa018dd497d69a3\": rpc error: code = NotFound desc = could not find container \"1db788b8fd913d9c3c76dffad1fe03f068960d0f1623e642cfa018dd497d69a3\": container with ID starting with 1db788b8fd913d9c3c76dffad1fe03f068960d0f1623e642cfa018dd497d69a3 not found: ID does not exist"
Sep 17 22:03:10.457811 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:10.457780    2564 generic.go:356] "Generic (PLEG): container finished" podID="a33d84f6-937e-44b5-9d4d-a219fe650e19" containerID="8da09c52221a0b8db7a630197f2d7b5504dd97e13813cfed78d7c8db35fae41d" exitCode=0
Sep 17 22:03:10.458161 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:10.457868    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d5c73-4872g" event={"ID":"a33d84f6-937e-44b5-9d4d-a219fe650e19","Type":"ContainerDied","Data":"8da09c52221a0b8db7a630197f2d7b5504dd97e13813cfed78d7c8db35fae41d"}
Sep 17 22:03:10.642087 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:10.642056    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7820-c6d2c7"
Sep 17 22:03:10.754724 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:10.754630    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef2d5ed5-11e0-4edb-9cf3-b141ed2eabc5" path="/var/lib/kubelet/pods/ef2d5ed5-11e0-4edb-9cf3-b141ed2eabc5/volumes"
Sep 17 22:03:10.802018 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:10.801981    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nphcn\" (UniqueName: \"kubernetes.io/projected/a10ebd5e-3da5-4951-9a35-41b4d40fc930-kube-api-access-nphcn\") pod \"a10ebd5e-3da5-4951-9a35-41b4d40fc930\" (UID: \"a10ebd5e-3da5-4951-9a35-41b4d40fc930\") "
Sep 17 22:03:10.804289 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:10.804266    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10ebd5e-3da5-4951-9a35-41b4d40fc930-kube-api-access-nphcn" (OuterVolumeSpecName: "kube-api-access-nphcn") pod "a10ebd5e-3da5-4951-9a35-41b4d40fc930" (UID: "a10ebd5e-3da5-4951-9a35-41b4d40fc930"). InnerVolumeSpecName "kube-api-access-nphcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:03:10.903534 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:10.903498    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nphcn\" (UniqueName: \"kubernetes.io/projected/a10ebd5e-3da5-4951-9a35-41b4d40fc930-kube-api-access-nphcn\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:11.466303 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:11.466268    2564 generic.go:356] "Generic (PLEG): container finished" podID="a10ebd5e-3da5-4951-9a35-41b4d40fc930" containerID="23addd9c3a4577e782220735fdc6a60508e1e260ddb85ecd54a8fec023818c51" exitCode=137
Sep 17 22:03:11.466719 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:11.466313    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7820-c6d2c7"
Sep 17 22:03:11.466719 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:11.466369    2564 scope.go:117] "RemoveContainer" containerID="23addd9c3a4577e782220735fdc6a60508e1e260ddb85ecd54a8fec023818c51"
Sep 17 22:03:11.476533 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:11.476506    2564 scope.go:117] "RemoveContainer" containerID="23addd9c3a4577e782220735fdc6a60508e1e260ddb85ecd54a8fec023818c51"
Sep 17 22:03:11.476881 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:03:11.476860    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23addd9c3a4577e782220735fdc6a60508e1e260ddb85ecd54a8fec023818c51\": container with ID starting with 23addd9c3a4577e782220735fdc6a60508e1e260ddb85ecd54a8fec023818c51 not found: ID does not exist" containerID="23addd9c3a4577e782220735fdc6a60508e1e260ddb85ecd54a8fec023818c51"
Sep 17 22:03:11.476961 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:11.476893    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23addd9c3a4577e782220735fdc6a60508e1e260ddb85ecd54a8fec023818c51"} err="failed to get container status \"23addd9c3a4577e782220735fdc6a60508e1e260ddb85ecd54a8fec023818c51\": rpc error: code = NotFound desc = could not find container \"23addd9c3a4577e782220735fdc6a60508e1e260ddb85ecd54a8fec023818c51\": container with ID starting with 23addd9c3a4577e782220735fdc6a60508e1e260ddb85ecd54a8fec023818c51 not found: ID does not exist"
Sep 17 22:03:11.612552 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:11.612529    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d5c73-4872g"
Sep 17 22:03:11.710555 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:11.710522    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a33d84f6-937e-44b5-9d4d-a219fe650e19-maas-api-service-ca\") pod \"a33d84f6-937e-44b5-9d4d-a219fe650e19\" (UID: \"a33d84f6-937e-44b5-9d4d-a219fe650e19\") "
Sep 17 22:03:11.710901 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:11.710871    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33d84f6-937e-44b5-9d4d-a219fe650e19-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a33d84f6-937e-44b5-9d4d-a219fe650e19" (UID: "a33d84f6-937e-44b5-9d4d-a219fe650e19"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:03:11.812128 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:11.812040    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a33d84f6-937e-44b5-9d4d-a219fe650e19-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:12.474452 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:12.474421    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d5c73-4872g"
Sep 17 22:03:12.474908 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:12.474414    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d5c73-4872g" event={"ID":"a33d84f6-937e-44b5-9d4d-a219fe650e19","Type":"ContainerDied","Data":"6bb5766b719d57c613a2cac07388b885f67ad7746386dbc29932180d45b255fe"}
Sep 17 22:03:12.474908 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:12.474555    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bb5766b719d57c613a2cac07388b885f67ad7746386dbc29932180d45b255fe"
Sep 17 22:03:12.754059 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:12.753979    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a10ebd5e-3da5-4951-9a35-41b4d40fc930" path="/var/lib/kubelet/pods/a10ebd5e-3da5-4951-9a35-41b4d40fc930/volumes"
Sep 17 22:03:13.489760 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.489574    2564 generic.go:356] "Generic (PLEG): container finished" podID="b112866e-0590-411f-bc6f-8f5fe79ed928" containerID="9d756e8853db6437f6f042ec036f96ec0313e549aa865c4a0b78bd9555ee952a" exitCode=0
Sep 17 22:03:13.489760 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.489631    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv" event={"ID":"b112866e-0590-411f-bc6f-8f5fe79ed928","Type":"ContainerDied","Data":"9d756e8853db6437f6f042ec036f96ec0313e549aa865c4a0b78bd9555ee952a"}
Sep 17 22:03:13.489760 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.489722    2564 scope.go:117] "RemoveContainer" containerID="ec4d54500c65c479b8a1b3bfc5abc4ba8fd061943b3453ee25bd28e8f74e7449"
Sep 17 22:03:13.560970 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.560943    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 22:03:13.729496 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.729466    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9qrt\" (UniqueName: \"kubernetes.io/projected/b112866e-0590-411f-bc6f-8f5fe79ed928-kube-api-access-n9qrt\") pod \"b112866e-0590-411f-bc6f-8f5fe79ed928\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") "
Sep 17 22:03:13.729704 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.729508    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-istio-envoy\") pod \"b112866e-0590-411f-bc6f-8f5fe79ed928\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") "
Sep 17 22:03:13.729704 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.729543    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-workload-certs\") pod \"b112866e-0590-411f-bc6f-8f5fe79ed928\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") "
Sep 17 22:03:13.729704 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.729591    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-credential-socket\") pod \"b112866e-0590-411f-bc6f-8f5fe79ed928\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") "
Sep 17 22:03:13.729704 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.729633    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-istio-data\") pod \"b112866e-0590-411f-bc6f-8f5fe79ed928\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") "
Sep 17 22:03:13.729704 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.729655    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b112866e-0590-411f-bc6f-8f5fe79ed928-istio-podinfo\") pod \"b112866e-0590-411f-bc6f-8f5fe79ed928\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") "
Sep 17 22:03:13.729704 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.729685    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b112866e-0590-411f-bc6f-8f5fe79ed928-istiod-ca-cert\") pod \"b112866e-0590-411f-bc6f-8f5fe79ed928\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") "
Sep 17 22:03:13.730035 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.729715    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-workload-socket\") pod \"b112866e-0590-411f-bc6f-8f5fe79ed928\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") "
Sep 17 22:03:13.730035 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.729740    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b112866e-0590-411f-bc6f-8f5fe79ed928-istio-token\") pod \"b112866e-0590-411f-bc6f-8f5fe79ed928\" (UID: \"b112866e-0590-411f-bc6f-8f5fe79ed928\") "
Sep 17 22:03:13.730035 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.729943    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b112866e-0590-411f-bc6f-8f5fe79ed928" (UID: "b112866e-0590-411f-bc6f-8f5fe79ed928"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:03:13.730035 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.729961    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b112866e-0590-411f-bc6f-8f5fe79ed928" (UID: "b112866e-0590-411f-bc6f-8f5fe79ed928"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:03:13.730297 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.730090    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-workload-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:13.730297 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.730112    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-credential-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:13.730297 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.730214    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b112866e-0590-411f-bc6f-8f5fe79ed928" (UID: "b112866e-0590-411f-bc6f-8f5fe79ed928"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:03:13.730629 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.730139    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b112866e-0590-411f-bc6f-8f5fe79ed928-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b112866e-0590-411f-bc6f-8f5fe79ed928" (UID: "b112866e-0590-411f-bc6f-8f5fe79ed928"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:03:13.730629 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.730327    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-istio-data" (OuterVolumeSpecName: "istio-data") pod "b112866e-0590-411f-bc6f-8f5fe79ed928" (UID: "b112866e-0590-411f-bc6f-8f5fe79ed928"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:03:13.731910 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.731881    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b112866e-0590-411f-bc6f-8f5fe79ed928-kube-api-access-n9qrt" (OuterVolumeSpecName: "kube-api-access-n9qrt") pod "b112866e-0590-411f-bc6f-8f5fe79ed928" (UID: "b112866e-0590-411f-bc6f-8f5fe79ed928"). InnerVolumeSpecName "kube-api-access-n9qrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:03:13.732052 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.732032    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b112866e-0590-411f-bc6f-8f5fe79ed928" (UID: "b112866e-0590-411f-bc6f-8f5fe79ed928"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:03:13.732527 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.732501    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b112866e-0590-411f-bc6f-8f5fe79ed928-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b112866e-0590-411f-bc6f-8f5fe79ed928" (UID: "b112866e-0590-411f-bc6f-8f5fe79ed928"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 22:03:13.732527 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.732516    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b112866e-0590-411f-bc6f-8f5fe79ed928-istio-token" (OuterVolumeSpecName: "istio-token") pod "b112866e-0590-411f-bc6f-8f5fe79ed928" (UID: "b112866e-0590-411f-bc6f-8f5fe79ed928"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:03:13.831463 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.831418    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9qrt\" (UniqueName: \"kubernetes.io/projected/b112866e-0590-411f-bc6f-8f5fe79ed928-kube-api-access-n9qrt\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:13.831463 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.831455    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-istio-envoy\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:13.831463 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.831464    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-istio-data\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:13.831463 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.831472    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b112866e-0590-411f-bc6f-8f5fe79ed928-istio-podinfo\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:13.831463 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.831480    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b112866e-0590-411f-bc6f-8f5fe79ed928-istiod-ca-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:13.831780 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.831489    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b112866e-0590-411f-bc6f-8f5fe79ed928-workload-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:13.831780 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:13.831497    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b112866e-0590-411f-bc6f-8f5fe79ed928-istio-token\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:14.497710 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:14.497681    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"
Sep 17 22:03:14.498209 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:14.497729    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv" event={"ID":"b112866e-0590-411f-bc6f-8f5fe79ed928","Type":"ContainerDied","Data":"6fe94a9b205cf7d6b8e6bbf65e7b158eaff4ebb6740837ca7d61935d7e02e75f"}
Sep 17 22:03:14.498209 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:14.497790    2564 scope.go:117] "RemoveContainer" containerID="9d756e8853db6437f6f042ec036f96ec0313e549aa865c4a0b78bd9555ee952a"
Sep 17 22:03:14.526082 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:14.526052    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"]
Sep 17 22:03:14.533563 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:14.533534    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6a6c05-openshift-default-58c744bcb8-44jbv"]
Sep 17 22:03:14.754226 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:14.754111    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b112866e-0590-411f-bc6f-8f5fe79ed928" path="/var/lib/kubelet/pods/b112866e-0590-411f-bc6f-8f5fe79ed928/volumes"
Sep 17 22:03:17.525579 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:17.525536    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8" podUID="17b98d06-7b3f-4910-971c-7acbc508ada8" containerName="main" containerID="cri-o://f3591fa42854aa99a30cb3da760d9e0425c9bcc4fdc478035a82971b4debecf3" gracePeriod=45
Sep 17 22:03:17.777310 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:17.777234    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:03:17.875457 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:17.875423    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b98d06-7b3f-4910-971c-7acbc508ada8-dshm\") pod \"17b98d06-7b3f-4910-971c-7acbc508ada8\" (UID: \"17b98d06-7b3f-4910-971c-7acbc508ada8\") "
Sep 17 22:03:17.875649 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:17.875493    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b98d06-7b3f-4910-971c-7acbc508ada8-tls-certs\") pod \"17b98d06-7b3f-4910-971c-7acbc508ada8\" (UID: \"17b98d06-7b3f-4910-971c-7acbc508ada8\") "
Sep 17 22:03:17.875649 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:17.875538    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b98d06-7b3f-4910-971c-7acbc508ada8-home\") pod \"17b98d06-7b3f-4910-971c-7acbc508ada8\" (UID: \"17b98d06-7b3f-4910-971c-7acbc508ada8\") "
Sep 17 22:03:17.875649 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:17.875575    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfcjp\" (UniqueName: \"kubernetes.io/projected/17b98d06-7b3f-4910-971c-7acbc508ada8-kube-api-access-wfcjp\") pod \"17b98d06-7b3f-4910-971c-7acbc508ada8\" (UID: \"17b98d06-7b3f-4910-971c-7acbc508ada8\") "
Sep 17 22:03:17.875837 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:17.875687    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b98d06-7b3f-4910-971c-7acbc508ada8-tmp-dir\") pod \"17b98d06-7b3f-4910-971c-7acbc508ada8\" (UID: \"17b98d06-7b3f-4910-971c-7acbc508ada8\") "
Sep 17 22:03:17.875837 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:17.875721    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b98d06-7b3f-4910-971c-7acbc508ada8-model-cache\") pod \"17b98d06-7b3f-4910-971c-7acbc508ada8\" (UID: \"17b98d06-7b3f-4910-971c-7acbc508ada8\") "
Sep 17 22:03:17.876041 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:17.875984    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b98d06-7b3f-4910-971c-7acbc508ada8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "17b98d06-7b3f-4910-971c-7acbc508ada8" (UID: "17b98d06-7b3f-4910-971c-7acbc508ada8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:03:17.876218 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:17.876193    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b98d06-7b3f-4910-971c-7acbc508ada8-model-cache" (OuterVolumeSpecName: "model-cache") pod "17b98d06-7b3f-4910-971c-7acbc508ada8" (UID: "17b98d06-7b3f-4910-971c-7acbc508ada8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:03:17.876386 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:17.876360    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b98d06-7b3f-4910-971c-7acbc508ada8-home" (OuterVolumeSpecName: "home") pod "17b98d06-7b3f-4910-971c-7acbc508ada8" (UID: "17b98d06-7b3f-4910-971c-7acbc508ada8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:03:17.877987 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:17.877949    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b98d06-7b3f-4910-971c-7acbc508ada8-dshm" (OuterVolumeSpecName: "dshm") pod "17b98d06-7b3f-4910-971c-7acbc508ada8" (UID: "17b98d06-7b3f-4910-971c-7acbc508ada8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:03:17.878095 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:17.878005    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b98d06-7b3f-4910-971c-7acbc508ada8-kube-api-access-wfcjp" (OuterVolumeSpecName: "kube-api-access-wfcjp") pod "17b98d06-7b3f-4910-971c-7acbc508ada8" (UID: "17b98d06-7b3f-4910-971c-7acbc508ada8"). InnerVolumeSpecName "kube-api-access-wfcjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:03:17.878095 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:17.878012    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17b98d06-7b3f-4910-971c-7acbc508ada8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "17b98d06-7b3f-4910-971c-7acbc508ada8" (UID: "17b98d06-7b3f-4910-971c-7acbc508ada8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:03:17.977016 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:17.976980    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b98d06-7b3f-4910-971c-7acbc508ada8-tmp-dir\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:17.977016 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:17.977007    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b98d06-7b3f-4910-971c-7acbc508ada8-model-cache\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:17.977016 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:17.977016    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b98d06-7b3f-4910-971c-7acbc508ada8-dshm\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:17.977016 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:17.977024    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b98d06-7b3f-4910-971c-7acbc508ada8-tls-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:17.977327 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:17.977031    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b98d06-7b3f-4910-971c-7acbc508ada8-home\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:17.977327 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:17.977041    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfcjp\" (UniqueName: \"kubernetes.io/projected/17b98d06-7b3f-4910-971c-7acbc508ada8-kube-api-access-wfcjp\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:18.391742 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.391712    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw"]
Sep 17 22:03:18.392158 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.392145    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b112866e-0590-411f-bc6f-8f5fe79ed928" containerName="istio-proxy"
Sep 17 22:03:18.392229 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.392161    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b112866e-0590-411f-bc6f-8f5fe79ed928" containerName="istio-proxy"
Sep 17 22:03:18.392229 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.392184    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a33d84f6-937e-44b5-9d4d-a219fe650e19" containerName="revoke-keys"
Sep 17 22:03:18.392229 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.392190    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a33d84f6-937e-44b5-9d4d-a219fe650e19" containerName="revoke-keys"
Sep 17 22:03:18.392229 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.392206    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b112866e-0590-411f-bc6f-8f5fe79ed928" containerName="istio-proxy"
Sep 17 22:03:18.392229 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.392212    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b112866e-0590-411f-bc6f-8f5fe79ed928" containerName="istio-proxy"
Sep 17 22:03:18.392229 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.392219    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17b98d06-7b3f-4910-971c-7acbc508ada8" containerName="main"
Sep 17 22:03:18.392229 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.392224    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="17b98d06-7b3f-4910-971c-7acbc508ada8" containerName="main"
Sep 17 22:03:18.392432 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.392288    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b112866e-0590-411f-bc6f-8f5fe79ed928" containerName="istio-proxy"
Sep 17 22:03:18.392432 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.392296    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="17b98d06-7b3f-4910-971c-7acbc508ada8" containerName="main"
Sep 17 22:03:18.392432 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.392303    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b112866e-0590-411f-bc6f-8f5fe79ed928" containerName="istio-proxy"
Sep 17 22:03:18.392432 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.392311    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a33d84f6-937e-44b5-9d4d-a219fe650e19" containerName="revoke-keys"
Sep 17 22:03:18.395601 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.395587    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw"
Sep 17 22:03:18.398450 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.398427    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d8ea1ea7-dockercfg-cm7ss\""
Sep 17 22:03:18.398569 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.398463    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d8ea1ea7\""
Sep 17 22:03:18.411541 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.411513    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw"]
Sep 17 22:03:18.511657 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.511620    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr"]
Sep 17 22:03:18.515216 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.515195    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr"
Sep 17 22:03:18.526164 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.526128    2564 generic.go:356] "Generic (PLEG): container finished" podID="17b98d06-7b3f-4910-971c-7acbc508ada8" containerID="f3591fa42854aa99a30cb3da760d9e0425c9bcc4fdc478035a82971b4debecf3" exitCode=0
Sep 17 22:03:18.526568 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.526193    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"
Sep 17 22:03:18.526568 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.526213    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8" event={"ID":"17b98d06-7b3f-4910-971c-7acbc508ada8","Type":"ContainerDied","Data":"f3591fa42854aa99a30cb3da760d9e0425c9bcc4fdc478035a82971b4debecf3"}
Sep 17 22:03:18.526568 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.526251    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8" event={"ID":"17b98d06-7b3f-4910-971c-7acbc508ada8","Type":"ContainerDied","Data":"bc013374046d1a3d495f78cc00e36fce5f3f74683b6be21751432c430b5ecc29"}
Sep 17 22:03:18.526568 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.526268    2564 scope.go:117] "RemoveContainer" containerID="f3591fa42854aa99a30cb3da760d9e0425c9bcc4fdc478035a82971b4debecf3"
Sep 17 22:03:18.534839 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.534811    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr"]
Sep 17 22:03:18.544607 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.544583    2564 scope.go:117] "RemoveContainer" containerID="f3591fa42854aa99a30cb3da760d9e0425c9bcc4fdc478035a82971b4debecf3"
Sep 17 22:03:18.545057 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:03:18.545036    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3591fa42854aa99a30cb3da760d9e0425c9bcc4fdc478035a82971b4debecf3\": container with ID starting with f3591fa42854aa99a30cb3da760d9e0425c9bcc4fdc478035a82971b4debecf3 not found: ID does not exist" containerID="f3591fa42854aa99a30cb3da760d9e0425c9bcc4fdc478035a82971b4debecf3"
Sep 17 22:03:18.545122 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.545070    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3591fa42854aa99a30cb3da760d9e0425c9bcc4fdc478035a82971b4debecf3"} err="failed to get container status \"f3591fa42854aa99a30cb3da760d9e0425c9bcc4fdc478035a82971b4debecf3\": rpc error: code = NotFound desc = could not find container \"f3591fa42854aa99a30cb3da760d9e0425c9bcc4fdc478035a82971b4debecf3\": container with ID starting with f3591fa42854aa99a30cb3da760d9e0425c9bcc4fdc478035a82971b4debecf3 not found: ID does not exist"
Sep 17 22:03:18.565056 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.565026    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"]
Sep 17 22:03:18.576453 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.576425    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-2d1bcf/noten-model-2d1bcf-kserve-59ff6798bc-mkcc8"]
Sep 17 22:03:18.582216 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.582194    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vjc4\" (UniqueName: \"kubernetes.io/projected/ab75f792-2ee3-49a0-8951-bf0adcf77c6f-kube-api-access-9vjc4\") pod \"payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw\" (UID: \"ab75f792-2ee3-49a0-8951-bf0adcf77c6f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw"
Sep 17 22:03:18.582278 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.582239    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab75f792-2ee3-49a0-8951-bf0adcf77c6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw\" (UID: \"ab75f792-2ee3-49a0-8951-bf0adcf77c6f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw"
Sep 17 22:03:18.683295 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.683257    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b6db624-85d6-4d20-b837-936b778ad264-plugins-config-volume\") pod \"payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr\" (UID: \"9b6db624-85d6-4d20-b837-936b778ad264\") " pod="openshift-ingress/payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr"
Sep 17 22:03:18.683490 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.683321    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vjc4\" (UniqueName: \"kubernetes.io/projected/ab75f792-2ee3-49a0-8951-bf0adcf77c6f-kube-api-access-9vjc4\") pod \"payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw\" (UID: \"ab75f792-2ee3-49a0-8951-bf0adcf77c6f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw"
Sep 17 22:03:18.683490 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.683368    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn26b\" (UniqueName: \"kubernetes.io/projected/9b6db624-85d6-4d20-b837-936b778ad264-kube-api-access-cn26b\") pod \"payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr\" (UID: \"9b6db624-85d6-4d20-b837-936b778ad264\") " pod="openshift-ingress/payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr"
Sep 17 22:03:18.683566 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.683481    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab75f792-2ee3-49a0-8951-bf0adcf77c6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw\" (UID: \"ab75f792-2ee3-49a0-8951-bf0adcf77c6f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw"
Sep 17 22:03:18.684196 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.684157    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab75f792-2ee3-49a0-8951-bf0adcf77c6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw\" (UID: \"ab75f792-2ee3-49a0-8951-bf0adcf77c6f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw"
Sep 17 22:03:18.696446 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.696422    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vjc4\" (UniqueName: \"kubernetes.io/projected/ab75f792-2ee3-49a0-8951-bf0adcf77c6f-kube-api-access-9vjc4\") pod \"payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw\" (UID: \"ab75f792-2ee3-49a0-8951-bf0adcf77c6f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw"
Sep 17 22:03:18.706285 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.706260    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw"
Sep 17 22:03:18.756849 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.756818    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17b98d06-7b3f-4910-971c-7acbc508ada8" path="/var/lib/kubelet/pods/17b98d06-7b3f-4910-971c-7acbc508ada8/volumes"
Sep 17 22:03:18.784779 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.784744    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn26b\" (UniqueName: \"kubernetes.io/projected/9b6db624-85d6-4d20-b837-936b778ad264-kube-api-access-cn26b\") pod \"payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr\" (UID: \"9b6db624-85d6-4d20-b837-936b778ad264\") " pod="openshift-ingress/payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr"
Sep 17 22:03:18.784958 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.784864    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b6db624-85d6-4d20-b837-936b778ad264-plugins-config-volume\") pod \"payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr\" (UID: \"9b6db624-85d6-4d20-b837-936b778ad264\") " pod="openshift-ingress/payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr"
Sep 17 22:03:18.785568 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.785543    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b6db624-85d6-4d20-b837-936b778ad264-plugins-config-volume\") pod \"payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr\" (UID: \"9b6db624-85d6-4d20-b837-936b778ad264\") " pod="openshift-ingress/payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr"
Sep 17 22:03:18.799010 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.798958    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn26b\" (UniqueName: \"kubernetes.io/projected/9b6db624-85d6-4d20-b837-936b778ad264-kube-api-access-cn26b\") pod \"payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr\" (UID: \"9b6db624-85d6-4d20-b837-936b778ad264\") " pod="openshift-ingress/payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr"
Sep 17 22:03:18.826705 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.826672    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr"
Sep 17 22:03:18.988694 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:18.988663    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw"]
Sep 17 22:03:18.989893 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:03:18.989863    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab75f792_2ee3_49a0_8951_bf0adcf77c6f.slice/crio-dddbe7ea5888ab46f837e39a9cf10e1da07b5e891f0de042e3530d9009d45e73 WatchSource:0}: Error finding container dddbe7ea5888ab46f837e39a9cf10e1da07b5e891f0de042e3530d9009d45e73: Status 404 returned error can't find the container with id dddbe7ea5888ab46f837e39a9cf10e1da07b5e891f0de042e3530d9009d45e73
Sep 17 22:03:19.113455 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:19.113392    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr"]
Sep 17 22:03:19.115873 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:03:19.115830    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b6db624_85d6_4d20_b837_936b778ad264.slice/crio-8bf54a799ff684147da47e6485e4a0307262c9866409a8d45f28b8fd9bc294ee WatchSource:0}: Error finding container 8bf54a799ff684147da47e6485e4a0307262c9866409a8d45f28b8fd9bc294ee: Status 404 returned error can't find the container with id 8bf54a799ff684147da47e6485e4a0307262c9866409a8d45f28b8fd9bc294ee
Sep 17 22:03:19.534865 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:19.534831    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw" event={"ID":"ab75f792-2ee3-49a0-8951-bf0adcf77c6f","Type":"ContainerStarted","Data":"3ba8e4ba126cf60d5975b8d9930b8cc1c8e0a569f5dbfdd2ee9119e633135c30"}
Sep 17 22:03:19.535330 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:19.534889    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw" event={"ID":"ab75f792-2ee3-49a0-8951-bf0adcf77c6f","Type":"ContainerStarted","Data":"dddbe7ea5888ab46f837e39a9cf10e1da07b5e891f0de042e3530d9009d45e73"}
Sep 17 22:03:19.535330 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:19.534992    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw"
Sep 17 22:03:19.536766 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:19.536730    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr" event={"ID":"9b6db624-85d6-4d20-b837-936b778ad264","Type":"ContainerStarted","Data":"5a30af954a73963b4abe9df6f8eb6acc38d03c21eeb66296c49f017338163a21"}
Sep 17 22:03:19.536894 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:19.536780    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr" event={"ID":"9b6db624-85d6-4d20-b837-936b778ad264","Type":"ContainerStarted","Data":"8bf54a799ff684147da47e6485e4a0307262c9866409a8d45f28b8fd9bc294ee"}
Sep 17 22:03:19.536894 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:19.536812    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr"
Sep 17 22:03:19.559656 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:19.559600    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw" podStartSLOduration=1.559582581 podStartE2EDuration="1.559582581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:03:19.558390599 +0000 UTC m=+1315.376716912" watchObservedRunningTime="2026-09-17 22:03:19.559582581 +0000 UTC m=+1315.377908892"
Sep 17 22:03:19.580144 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:19.580089    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr" podStartSLOduration=1.580071585 podStartE2EDuration="1.580071585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:03:19.577531226 +0000 UTC m=+1315.395857539" watchObservedRunningTime="2026-09-17 22:03:19.580071585 +0000 UTC m=+1315.398397902"
Sep 17 22:03:21.563330 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:21.563298    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3d5c73-29828040-php7j"]
Sep 17 22:03:21.566337 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:21.566315    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3d5c73-29828040-php7j"]
Sep 17 22:03:21.601183 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:21.601135    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf"]
Sep 17 22:03:21.601452 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:21.601422    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf" podUID="104b5a96-2d3c-40b1-be05-8f44d1619aef" containerName="payload-processing" containerID="cri-o://9abc11a74300e12fbb721168e4d681358b969ae5baa22e8a16c0ecbfa8748bcd" gracePeriod=30
Sep 17 22:03:21.871260 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:21.871237    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf"
Sep 17 22:03:21.879194 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:21.879145    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d5c73-4872g"]
Sep 17 22:03:21.881340 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:21.881321    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d5c73-4872g"]
Sep 17 22:03:21.914580 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:21.914554    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74l5d\" (UniqueName: \"kubernetes.io/projected/104b5a96-2d3c-40b1-be05-8f44d1619aef-kube-api-access-74l5d\") pod \"104b5a96-2d3c-40b1-be05-8f44d1619aef\" (UID: \"104b5a96-2d3c-40b1-be05-8f44d1619aef\") "
Sep 17 22:03:21.914732 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:21.914618    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/104b5a96-2d3c-40b1-be05-8f44d1619aef-plugins-config-volume\") pod \"104b5a96-2d3c-40b1-be05-8f44d1619aef\" (UID: \"104b5a96-2d3c-40b1-be05-8f44d1619aef\") "
Sep 17 22:03:21.914970 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:21.914949    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104b5a96-2d3c-40b1-be05-8f44d1619aef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "104b5a96-2d3c-40b1-be05-8f44d1619aef" (UID: "104b5a96-2d3c-40b1-be05-8f44d1619aef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:03:21.916876 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:21.916848    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104b5a96-2d3c-40b1-be05-8f44d1619aef-kube-api-access-74l5d" (OuterVolumeSpecName: "kube-api-access-74l5d") pod "104b5a96-2d3c-40b1-be05-8f44d1619aef" (UID: "104b5a96-2d3c-40b1-be05-8f44d1619aef"). InnerVolumeSpecName "kube-api-access-74l5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:03:22.015603 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:22.015560    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74l5d\" (UniqueName: \"kubernetes.io/projected/104b5a96-2d3c-40b1-be05-8f44d1619aef-kube-api-access-74l5d\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:22.015603 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:22.015595    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/104b5a96-2d3c-40b1-be05-8f44d1619aef-plugins-config-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:03:22.557162 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:22.557124    2564 generic.go:356] "Generic (PLEG): container finished" podID="104b5a96-2d3c-40b1-be05-8f44d1619aef" containerID="9abc11a74300e12fbb721168e4d681358b969ae5baa22e8a16c0ecbfa8748bcd" exitCode=0
Sep 17 22:03:22.557346 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:22.557242    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf"
Sep 17 22:03:22.557346 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:22.557277    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf" event={"ID":"104b5a96-2d3c-40b1-be05-8f44d1619aef","Type":"ContainerDied","Data":"9abc11a74300e12fbb721168e4d681358b969ae5baa22e8a16c0ecbfa8748bcd"}
Sep 17 22:03:22.557346 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:22.557340    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf" event={"ID":"104b5a96-2d3c-40b1-be05-8f44d1619aef","Type":"ContainerDied","Data":"d60d9da78eef087cfd2b4a3ab3846324ca03f14627c86a2583dc7100bf28d878"}
Sep 17 22:03:22.557469 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:22.557364    2564 scope.go:117] "RemoveContainer" containerID="9abc11a74300e12fbb721168e4d681358b969ae5baa22e8a16c0ecbfa8748bcd"
Sep 17 22:03:22.569961 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:22.569938    2564 scope.go:117] "RemoveContainer" containerID="9abc11a74300e12fbb721168e4d681358b969ae5baa22e8a16c0ecbfa8748bcd"
Sep 17 22:03:22.570500 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:03:22.570480    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9abc11a74300e12fbb721168e4d681358b969ae5baa22e8a16c0ecbfa8748bcd\": container with ID starting with 9abc11a74300e12fbb721168e4d681358b969ae5baa22e8a16c0ecbfa8748bcd not found: ID does not exist" containerID="9abc11a74300e12fbb721168e4d681358b969ae5baa22e8a16c0ecbfa8748bcd"
Sep 17 22:03:22.570569 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:22.570510    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9abc11a74300e12fbb721168e4d681358b969ae5baa22e8a16c0ecbfa8748bcd"} err="failed to get container status \"9abc11a74300e12fbb721168e4d681358b969ae5baa22e8a16c0ecbfa8748bcd\": rpc error: code = NotFound desc = could not find container \"9abc11a74300e12fbb721168e4d681358b969ae5baa22e8a16c0ecbfa8748bcd\": container with ID starting with 9abc11a74300e12fbb721168e4d681358b969ae5baa22e8a16c0ecbfa8748bcd not found: ID does not exist"
Sep 17 22:03:22.642871 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:22.642843    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf"]
Sep 17 22:03:22.660935 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:22.660904    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3d5c73-669bb597d9-7l6lf"]
Sep 17 22:03:22.754551 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:22.754515    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="104b5a96-2d3c-40b1-be05-8f44d1619aef" path="/var/lib/kubelet/pods/104b5a96-2d3c-40b1-be05-8f44d1619aef/volumes"
Sep 17 22:03:22.754909 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:22.754883    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a33d84f6-937e-44b5-9d4d-a219fe650e19" path="/var/lib/kubelet/pods/a33d84f6-937e-44b5-9d4d-a219fe650e19/volumes"
Sep 17 22:03:22.755206 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:22.755192    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c523d0e7-283a-4668-b776-59cfe4800cbf" path="/var/lib/kubelet/pods/c523d0e7-283a-4668-b776-59cfe4800cbf/volumes"
Sep 17 22:03:24.820942 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:24.820906    2564 scope.go:117] "RemoveContainer" containerID="a449acbf13910db6387f67b665854eacf0671b2e48610971524c59cc87e67279"
Sep 17 22:03:30.545872 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:30.545841    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr"
Sep 17 22:03:30.546275 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:30.545909    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw"
Sep 17 22:03:43.557964 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.557932    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"]
Sep 17 22:03:43.559037 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.558399    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104b5a96-2d3c-40b1-be05-8f44d1619aef" containerName="payload-processing"
Sep 17 22:03:43.559037 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.558414    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="104b5a96-2d3c-40b1-be05-8f44d1619aef" containerName="payload-processing"
Sep 17 22:03:43.559037 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.558490    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="104b5a96-2d3c-40b1-be05-8f44d1619aef" containerName="payload-processing"
Sep 17 22:03:43.563312 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.563292    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:03:43.566690 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.566665    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f996ae\"/\"distinct-2-w0-f996ae-kserve-self-signed-certs\""
Sep 17 22:03:43.573448 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.573428    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"]
Sep 17 22:03:43.623612 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.623533    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-tls-certs\") pod \"distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl\" (UID: \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\") " pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:03:43.623612 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.623590    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-model-cache\") pod \"distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl\" (UID: \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\") " pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:03:43.623878 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.623624    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-dshm\") pod \"distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl\" (UID: \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\") " pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:03:43.623878 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.623663    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-home\") pod \"distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl\" (UID: \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\") " pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:03:43.623878 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.623792    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-tmp-dir\") pod \"distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl\" (UID: \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\") " pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:03:43.624020 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.623886    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjxhp\" (UniqueName: \"kubernetes.io/projected/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-kube-api-access-xjxhp\") pod \"distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl\" (UID: \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\") " pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:03:43.724670 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.724592    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-tmp-dir\") pod \"distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl\" (UID: \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\") " pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:03:43.724867 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.724716    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjxhp\" (UniqueName: \"kubernetes.io/projected/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-kube-api-access-xjxhp\") pod \"distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl\" (UID: \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\") " pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:03:43.724867 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.724783    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-tls-certs\") pod \"distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl\" (UID: \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\") " pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:03:43.724867 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.724815    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-model-cache\") pod \"distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl\" (UID: \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\") " pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:03:43.724867 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.724844    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-dshm\") pod \"distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl\" (UID: \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\") " pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:03:43.725079 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.724880    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-home\") pod \"distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl\" (UID: \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\") " pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:03:43.725079 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.725035    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-tmp-dir\") pod \"distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl\" (UID: \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\") " pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:03:43.725411 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.725382    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-home\") pod \"distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl\" (UID: \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\") " pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:03:43.725550 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.725500    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-model-cache\") pod \"distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl\" (UID: \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\") " pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:03:43.727866 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.727841    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-dshm\") pod \"distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl\" (UID: \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\") " pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:03:43.728037 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.728021    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-tls-certs\") pod \"distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl\" (UID: \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\") " pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:03:43.735070 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.735044    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjxhp\" (UniqueName: \"kubernetes.io/projected/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-kube-api-access-xjxhp\") pod \"distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl\" (UID: \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\") " pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:03:43.873641 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:43.873545    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:03:44.151978 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:03:44.151949    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c33695c_bc26_49f7_9d90_4b75b5cdaf81.slice/crio-aeb5058a72ea99f1508fa84c41da75ad1052b560322be58a85f88d8d99cd7d36 WatchSource:0}: Error finding container aeb5058a72ea99f1508fa84c41da75ad1052b560322be58a85f88d8d99cd7d36: Status 404 returned error can't find the container with id aeb5058a72ea99f1508fa84c41da75ad1052b560322be58a85f88d8d99cd7d36
Sep 17 22:03:44.152633 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:44.152601    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"]
Sep 17 22:03:44.694218 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:44.694178    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl" event={"ID":"9c33695c-bc26-49f7-9d90-4b75b5cdaf81","Type":"ContainerStarted","Data":"48288f49dee257ed2ad53884c0d34f1e015145fc77c8d7a98d365c2a1872f75c"}
Sep 17 22:03:44.694650 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:44.694228    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl" event={"ID":"9c33695c-bc26-49f7-9d90-4b75b5cdaf81","Type":"ContainerStarted","Data":"aeb5058a72ea99f1508fa84c41da75ad1052b560322be58a85f88d8d99cd7d36"}
Sep 17 22:03:44.719738 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:44.719684    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl" podStartSLOduration=1.71966627 podStartE2EDuration="1.71966627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:03:44.716045754 +0000 UTC m=+1340.534372066" watchObservedRunningTime="2026-09-17 22:03:44.71966627 +0000 UTC m=+1340.537992628"
Sep 17 22:03:53.874343 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:53.874245    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:03:53.874343 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:53.874302    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:03:53.887618 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:53.887587    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:03:54.768106 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:54.768076    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:03:57.263949 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.263917    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"]
Sep 17 22:03:57.266614 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.266597    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:03:57.269720 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.269693    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d8ea1ea7\"/\"kube-root-ca.crt\""
Sep 17 22:03:57.269867 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.269696    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d8ea1ea7\"/\"openshift-service-ca.crt\""
Sep 17 22:03:57.269867 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.269817    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d8ea1ea7\"/\"default-dockercfg-j7nzh\""
Sep 17 22:03:57.270907 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.270883    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d8ea1ea7\"/\"e2e-lifecycle-model-d8ea1ea7-kserve-self-signed-certs\""
Sep 17 22:03:57.277034 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.277012    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"]
Sep 17 22:03:57.360723 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.360690    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65mml\" (UniqueName: \"kubernetes.io/projected/7e37c011-804d-405c-8cb6-791cce8e2f2e-kube-api-access-65mml\") pod \"e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82\" (UID: \"7e37c011-804d-405c-8cb6-791cce8e2f2e\") " pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:03:57.360871 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.360734    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e37c011-804d-405c-8cb6-791cce8e2f2e-dshm\") pod \"e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82\" (UID: \"7e37c011-804d-405c-8cb6-791cce8e2f2e\") " pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:03:57.360871 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.360765    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e37c011-804d-405c-8cb6-791cce8e2f2e-home\") pod \"e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82\" (UID: \"7e37c011-804d-405c-8cb6-791cce8e2f2e\") " pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:03:57.360871 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.360821    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e37c011-804d-405c-8cb6-791cce8e2f2e-tls-certs\") pod \"e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82\" (UID: \"7e37c011-804d-405c-8cb6-791cce8e2f2e\") " pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:03:57.360992 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.360933    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e37c011-804d-405c-8cb6-791cce8e2f2e-tmp-dir\") pod \"e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82\" (UID: \"7e37c011-804d-405c-8cb6-791cce8e2f2e\") " pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:03:57.360992 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.360968    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e37c011-804d-405c-8cb6-791cce8e2f2e-model-cache\") pod \"e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82\" (UID: \"7e37c011-804d-405c-8cb6-791cce8e2f2e\") " pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:03:57.462298 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.462252    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e37c011-804d-405c-8cb6-791cce8e2f2e-tmp-dir\") pod \"e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82\" (UID: \"7e37c011-804d-405c-8cb6-791cce8e2f2e\") " pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:03:57.462454 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.462307    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e37c011-804d-405c-8cb6-791cce8e2f2e-model-cache\") pod \"e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82\" (UID: \"7e37c011-804d-405c-8cb6-791cce8e2f2e\") " pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:03:57.462454 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.462350    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65mml\" (UniqueName: \"kubernetes.io/projected/7e37c011-804d-405c-8cb6-791cce8e2f2e-kube-api-access-65mml\") pod \"e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82\" (UID: \"7e37c011-804d-405c-8cb6-791cce8e2f2e\") " pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:03:57.462454 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.462387    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e37c011-804d-405c-8cb6-791cce8e2f2e-dshm\") pod \"e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82\" (UID: \"7e37c011-804d-405c-8cb6-791cce8e2f2e\") " pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:03:57.462454 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.462417    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e37c011-804d-405c-8cb6-791cce8e2f2e-home\") pod \"e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82\" (UID: \"7e37c011-804d-405c-8cb6-791cce8e2f2e\") " pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:03:57.462454 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.462443    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e37c011-804d-405c-8cb6-791cce8e2f2e-tls-certs\") pod \"e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82\" (UID: \"7e37c011-804d-405c-8cb6-791cce8e2f2e\") " pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:03:57.462770 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.462746    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e37c011-804d-405c-8cb6-791cce8e2f2e-tmp-dir\") pod \"e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82\" (UID: \"7e37c011-804d-405c-8cb6-791cce8e2f2e\") " pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:03:57.462833 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.462777    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e37c011-804d-405c-8cb6-791cce8e2f2e-model-cache\") pod \"e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82\" (UID: \"7e37c011-804d-405c-8cb6-791cce8e2f2e\") " pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:03:57.462896 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.462836    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e37c011-804d-405c-8cb6-791cce8e2f2e-home\") pod \"e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82\" (UID: \"7e37c011-804d-405c-8cb6-791cce8e2f2e\") " pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:03:57.464791 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.464769    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e37c011-804d-405c-8cb6-791cce8e2f2e-dshm\") pod \"e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82\" (UID: \"7e37c011-804d-405c-8cb6-791cce8e2f2e\") " pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:03:57.465076 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.465056    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e37c011-804d-405c-8cb6-791cce8e2f2e-tls-certs\") pod \"e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82\" (UID: \"7e37c011-804d-405c-8cb6-791cce8e2f2e\") " pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:03:57.471469 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.471450    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65mml\" (UniqueName: \"kubernetes.io/projected/7e37c011-804d-405c-8cb6-791cce8e2f2e-kube-api-access-65mml\") pod \"e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82\" (UID: \"7e37c011-804d-405c-8cb6-791cce8e2f2e\") " pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:03:57.577809 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.577726    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:03:57.856748 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:57.856714    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"]
Sep 17 22:03:57.857154 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:03:57.857129    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e37c011_804d_405c_8cb6_791cce8e2f2e.slice/crio-c1566b4fcac0acc46e1649b581eb42b79946c32b1b544cf11fa0964fd7f530ec WatchSource:0}: Error finding container c1566b4fcac0acc46e1649b581eb42b79946c32b1b544cf11fa0964fd7f530ec: Status 404 returned error can't find the container with id c1566b4fcac0acc46e1649b581eb42b79946c32b1b544cf11fa0964fd7f530ec
Sep 17 22:03:58.779815 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:58.779778    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82" event={"ID":"7e37c011-804d-405c-8cb6-791cce8e2f2e","Type":"ContainerStarted","Data":"f26a2cf93a857b346503ee0e1493654c6848b6bdf9ece8447746ae849a49177b"}
Sep 17 22:03:58.779815 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:58.779819    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82" event={"ID":"7e37c011-804d-405c-8cb6-791cce8e2f2e","Type":"ContainerStarted","Data":"c1566b4fcac0acc46e1649b581eb42b79946c32b1b544cf11fa0964fd7f530ec"}
Sep 17 22:03:58.802739 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:03:58.802677    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82" podStartSLOduration=1.80266167 podStartE2EDuration="1.80266167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:03:58.800103237 +0000 UTC m=+1354.618429544" watchObservedRunningTime="2026-09-17 22:03:58.80266167 +0000 UTC m=+1354.620987982"
Sep 17 22:04:05.622240 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.622198    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"]
Sep 17 22:04:05.624872 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.624854    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.628217 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.628192    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-8f8dad-openshift-default-dockercfg-mtmhh\""
Sep 17 22:04:05.642065 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.642040    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istio-token\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.642237 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.642082    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istio-podinfo\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.642237 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.642114    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-workload-certs\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.642237 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.642164    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-credential-socket\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.642237 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.642213    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dzxp\" (UniqueName: \"kubernetes.io/projected/c4e2600c-dda1-43fb-8b0f-a9128366c25d-kube-api-access-6dzxp\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.642475 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.642250    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istiod-ca-cert\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.642475 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.642300    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istio-data\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.642475 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.642364    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istio-envoy\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.642475 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.642402    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-workload-socket\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.642475 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.642463    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"]
Sep 17 22:04:05.743717 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.743684    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istiod-ca-cert\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.743899 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.743740    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istio-data\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.743899 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.743773    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istio-envoy\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.743899 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.743792    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-workload-socket\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.743899 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.743829    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istio-token\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.743899 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.743873    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istio-podinfo\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.744156 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.743912    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-workload-certs\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.744156 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.743944    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-credential-socket\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.744156 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.743976    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dzxp\" (UniqueName: \"kubernetes.io/projected/c4e2600c-dda1-43fb-8b0f-a9128366c25d-kube-api-access-6dzxp\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.744347 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.744290    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istio-data\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.744347 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.744335    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-workload-certs\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.744476 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.744456    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-credential-socket\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.744541 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.744515    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istiod-ca-cert\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.744612 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.744571    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-workload-socket\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.746492 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.746467    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istio-envoy\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.746811 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.746792    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istio-podinfo\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.753095 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.753071    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istio-token\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.753228 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.753112    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dzxp\" (UniqueName: \"kubernetes.io/projected/c4e2600c-dda1-43fb-8b0f-a9128366c25d-kube-api-access-6dzxp\") pod \"e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") " pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:05.936770 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:05.936730    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:06.224479 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:06.224444    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"]
Sep 17 22:04:06.225649 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:04:06.225623    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e2600c_dda1_43fb_8b0f_a9128366c25d.slice/crio-33d3b9550b05d0327660fa3eb1610db063b32a1605895c105b7b6481b3efa496 WatchSource:0}: Error finding container 33d3b9550b05d0327660fa3eb1610db063b32a1605895c105b7b6481b3efa496: Status 404 returned error can't find the container with id 33d3b9550b05d0327660fa3eb1610db063b32a1605895c105b7b6481b3efa496
Sep 17 22:04:06.228448 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:06.228413    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 22:04:06.228539 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:06.228498    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 22:04:06.228539 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:06.228527    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 22:04:06.828054 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:06.828011    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs" event={"ID":"c4e2600c-dda1-43fb-8b0f-a9128366c25d","Type":"ContainerStarted","Data":"33de90316c6f9bf770b6e5f9feb16c5bb2d103b0fb857cbd4dffca7cc14eb5aa"}
Sep 17 22:04:06.828508 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:06.828078    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs" event={"ID":"c4e2600c-dda1-43fb-8b0f-a9128366c25d","Type":"ContainerStarted","Data":"33d3b9550b05d0327660fa3eb1610db063b32a1605895c105b7b6481b3efa496"}
Sep 17 22:04:06.898295 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:06.898241    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs" podStartSLOduration=1.8982234550000001 podStartE2EDuration="1.898223455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:04:06.895779888 +0000 UTC m=+1362.714106200" watchObservedRunningTime="2026-09-17 22:04:06.898223455 +0000 UTC m=+1362.716549767"
Sep 17 22:04:06.937709 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:06.937670    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:07.578283 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:07.578246    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:04:07.578474 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:07.578298    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:04:07.592379 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:07.592350    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:04:07.850513 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:07.850413    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:04:07.942572 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:07.942542    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:08.842143 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:08.842107    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:08.843025 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:08.843004    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:14.633212 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:14.633164    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc"]
Sep 17 22:04:14.635641 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:14.635618    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc"
Sep 17 22:04:14.638611 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:14.638588    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-8f8dad\""
Sep 17 22:04:14.638895 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:14.638867    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-8f8dad-dockercfg-2nngh\""
Sep 17 22:04:14.650709 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:14.650684    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc"]
Sep 17 22:04:14.735776 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:14.735747    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8172d37e-4071-4548-9ac1-c6378f7e6398-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc\" (UID: \"8172d37e-4071-4548-9ac1-c6378f7e6398\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc"
Sep 17 22:04:14.736012 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:14.735787    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ml4c\" (UniqueName: \"kubernetes.io/projected/8172d37e-4071-4548-9ac1-c6378f7e6398-kube-api-access-8ml4c\") pod \"payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc\" (UID: \"8172d37e-4071-4548-9ac1-c6378f7e6398\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc"
Sep 17 22:04:14.836359 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:14.836322    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ml4c\" (UniqueName: \"kubernetes.io/projected/8172d37e-4071-4548-9ac1-c6378f7e6398-kube-api-access-8ml4c\") pod \"payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc\" (UID: \"8172d37e-4071-4548-9ac1-c6378f7e6398\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc"
Sep 17 22:04:14.836587 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:14.836469    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8172d37e-4071-4548-9ac1-c6378f7e6398-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc\" (UID: \"8172d37e-4071-4548-9ac1-c6378f7e6398\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc"
Sep 17 22:04:14.837153 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:14.837131    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8172d37e-4071-4548-9ac1-c6378f7e6398-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc\" (UID: \"8172d37e-4071-4548-9ac1-c6378f7e6398\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc"
Sep 17 22:04:14.846510 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:14.846491    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ml4c\" (UniqueName: \"kubernetes.io/projected/8172d37e-4071-4548-9ac1-c6378f7e6398-kube-api-access-8ml4c\") pod \"payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc\" (UID: \"8172d37e-4071-4548-9ac1-c6378f7e6398\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc"
Sep 17 22:04:14.949822 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:14.949790    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc"
Sep 17 22:04:15.235660 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:15.235632    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc"]
Sep 17 22:04:15.237244 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:04:15.237213    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8172d37e_4071_4548_9ac1_c6378f7e6398.slice/crio-124f1d43decdfa548b366ee3d4c1b6688540580332746fddf710126c23502b1f WatchSource:0}: Error finding container 124f1d43decdfa548b366ee3d4c1b6688540580332746fddf710126c23502b1f: Status 404 returned error can't find the container with id 124f1d43decdfa548b366ee3d4c1b6688540580332746fddf710126c23502b1f
Sep 17 22:04:15.884955 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:15.884921    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc" event={"ID":"8172d37e-4071-4548-9ac1-c6378f7e6398","Type":"ContainerStarted","Data":"ed4e4bf6ae62653b2a8d569341b098626325f09b4e7ce1aa077b96fedc8055ee"}
Sep 17 22:04:15.885389 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:15.884976    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc" event={"ID":"8172d37e-4071-4548-9ac1-c6378f7e6398","Type":"ContainerStarted","Data":"124f1d43decdfa548b366ee3d4c1b6688540580332746fddf710126c23502b1f"}
Sep 17 22:04:15.885389 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:15.885061    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc"
Sep 17 22:04:15.917152 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:15.917102    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc" podStartSLOduration=1.917083681 podStartE2EDuration="1.917083681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:04:15.915578509 +0000 UTC m=+1371.733904823" watchObservedRunningTime="2026-09-17 22:04:15.917083681 +0000 UTC m=+1371.735410056"
Sep 17 22:04:24.877336 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:24.877302    2564 scope.go:117] "RemoveContainer" containerID="8b93f226cf90f2fbba7c031c56381b4739d49b392eba2e17430af5afff751008"
Sep 17 22:04:24.886272 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:24.886251    2564 scope.go:117] "RemoveContainer" containerID="8d0a36c34df710fb4f18138ab836da8f537cc6fee976d9ec2bc7a22b72edcf6a"
Sep 17 22:04:26.894343 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:26.894308    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc"
Sep 17 22:04:37.030999 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:37.030906    2564 generic.go:356] "Generic (PLEG): container finished" podID="c4e2600c-dda1-43fb-8b0f-a9128366c25d" containerID="33de90316c6f9bf770b6e5f9feb16c5bb2d103b0fb857cbd4dffca7cc14eb5aa" exitCode=0
Sep 17 22:04:37.031470 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:37.030988    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs" event={"ID":"c4e2600c-dda1-43fb-8b0f-a9128366c25d","Type":"ContainerDied","Data":"33de90316c6f9bf770b6e5f9feb16c5bb2d103b0fb857cbd4dffca7cc14eb5aa"}
Sep 17 22:04:37.031519 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:37.031479    2564 scope.go:117] "RemoveContainer" containerID="33de90316c6f9bf770b6e5f9feb16c5bb2d103b0fb857cbd4dffca7cc14eb5aa"
Sep 17 22:04:37.032711 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:37.032680    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 22:04:37.032763 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:37.032754    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 22:04:37.032808 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:37.032796    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 22:04:38.039816 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:38.039782    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs" event={"ID":"c4e2600c-dda1-43fb-8b0f-a9128366c25d","Type":"ContainerStarted","Data":"34e223c021db7ef2839cde6c3968835006c7118e85e220c3d2b8e37965087f8e"}
Sep 17 22:04:38.842424 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:38.842387    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:38.937409 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:38.937370    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:38.938758 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:38.938730    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 17 22:04:38.938834 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:38.938811    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs" podUID="c4e2600c-dda1-43fb-8b0f-a9128366c25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 17 22:04:39.938001 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:39.937962    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 17 22:04:39.938615 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:39.938580    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs" podUID="c4e2600c-dda1-43fb-8b0f-a9128366c25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 17 22:04:40.937553 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:40.937514    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 17 22:04:40.937732 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:40.937578    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs" podUID="c4e2600c-dda1-43fb-8b0f-a9128366c25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 17 22:04:41.937462 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:41.937419    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 17 22:04:41.937940 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:41.937481    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs" podUID="c4e2600c-dda1-43fb-8b0f-a9128366c25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 17 22:04:42.937510 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:42.937477    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 17 22:04:42.937877 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:42.937530    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs" podUID="c4e2600c-dda1-43fb-8b0f-a9128366c25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 17 22:04:43.941744 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:43.941712    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:43.942768 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:43.942750    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:04:50.326852 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:50.326814    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h"]
Sep 17 22:04:50.331754 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:50.331727    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h"
Sep 17 22:04:50.335014 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:50.334982    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-35b181\""
Sep 17 22:04:50.335161 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:50.335050    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-35b181-dockercfg-7r6tc\""
Sep 17 22:04:50.345591 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:50.345563    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h"]
Sep 17 22:04:50.400657 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:50.400621    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc757bd4-4892-426a-b53b-6b7bab26d3cd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h\" (UID: \"dc757bd4-4892-426a-b53b-6b7bab26d3cd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h"
Sep 17 22:04:50.400851 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:50.400668    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzld4\" (UniqueName: \"kubernetes.io/projected/dc757bd4-4892-426a-b53b-6b7bab26d3cd-kube-api-access-nzld4\") pod \"payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h\" (UID: \"dc757bd4-4892-426a-b53b-6b7bab26d3cd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h"
Sep 17 22:04:50.502335 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:50.502289    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc757bd4-4892-426a-b53b-6b7bab26d3cd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h\" (UID: \"dc757bd4-4892-426a-b53b-6b7bab26d3cd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h"
Sep 17 22:04:50.502335 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:50.502339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzld4\" (UniqueName: \"kubernetes.io/projected/dc757bd4-4892-426a-b53b-6b7bab26d3cd-kube-api-access-nzld4\") pod \"payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h\" (UID: \"dc757bd4-4892-426a-b53b-6b7bab26d3cd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h"
Sep 17 22:04:50.502990 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:50.502969    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc757bd4-4892-426a-b53b-6b7bab26d3cd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h\" (UID: \"dc757bd4-4892-426a-b53b-6b7bab26d3cd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h"
Sep 17 22:04:50.511627 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:50.511591    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzld4\" (UniqueName: \"kubernetes.io/projected/dc757bd4-4892-426a-b53b-6b7bab26d3cd-kube-api-access-nzld4\") pod \"payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h\" (UID: \"dc757bd4-4892-426a-b53b-6b7bab26d3cd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h"
Sep 17 22:04:50.642012 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:50.641913    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h"
Sep 17 22:04:50.946916 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:50.946882    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h"]
Sep 17 22:04:50.950066 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:04:50.950033    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc757bd4_4892_426a_b53b_6b7bab26d3cd.slice/crio-175b848bd600500d587c87f7e3418a4ca10eff51dc9e0d2a80b80fe0ba92072c WatchSource:0}: Error finding container 175b848bd600500d587c87f7e3418a4ca10eff51dc9e0d2a80b80fe0ba92072c: Status 404 returned error can't find the container with id 175b848bd600500d587c87f7e3418a4ca10eff51dc9e0d2a80b80fe0ba92072c
Sep 17 22:04:51.120646 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:51.120612    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h" event={"ID":"dc757bd4-4892-426a-b53b-6b7bab26d3cd","Type":"ContainerStarted","Data":"239cdf3b742e6c9feb6159c92f691cf2034f36ffe6648b4b462c44b59448789d"}
Sep 17 22:04:51.120862 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:51.120674    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h" event={"ID":"dc757bd4-4892-426a-b53b-6b7bab26d3cd","Type":"ContainerStarted","Data":"175b848bd600500d587c87f7e3418a4ca10eff51dc9e0d2a80b80fe0ba92072c"}
Sep 17 22:04:51.120862 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:51.120799    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h"
Sep 17 22:04:51.140280 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:04:51.140215    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h" podStartSLOduration=1.140194637 podStartE2EDuration="1.140194637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:04:51.13795678 +0000 UTC m=+1406.956283116" watchObservedRunningTime="2026-09-17 22:04:51.140194637 +0000 UTC m=+1406.958520950"
Sep 17 22:05:02.128668 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:02.128634    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h"
Sep 17 22:05:23.211808 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:23.211771    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8ea1ea7-q5krb"]
Sep 17 22:05:23.215353 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:23.215336    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8ea1ea7-q5krb"
Sep 17 22:05:23.218319 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:23.218297    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 22:05:23.218612 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:23.218599    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmdbp\""
Sep 17 22:05:23.228394 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:23.228370    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8ea1ea7-q5krb"]
Sep 17 22:05:23.237384 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:23.237355    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac72a86b-b5b3-4a2b-8d58-c7a3b118e178-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d8ea1ea7-q5krb\" (UID: \"ac72a86b-b5b3-4a2b-8d58-c7a3b118e178\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8ea1ea7-q5krb"
Sep 17 22:05:23.338365 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:23.338326    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac72a86b-b5b3-4a2b-8d58-c7a3b118e178-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d8ea1ea7-q5krb\" (UID: \"ac72a86b-b5b3-4a2b-8d58-c7a3b118e178\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8ea1ea7-q5krb"
Sep 17 22:05:23.338994 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:23.338969    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac72a86b-b5b3-4a2b-8d58-c7a3b118e178-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d8ea1ea7-q5krb\" (UID: \"ac72a86b-b5b3-4a2b-8d58-c7a3b118e178\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8ea1ea7-q5krb"
Sep 17 22:05:23.526375 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:23.526287    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8ea1ea7-q5krb"
Sep 17 22:05:23.806087 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:23.806059    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8ea1ea7-q5krb"]
Sep 17 22:05:23.806627 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:05:23.806607    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac72a86b_b5b3_4a2b_8d58_c7a3b118e178.slice/crio-1a57bf80a4593ffa31dc1338e3898b8dd24e41f41aa06d91c34b49d22627a107 WatchSource:0}: Error finding container 1a57bf80a4593ffa31dc1338e3898b8dd24e41f41aa06d91c34b49d22627a107: Status 404 returned error can't find the container with id 1a57bf80a4593ffa31dc1338e3898b8dd24e41f41aa06d91c34b49d22627a107
Sep 17 22:05:24.324725 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:24.324681    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8ea1ea7-q5krb" event={"ID":"ac72a86b-b5b3-4a2b-8d58-c7a3b118e178","Type":"ContainerStarted","Data":"cb141663b0fd36bf7076ab7b840f1cb7f169c182034b28738bf32c00a9bd9ac4"}
Sep 17 22:05:24.325111 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:24.324773    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8ea1ea7-q5krb" event={"ID":"ac72a86b-b5b3-4a2b-8d58-c7a3b118e178","Type":"ContainerStarted","Data":"1a57bf80a4593ffa31dc1338e3898b8dd24e41f41aa06d91c34b49d22627a107"}
Sep 17 22:05:24.344486 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:24.344392    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8ea1ea7-q5krb" podStartSLOduration=1.344378086 podStartE2EDuration="1.344378086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:05:24.342233829 +0000 UTC m=+1440.160560141" watchObservedRunningTime="2026-09-17 22:05:24.344378086 +0000 UTC m=+1440.162704398"
Sep 17 22:05:24.914087 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:24.914052    2564 scope.go:117] "RemoveContainer" containerID="2d64368dfac8de3bb105526cded05736f140d7df304f6856a33f561f4f248df2"
Sep 17 22:05:25.336109 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:25.336027    2564 generic.go:356] "Generic (PLEG): container finished" podID="ac72a86b-b5b3-4a2b-8d58-c7a3b118e178" containerID="cb141663b0fd36bf7076ab7b840f1cb7f169c182034b28738bf32c00a9bd9ac4" exitCode=0
Sep 17 22:05:25.336501 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:25.336111    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8ea1ea7-q5krb" event={"ID":"ac72a86b-b5b3-4a2b-8d58-c7a3b118e178","Type":"ContainerDied","Data":"cb141663b0fd36bf7076ab7b840f1cb7f169c182034b28738bf32c00a9bd9ac4"}
Sep 17 22:05:26.502805 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:26.502778    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8ea1ea7-q5krb"
Sep 17 22:05:26.564602 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:26.564567    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac72a86b-b5b3-4a2b-8d58-c7a3b118e178-maas-api-service-ca\") pod \"ac72a86b-b5b3-4a2b-8d58-c7a3b118e178\" (UID: \"ac72a86b-b5b3-4a2b-8d58-c7a3b118e178\") "
Sep 17 22:05:26.564872 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:26.564848    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac72a86b-b5b3-4a2b-8d58-c7a3b118e178-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ac72a86b-b5b3-4a2b-8d58-c7a3b118e178" (UID: "ac72a86b-b5b3-4a2b-8d58-c7a3b118e178"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:05:26.565031 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:26.565013    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac72a86b-b5b3-4a2b-8d58-c7a3b118e178-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:05:27.349395 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:27.349362    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8ea1ea7-q5krb"
Sep 17 22:05:27.349561 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:27.349390    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8ea1ea7-q5krb" event={"ID":"ac72a86b-b5b3-4a2b-8d58-c7a3b118e178","Type":"ContainerDied","Data":"1a57bf80a4593ffa31dc1338e3898b8dd24e41f41aa06d91c34b49d22627a107"}
Sep 17 22:05:27.349561 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:27.349445    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a57bf80a4593ffa31dc1338e3898b8dd24e41f41aa06d91c34b49d22627a107"
Sep 17 22:05:38.983560 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:38.983521    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr"]
Sep 17 22:05:38.984054 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:38.983813    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr" podUID="9b6db624-85d6-4d20-b837-936b778ad264" containerName="payload-processing" containerID="cri-o://5a30af954a73963b4abe9df6f8eb6acc38d03c21eeb66296c49f017338163a21" gracePeriod=30
Sep 17 22:05:38.994914 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:38.994882    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw"]
Sep 17 22:05:38.995373 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:38.995321    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw" podUID="ab75f792-2ee3-49a0-8951-bf0adcf77c6f" containerName="payload-pre-processing" containerID="cri-o://3ba8e4ba126cf60d5975b8d9930b8cc1c8e0a569f5dbfdd2ee9119e633135c30" gracePeriod=30
Sep 17 22:05:39.183634 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.183600    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8ea1ea7-q5krb"]
Sep 17 22:05:39.188959 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.188931    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8ea1ea7-q5krb"]
Sep 17 22:05:39.281505 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.281471    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr"
Sep 17 22:05:39.284495 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.284473    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw"
Sep 17 22:05:39.397725 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.397690    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vjc4\" (UniqueName: \"kubernetes.io/projected/ab75f792-2ee3-49a0-8951-bf0adcf77c6f-kube-api-access-9vjc4\") pod \"ab75f792-2ee3-49a0-8951-bf0adcf77c6f\" (UID: \"ab75f792-2ee3-49a0-8951-bf0adcf77c6f\") "
Sep 17 22:05:39.397725 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.397729    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab75f792-2ee3-49a0-8951-bf0adcf77c6f-plugins-config-volume\") pod \"ab75f792-2ee3-49a0-8951-bf0adcf77c6f\" (UID: \"ab75f792-2ee3-49a0-8951-bf0adcf77c6f\") "
Sep 17 22:05:39.397979 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.397879    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cn26b\" (UniqueName: \"kubernetes.io/projected/9b6db624-85d6-4d20-b837-936b778ad264-kube-api-access-cn26b\") pod \"9b6db624-85d6-4d20-b837-936b778ad264\" (UID: \"9b6db624-85d6-4d20-b837-936b778ad264\") "
Sep 17 22:05:39.397979 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.397955    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b6db624-85d6-4d20-b837-936b778ad264-plugins-config-volume\") pod \"9b6db624-85d6-4d20-b837-936b778ad264\" (UID: \"9b6db624-85d6-4d20-b837-936b778ad264\") "
Sep 17 22:05:39.398123 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.398093    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab75f792-2ee3-49a0-8951-bf0adcf77c6f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ab75f792-2ee3-49a0-8951-bf0adcf77c6f" (UID: "ab75f792-2ee3-49a0-8951-bf0adcf77c6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:05:39.398335 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.398307    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b6db624-85d6-4d20-b837-936b778ad264-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9b6db624-85d6-4d20-b837-936b778ad264" (UID: "9b6db624-85d6-4d20-b837-936b778ad264"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:05:39.398452 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.398432    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab75f792-2ee3-49a0-8951-bf0adcf77c6f-plugins-config-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:05:39.398658 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.398457    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b6db624-85d6-4d20-b837-936b778ad264-plugins-config-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:05:39.400421 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.400362    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab75f792-2ee3-49a0-8951-bf0adcf77c6f-kube-api-access-9vjc4" (OuterVolumeSpecName: "kube-api-access-9vjc4") pod "ab75f792-2ee3-49a0-8951-bf0adcf77c6f" (UID: "ab75f792-2ee3-49a0-8951-bf0adcf77c6f"). InnerVolumeSpecName "kube-api-access-9vjc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:05:39.400529 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.400493    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b6db624-85d6-4d20-b837-936b778ad264-kube-api-access-cn26b" (OuterVolumeSpecName: "kube-api-access-cn26b") pod "9b6db624-85d6-4d20-b837-936b778ad264" (UID: "9b6db624-85d6-4d20-b837-936b778ad264"). InnerVolumeSpecName "kube-api-access-cn26b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:05:39.429857 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.429822    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b6db624-85d6-4d20-b837-936b778ad264" containerID="5a30af954a73963b4abe9df6f8eb6acc38d03c21eeb66296c49f017338163a21" exitCode=0
Sep 17 22:05:39.430015 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.429887    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr"
Sep 17 22:05:39.430015 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.429910    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr" event={"ID":"9b6db624-85d6-4d20-b837-936b778ad264","Type":"ContainerDied","Data":"5a30af954a73963b4abe9df6f8eb6acc38d03c21eeb66296c49f017338163a21"}
Sep 17 22:05:39.430015 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.429965    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr" event={"ID":"9b6db624-85d6-4d20-b837-936b778ad264","Type":"ContainerDied","Data":"8bf54a799ff684147da47e6485e4a0307262c9866409a8d45f28b8fd9bc294ee"}
Sep 17 22:05:39.430015 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.429988    2564 scope.go:117] "RemoveContainer" containerID="5a30af954a73963b4abe9df6f8eb6acc38d03c21eeb66296c49f017338163a21"
Sep 17 22:05:39.431516 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.431490    2564 generic.go:356] "Generic (PLEG): container finished" podID="ab75f792-2ee3-49a0-8951-bf0adcf77c6f" containerID="3ba8e4ba126cf60d5975b8d9930b8cc1c8e0a569f5dbfdd2ee9119e633135c30" exitCode=0
Sep 17 22:05:39.431638 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.431534    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw"
Sep 17 22:05:39.431638 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.431534    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw" event={"ID":"ab75f792-2ee3-49a0-8951-bf0adcf77c6f","Type":"ContainerDied","Data":"3ba8e4ba126cf60d5975b8d9930b8cc1c8e0a569f5dbfdd2ee9119e633135c30"}
Sep 17 22:05:39.431638 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.431585    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw" event={"ID":"ab75f792-2ee3-49a0-8951-bf0adcf77c6f","Type":"ContainerDied","Data":"dddbe7ea5888ab46f837e39a9cf10e1da07b5e891f0de042e3530d9009d45e73"}
Sep 17 22:05:39.439399 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.439382    2564 scope.go:117] "RemoveContainer" containerID="5a30af954a73963b4abe9df6f8eb6acc38d03c21eeb66296c49f017338163a21"
Sep 17 22:05:39.439727 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:05:39.439705    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a30af954a73963b4abe9df6f8eb6acc38d03c21eeb66296c49f017338163a21\": container with ID starting with 5a30af954a73963b4abe9df6f8eb6acc38d03c21eeb66296c49f017338163a21 not found: ID does not exist" containerID="5a30af954a73963b4abe9df6f8eb6acc38d03c21eeb66296c49f017338163a21"
Sep 17 22:05:39.439826 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.439740    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a30af954a73963b4abe9df6f8eb6acc38d03c21eeb66296c49f017338163a21"} err="failed to get container status \"5a30af954a73963b4abe9df6f8eb6acc38d03c21eeb66296c49f017338163a21\": rpc error: code = NotFound desc = could not find container \"5a30af954a73963b4abe9df6f8eb6acc38d03c21eeb66296c49f017338163a21\": container with ID starting with 5a30af954a73963b4abe9df6f8eb6acc38d03c21eeb66296c49f017338163a21 not found: ID does not exist"
Sep 17 22:05:39.439826 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.439765    2564 scope.go:117] "RemoveContainer" containerID="3ba8e4ba126cf60d5975b8d9930b8cc1c8e0a569f5dbfdd2ee9119e633135c30"
Sep 17 22:05:39.449920 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.449896    2564 scope.go:117] "RemoveContainer" containerID="3ba8e4ba126cf60d5975b8d9930b8cc1c8e0a569f5dbfdd2ee9119e633135c30"
Sep 17 22:05:39.450289 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:05:39.450269    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ba8e4ba126cf60d5975b8d9930b8cc1c8e0a569f5dbfdd2ee9119e633135c30\": container with ID starting with 3ba8e4ba126cf60d5975b8d9930b8cc1c8e0a569f5dbfdd2ee9119e633135c30 not found: ID does not exist" containerID="3ba8e4ba126cf60d5975b8d9930b8cc1c8e0a569f5dbfdd2ee9119e633135c30"
Sep 17 22:05:39.450373 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.450298    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba8e4ba126cf60d5975b8d9930b8cc1c8e0a569f5dbfdd2ee9119e633135c30"} err="failed to get container status \"3ba8e4ba126cf60d5975b8d9930b8cc1c8e0a569f5dbfdd2ee9119e633135c30\": rpc error: code = NotFound desc = could not find container \"3ba8e4ba126cf60d5975b8d9930b8cc1c8e0a569f5dbfdd2ee9119e633135c30\": container with ID starting with 3ba8e4ba126cf60d5975b8d9930b8cc1c8e0a569f5dbfdd2ee9119e633135c30 not found: ID does not exist"
Sep 17 22:05:39.459120 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.459088    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr"]
Sep 17 22:05:39.463028 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.462994    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8ea1ea7-64fccdc845-fwzlr"]
Sep 17 22:05:39.475300 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.475274    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw"]
Sep 17 22:05:39.478952 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.478926    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8ea1ea7-7fb4fb44fd-kxxfw"]
Sep 17 22:05:39.499786 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.499747    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn26b\" (UniqueName: \"kubernetes.io/projected/9b6db624-85d6-4d20-b837-936b778ad264-kube-api-access-cn26b\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:05:39.499786 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:39.499780    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vjc4\" (UniqueName: \"kubernetes.io/projected/ab75f792-2ee3-49a0-8951-bf0adcf77c6f-kube-api-access-9vjc4\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:05:40.754089 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:40.754053    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b6db624-85d6-4d20-b837-936b778ad264" path="/var/lib/kubelet/pods/9b6db624-85d6-4d20-b837-936b778ad264/volumes"
Sep 17 22:05:40.754502 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:40.754463    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab75f792-2ee3-49a0-8951-bf0adcf77c6f" path="/var/lib/kubelet/pods/ab75f792-2ee3-49a0-8951-bf0adcf77c6f/volumes"
Sep 17 22:05:40.754757 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:40.754743    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac72a86b-b5b3-4a2b-8d58-c7a3b118e178" path="/var/lib/kubelet/pods/ac72a86b-b5b3-4a2b-8d58-c7a3b118e178/volumes"
Sep 17 22:05:47.393406 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:47.393365    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"]
Sep 17 22:05:54.272452 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.272408    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"]
Sep 17 22:05:54.273052 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.273036    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b6db624-85d6-4d20-b837-936b778ad264" containerName="payload-processing"
Sep 17 22:05:54.273130 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.273055    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b6db624-85d6-4d20-b837-936b778ad264" containerName="payload-processing"
Sep 17 22:05:54.273130 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.273081    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab75f792-2ee3-49a0-8951-bf0adcf77c6f" containerName="payload-pre-processing"
Sep 17 22:05:54.273130 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.273089    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab75f792-2ee3-49a0-8951-bf0adcf77c6f" containerName="payload-pre-processing"
Sep 17 22:05:54.273130 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.273101    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac72a86b-b5b3-4a2b-8d58-c7a3b118e178" containerName="revoke-keys"
Sep 17 22:05:54.273130 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.273111    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac72a86b-b5b3-4a2b-8d58-c7a3b118e178" containerName="revoke-keys"
Sep 17 22:05:54.273322 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.273228    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac72a86b-b5b3-4a2b-8d58-c7a3b118e178" containerName="revoke-keys"
Sep 17 22:05:54.273322 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.273244    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b6db624-85d6-4d20-b837-936b778ad264" containerName="payload-processing"
Sep 17 22:05:54.273322 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.273253    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab75f792-2ee3-49a0-8951-bf0adcf77c6f" containerName="payload-pre-processing"
Sep 17 22:05:54.277017 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.276997    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:05:54.280531 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.280513    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8f8dad\"/\"kube-root-ca.crt\""
Sep 17 22:05:54.281332 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.281311    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8f8dad\"/\"default-dockercfg-q655d\""
Sep 17 22:05:54.281922 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.281874    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8f8dad\"/\"openshift-service-ca.crt\""
Sep 17 22:05:54.281922 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.281881    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8f8dad\"/\"auth-test-model-8f8dad-kserve-self-signed-certs\""
Sep 17 22:05:54.288186 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.288138    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"]
Sep 17 22:05:54.452068 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.452033    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12d0714-cb27-46cf-bdca-25dec1611ce7-home\") pod \"auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp\" (UID: \"c12d0714-cb27-46cf-bdca-25dec1611ce7\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:05:54.452480 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.452101    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12d0714-cb27-46cf-bdca-25dec1611ce7-tls-certs\") pod \"auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp\" (UID: \"c12d0714-cb27-46cf-bdca-25dec1611ce7\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:05:54.452480 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.452212    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12d0714-cb27-46cf-bdca-25dec1611ce7-dshm\") pod \"auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp\" (UID: \"c12d0714-cb27-46cf-bdca-25dec1611ce7\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:05:54.452480 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.452305    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6pw2\" (UniqueName: \"kubernetes.io/projected/c12d0714-cb27-46cf-bdca-25dec1611ce7-kube-api-access-r6pw2\") pod \"auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp\" (UID: \"c12d0714-cb27-46cf-bdca-25dec1611ce7\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:05:54.452480 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.452368    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12d0714-cb27-46cf-bdca-25dec1611ce7-tmp-dir\") pod \"auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp\" (UID: \"c12d0714-cb27-46cf-bdca-25dec1611ce7\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:05:54.452480 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.452457    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12d0714-cb27-46cf-bdca-25dec1611ce7-model-cache\") pod \"auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp\" (UID: \"c12d0714-cb27-46cf-bdca-25dec1611ce7\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:05:54.554030 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.553936    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6pw2\" (UniqueName: \"kubernetes.io/projected/c12d0714-cb27-46cf-bdca-25dec1611ce7-kube-api-access-r6pw2\") pod \"auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp\" (UID: \"c12d0714-cb27-46cf-bdca-25dec1611ce7\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:05:54.554030 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.553999    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12d0714-cb27-46cf-bdca-25dec1611ce7-tmp-dir\") pod \"auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp\" (UID: \"c12d0714-cb27-46cf-bdca-25dec1611ce7\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:05:54.554311 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.554033    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12d0714-cb27-46cf-bdca-25dec1611ce7-model-cache\") pod \"auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp\" (UID: \"c12d0714-cb27-46cf-bdca-25dec1611ce7\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:05:54.554311 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.554080    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12d0714-cb27-46cf-bdca-25dec1611ce7-home\") pod \"auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp\" (UID: \"c12d0714-cb27-46cf-bdca-25dec1611ce7\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:05:54.554311 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.554118    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12d0714-cb27-46cf-bdca-25dec1611ce7-tls-certs\") pod \"auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp\" (UID: \"c12d0714-cb27-46cf-bdca-25dec1611ce7\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:05:54.554311 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.554133    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12d0714-cb27-46cf-bdca-25dec1611ce7-dshm\") pod \"auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp\" (UID: \"c12d0714-cb27-46cf-bdca-25dec1611ce7\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:05:54.554529 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.554474    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12d0714-cb27-46cf-bdca-25dec1611ce7-tmp-dir\") pod \"auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp\" (UID: \"c12d0714-cb27-46cf-bdca-25dec1611ce7\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:05:54.554529 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.554506    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12d0714-cb27-46cf-bdca-25dec1611ce7-home\") pod \"auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp\" (UID: \"c12d0714-cb27-46cf-bdca-25dec1611ce7\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:05:54.554621 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.554554    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12d0714-cb27-46cf-bdca-25dec1611ce7-model-cache\") pod \"auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp\" (UID: \"c12d0714-cb27-46cf-bdca-25dec1611ce7\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:05:54.556876 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.556853    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12d0714-cb27-46cf-bdca-25dec1611ce7-dshm\") pod \"auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp\" (UID: \"c12d0714-cb27-46cf-bdca-25dec1611ce7\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:05:54.557064 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.557048    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12d0714-cb27-46cf-bdca-25dec1611ce7-tls-certs\") pod \"auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp\" (UID: \"c12d0714-cb27-46cf-bdca-25dec1611ce7\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:05:54.574788 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.574765    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6pw2\" (UniqueName: \"kubernetes.io/projected/c12d0714-cb27-46cf-bdca-25dec1611ce7-kube-api-access-r6pw2\") pod \"auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp\" (UID: \"c12d0714-cb27-46cf-bdca-25dec1611ce7\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:05:54.593797 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.593766    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:05:54.891991 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:54.891903    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"]
Sep 17 22:05:54.900314 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:05:54.900278    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc12d0714_cb27_46cf_bdca_25dec1611ce7.slice/crio-146caaa7d97fa49daac69c6fa86b1f550fc68abb5ce4e45babb4729814c5c20c WatchSource:0}: Error finding container 146caaa7d97fa49daac69c6fa86b1f550fc68abb5ce4e45babb4729814c5c20c: Status 404 returned error can't find the container with id 146caaa7d97fa49daac69c6fa86b1f550fc68abb5ce4e45babb4729814c5c20c
Sep 17 22:05:55.531644 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:55.531611    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp" event={"ID":"c12d0714-cb27-46cf-bdca-25dec1611ce7","Type":"ContainerStarted","Data":"494b0048ca3e4d06b8fbb8e36a8ca40cdd440a65c72412e7f47c9678a3d1486d"}
Sep 17 22:05:55.532051 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:55.531657    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp" event={"ID":"c12d0714-cb27-46cf-bdca-25dec1611ce7","Type":"ContainerStarted","Data":"146caaa7d97fa49daac69c6fa86b1f550fc68abb5ce4e45babb4729814c5c20c"}
Sep 17 22:05:55.553160 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:05:55.553112    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp" podStartSLOduration=1.553096993 podStartE2EDuration="1.553096993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:05:55.551523256 +0000 UTC m=+1471.369849568" watchObservedRunningTime="2026-09-17 22:05:55.553096993 +0000 UTC m=+1471.371423305"
Sep 17 22:06:02.413349 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:02.413271    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82" podUID="7e37c011-804d-405c-8cb6-791cce8e2f2e" containerName="main" containerID="cri-o://f26a2cf93a857b346503ee0e1493654c6848b6bdf9ece8447746ae849a49177b" gracePeriod=45
Sep 17 22:06:02.573382 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:02.573343    2564 generic.go:356] "Generic (PLEG): container finished" podID="7e37c011-804d-405c-8cb6-791cce8e2f2e" containerID="f26a2cf93a857b346503ee0e1493654c6848b6bdf9ece8447746ae849a49177b" exitCode=0
Sep 17 22:06:02.573556 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:02.573420    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82" event={"ID":"7e37c011-804d-405c-8cb6-791cce8e2f2e","Type":"ContainerDied","Data":"f26a2cf93a857b346503ee0e1493654c6848b6bdf9ece8447746ae849a49177b"}
Sep 17 22:06:02.685061 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:02.685036    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:06:02.732297 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:02.732259    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e37c011-804d-405c-8cb6-791cce8e2f2e-tmp-dir\") pod \"7e37c011-804d-405c-8cb6-791cce8e2f2e\" (UID: \"7e37c011-804d-405c-8cb6-791cce8e2f2e\") "
Sep 17 22:06:02.732495 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:02.732315    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e37c011-804d-405c-8cb6-791cce8e2f2e-dshm\") pod \"7e37c011-804d-405c-8cb6-791cce8e2f2e\" (UID: \"7e37c011-804d-405c-8cb6-791cce8e2f2e\") "
Sep 17 22:06:02.732495 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:02.732398    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e37c011-804d-405c-8cb6-791cce8e2f2e-model-cache\") pod \"7e37c011-804d-405c-8cb6-791cce8e2f2e\" (UID: \"7e37c011-804d-405c-8cb6-791cce8e2f2e\") "
Sep 17 22:06:02.732495 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:02.732458    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65mml\" (UniqueName: \"kubernetes.io/projected/7e37c011-804d-405c-8cb6-791cce8e2f2e-kube-api-access-65mml\") pod \"7e37c011-804d-405c-8cb6-791cce8e2f2e\" (UID: \"7e37c011-804d-405c-8cb6-791cce8e2f2e\") "
Sep 17 22:06:02.732676 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:02.732507    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e37c011-804d-405c-8cb6-791cce8e2f2e-home\") pod \"7e37c011-804d-405c-8cb6-791cce8e2f2e\" (UID: \"7e37c011-804d-405c-8cb6-791cce8e2f2e\") "
Sep 17 22:06:02.732676 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:02.732538    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e37c011-804d-405c-8cb6-791cce8e2f2e-tls-certs\") pod \"7e37c011-804d-405c-8cb6-791cce8e2f2e\" (UID: \"7e37c011-804d-405c-8cb6-791cce8e2f2e\") "
Sep 17 22:06:02.732676 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:02.732616    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e37c011-804d-405c-8cb6-791cce8e2f2e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e37c011-804d-405c-8cb6-791cce8e2f2e" (UID: "7e37c011-804d-405c-8cb6-791cce8e2f2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:06:02.732838 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:02.732684    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e37c011-804d-405c-8cb6-791cce8e2f2e-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e37c011-804d-405c-8cb6-791cce8e2f2e" (UID: "7e37c011-804d-405c-8cb6-791cce8e2f2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:06:02.732900 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:02.732862    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e37c011-804d-405c-8cb6-791cce8e2f2e-home" (OuterVolumeSpecName: "home") pod "7e37c011-804d-405c-8cb6-791cce8e2f2e" (UID: "7e37c011-804d-405c-8cb6-791cce8e2f2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:06:02.732957 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:02.732919    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e37c011-804d-405c-8cb6-791cce8e2f2e-tmp-dir\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:06:02.732957 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:02.732938    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e37c011-804d-405c-8cb6-791cce8e2f2e-model-cache\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:06:02.734953 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:02.734929    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e37c011-804d-405c-8cb6-791cce8e2f2e-dshm" (OuterVolumeSpecName: "dshm") pod "7e37c011-804d-405c-8cb6-791cce8e2f2e" (UID: "7e37c011-804d-405c-8cb6-791cce8e2f2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:06:02.735090 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:02.735074    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e37c011-804d-405c-8cb6-791cce8e2f2e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e37c011-804d-405c-8cb6-791cce8e2f2e" (UID: "7e37c011-804d-405c-8cb6-791cce8e2f2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:06:02.735153 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:02.735125    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e37c011-804d-405c-8cb6-791cce8e2f2e-kube-api-access-65mml" (OuterVolumeSpecName: "kube-api-access-65mml") pod "7e37c011-804d-405c-8cb6-791cce8e2f2e" (UID: "7e37c011-804d-405c-8cb6-791cce8e2f2e"). InnerVolumeSpecName "kube-api-access-65mml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:06:02.833543 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:02.833498    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65mml\" (UniqueName: \"kubernetes.io/projected/7e37c011-804d-405c-8cb6-791cce8e2f2e-kube-api-access-65mml\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:06:02.833543 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:02.833534    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e37c011-804d-405c-8cb6-791cce8e2f2e-home\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:06:02.833543 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:02.833548    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e37c011-804d-405c-8cb6-791cce8e2f2e-tls-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:06:02.833543 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:02.833556    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e37c011-804d-405c-8cb6-791cce8e2f2e-dshm\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:06:03.580667 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:03.580640    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"
Sep 17 22:06:03.581144 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:03.580686    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82" event={"ID":"7e37c011-804d-405c-8cb6-791cce8e2f2e","Type":"ContainerDied","Data":"c1566b4fcac0acc46e1649b581eb42b79946c32b1b544cf11fa0964fd7f530ec"}
Sep 17 22:06:03.581144 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:03.580732    2564 scope.go:117] "RemoveContainer" containerID="f26a2cf93a857b346503ee0e1493654c6848b6bdf9ece8447746ae849a49177b"
Sep 17 22:06:03.606029 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:03.605992    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"]
Sep 17 22:06:03.611378 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:03.611352    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d8ea1ea7/e2e-lifecycle-model-d8ea1ea7-kserve-64f6ff755f-g7q82"]
Sep 17 22:06:04.594150 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:04.594112    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:06:04.594669 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:04.594208    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:06:04.607086 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:04.607055    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:06:04.754055 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:04.754022    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e37c011-804d-405c-8cb6-791cce8e2f2e" path="/var/lib/kubelet/pods/7e37c011-804d-405c-8cb6-791cce8e2f2e/volumes"
Sep 17 22:06:05.612337 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:05.612304    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:06:24.758083 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:24.758056    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdcnw_6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2/ovn-acl-logging/0.log"
Sep 17 22:06:24.758665 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:24.758125    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdcnw_6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2/ovn-acl-logging/0.log"
Sep 17 22:06:24.936931 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:24.936900    2564 scope.go:117] "RemoveContainer" containerID="89c7d5e15e535aa9de8f31c2f744fdfeff3f83e3b83e5d3e975d77931478407e"
Sep 17 22:06:24.945990 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:24.945971    2564 scope.go:117] "RemoveContainer" containerID="7f8ccbc18b6a86a2679e489b9bcf68ff46e8fa0fb1cf2235b3c694f823a9afc8"
Sep 17 22:06:32.739546 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.739509    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"]
Sep 17 22:06:32.740008 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.739962    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e37c011-804d-405c-8cb6-791cce8e2f2e" containerName="main"
Sep 17 22:06:32.740008 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.739985    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e37c011-804d-405c-8cb6-791cce8e2f2e" containerName="main"
Sep 17 22:06:32.740133 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.740095    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e37c011-804d-405c-8cb6-791cce8e2f2e" containerName="main"
Sep 17 22:06:32.744634 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.744617    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:06:32.748258 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.748230    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-dbe8a7\"/\"distinct-2-w4-dbe8a7-kserve-self-signed-certs\""
Sep 17 22:06:32.755775 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.755749    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"]
Sep 17 22:06:32.830586 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.830548    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96dccd7c-c657-4888-ad95-d65a36d269ca-model-cache\") pod \"distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl\" (UID: \"96dccd7c-c657-4888-ad95-d65a36d269ca\") " pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:06:32.830787 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.830594    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcvpj\" (UniqueName: \"kubernetes.io/projected/96dccd7c-c657-4888-ad95-d65a36d269ca-kube-api-access-mcvpj\") pod \"distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl\" (UID: \"96dccd7c-c657-4888-ad95-d65a36d269ca\") " pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:06:32.830787 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.830707    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96dccd7c-c657-4888-ad95-d65a36d269ca-home\") pod \"distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl\" (UID: \"96dccd7c-c657-4888-ad95-d65a36d269ca\") " pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:06:32.830787 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.830750    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96dccd7c-c657-4888-ad95-d65a36d269ca-tls-certs\") pod \"distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl\" (UID: \"96dccd7c-c657-4888-ad95-d65a36d269ca\") " pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:06:32.830979 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.830831    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96dccd7c-c657-4888-ad95-d65a36d269ca-tmp-dir\") pod \"distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl\" (UID: \"96dccd7c-c657-4888-ad95-d65a36d269ca\") " pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:06:32.830979 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.830920    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96dccd7c-c657-4888-ad95-d65a36d269ca-dshm\") pod \"distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl\" (UID: \"96dccd7c-c657-4888-ad95-d65a36d269ca\") " pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:06:32.932206 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.932155    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96dccd7c-c657-4888-ad95-d65a36d269ca-model-cache\") pod \"distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl\" (UID: \"96dccd7c-c657-4888-ad95-d65a36d269ca\") " pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:06:32.932206 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.932206    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcvpj\" (UniqueName: \"kubernetes.io/projected/96dccd7c-c657-4888-ad95-d65a36d269ca-kube-api-access-mcvpj\") pod \"distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl\" (UID: \"96dccd7c-c657-4888-ad95-d65a36d269ca\") " pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:06:32.932517 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.932239    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96dccd7c-c657-4888-ad95-d65a36d269ca-home\") pod \"distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl\" (UID: \"96dccd7c-c657-4888-ad95-d65a36d269ca\") " pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:06:32.932517 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.932260    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96dccd7c-c657-4888-ad95-d65a36d269ca-tls-certs\") pod \"distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl\" (UID: \"96dccd7c-c657-4888-ad95-d65a36d269ca\") " pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:06:32.932517 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.932294    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96dccd7c-c657-4888-ad95-d65a36d269ca-tmp-dir\") pod \"distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl\" (UID: \"96dccd7c-c657-4888-ad95-d65a36d269ca\") " pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:06:32.932681 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.932528    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96dccd7c-c657-4888-ad95-d65a36d269ca-dshm\") pod \"distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl\" (UID: \"96dccd7c-c657-4888-ad95-d65a36d269ca\") " pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:06:32.932681 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.932590    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96dccd7c-c657-4888-ad95-d65a36d269ca-model-cache\") pod \"distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl\" (UID: \"96dccd7c-c657-4888-ad95-d65a36d269ca\") " pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:06:32.932789 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.932714    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96dccd7c-c657-4888-ad95-d65a36d269ca-home\") pod \"distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl\" (UID: \"96dccd7c-c657-4888-ad95-d65a36d269ca\") " pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:06:32.932789 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.932735    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96dccd7c-c657-4888-ad95-d65a36d269ca-tmp-dir\") pod \"distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl\" (UID: \"96dccd7c-c657-4888-ad95-d65a36d269ca\") " pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:06:32.934760 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.934736    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96dccd7c-c657-4888-ad95-d65a36d269ca-dshm\") pod \"distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl\" (UID: \"96dccd7c-c657-4888-ad95-d65a36d269ca\") " pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:06:32.934942 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.934924    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96dccd7c-c657-4888-ad95-d65a36d269ca-tls-certs\") pod \"distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl\" (UID: \"96dccd7c-c657-4888-ad95-d65a36d269ca\") " pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:06:32.959451 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:32.959425    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcvpj\" (UniqueName: \"kubernetes.io/projected/96dccd7c-c657-4888-ad95-d65a36d269ca-kube-api-access-mcvpj\") pod \"distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl\" (UID: \"96dccd7c-c657-4888-ad95-d65a36d269ca\") " pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:06:33.055950 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:33.055840    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:06:33.336718 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:33.336687    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"]
Sep 17 22:06:33.338674 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:06:33.338641    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96dccd7c_c657_4888_ad95_d65a36d269ca.slice/crio-f08335b85f614e61ac0a1a4b155fb6d7dd90cb182f86c5e2d46b13942d98234b WatchSource:0}: Error finding container f08335b85f614e61ac0a1a4b155fb6d7dd90cb182f86c5e2d46b13942d98234b: Status 404 returned error can't find the container with id f08335b85f614e61ac0a1a4b155fb6d7dd90cb182f86c5e2d46b13942d98234b
Sep 17 22:06:33.759781 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:33.759743    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl" event={"ID":"96dccd7c-c657-4888-ad95-d65a36d269ca","Type":"ContainerStarted","Data":"da5a36765e27fca7962b5c13fa0566f902dc21160c5d7e327b18e78c6aca370a"}
Sep 17 22:06:33.760207 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:33.759793    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl" event={"ID":"96dccd7c-c657-4888-ad95-d65a36d269ca","Type":"ContainerStarted","Data":"f08335b85f614e61ac0a1a4b155fb6d7dd90cb182f86c5e2d46b13942d98234b"}
Sep 17 22:06:33.782132 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:33.782077    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl" podStartSLOduration=1.782058286 podStartE2EDuration="1.782058286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:06:33.780211955 +0000 UTC m=+1509.598538268" watchObservedRunningTime="2026-09-17 22:06:33.782058286 +0000 UTC m=+1509.600384599"
Sep 17 22:06:43.056437 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:43.056399    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:06:43.056897 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:43.056471    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:06:43.069454 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:43.069427    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:06:43.829206 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:43.829156    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:06:48.452947 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:48.452857    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"]
Sep 17 22:06:48.456118 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:48.456100    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:06:48.458934 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:48.458910    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f996ae\"/\"embedding-w0-f996ae-kserve-self-signed-certs\""
Sep 17 22:06:48.465942 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:48.465921    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"]
Sep 17 22:06:48.488026 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:48.487991    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-dshm\") pod \"embedding-w0-f996ae-kserve-7b7579d988-mp5rm\" (UID: \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\") " pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:06:48.488221 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:48.488056    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-tmp-dir\") pod \"embedding-w0-f996ae-kserve-7b7579d988-mp5rm\" (UID: \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\") " pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:06:48.488221 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:48.488164    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks9kl\" (UniqueName: \"kubernetes.io/projected/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-kube-api-access-ks9kl\") pod \"embedding-w0-f996ae-kserve-7b7579d988-mp5rm\" (UID: \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\") " pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:06:48.488221 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:48.488209    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-home\") pod \"embedding-w0-f996ae-kserve-7b7579d988-mp5rm\" (UID: \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\") " pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:06:48.488404 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:48.488276    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-model-cache\") pod \"embedding-w0-f996ae-kserve-7b7579d988-mp5rm\" (UID: \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\") " pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:06:48.488404 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:48.488301    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-tls-certs\") pod \"embedding-w0-f996ae-kserve-7b7579d988-mp5rm\" (UID: \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\") " pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:06:48.589732 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:48.589687    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-model-cache\") pod \"embedding-w0-f996ae-kserve-7b7579d988-mp5rm\" (UID: \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\") " pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:06:48.589732 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:48.589740    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-tls-certs\") pod \"embedding-w0-f996ae-kserve-7b7579d988-mp5rm\" (UID: \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\") " pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:06:48.590009 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:48.589830    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-dshm\") pod \"embedding-w0-f996ae-kserve-7b7579d988-mp5rm\" (UID: \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\") " pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:06:48.590009 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:48.589860    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-tmp-dir\") pod \"embedding-w0-f996ae-kserve-7b7579d988-mp5rm\" (UID: \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\") " pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:06:48.590009 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:48.589944    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks9kl\" (UniqueName: \"kubernetes.io/projected/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-kube-api-access-ks9kl\") pod \"embedding-w0-f996ae-kserve-7b7579d988-mp5rm\" (UID: \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\") " pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:06:48.590009 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:48.589966    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-home\") pod \"embedding-w0-f996ae-kserve-7b7579d988-mp5rm\" (UID: \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\") " pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:06:48.590258 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:48.590189    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-model-cache\") pod \"embedding-w0-f996ae-kserve-7b7579d988-mp5rm\" (UID: \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\") " pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:06:48.590422 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:48.590400    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-home\") pod \"embedding-w0-f996ae-kserve-7b7579d988-mp5rm\" (UID: \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\") " pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:06:48.590650 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:48.590503    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-tmp-dir\") pod \"embedding-w0-f996ae-kserve-7b7579d988-mp5rm\" (UID: \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\") " pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:06:48.592503 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:48.592480    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-dshm\") pod \"embedding-w0-f996ae-kserve-7b7579d988-mp5rm\" (UID: \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\") " pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:06:48.592942 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:48.592923    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-tls-certs\") pod \"embedding-w0-f996ae-kserve-7b7579d988-mp5rm\" (UID: \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\") " pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:06:48.599523 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:48.599500    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks9kl\" (UniqueName: \"kubernetes.io/projected/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-kube-api-access-ks9kl\") pod \"embedding-w0-f996ae-kserve-7b7579d988-mp5rm\" (UID: \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\") " pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:06:48.766954 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:48.766873    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:06:49.043324 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:49.043300    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"]
Sep 17 22:06:49.045067 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:06:49.045037    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9d45cfa_fa11_49b6_8316_47d2ad76c7ed.slice/crio-5fcaee621d3c9c2011298813db0a92507825438ad3065db7c341f828bb72ead7 WatchSource:0}: Error finding container 5fcaee621d3c9c2011298813db0a92507825438ad3065db7c341f828bb72ead7: Status 404 returned error can't find the container with id 5fcaee621d3c9c2011298813db0a92507825438ad3065db7c341f828bb72ead7
Sep 17 22:06:49.858314 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:49.858280    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm" event={"ID":"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed","Type":"ContainerStarted","Data":"1d947e950461456cb64668c757a50463db2783ad555ff4a81aaffcebcb48558a"}
Sep 17 22:06:49.858680 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:49.858325    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm" event={"ID":"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed","Type":"ContainerStarted","Data":"5fcaee621d3c9c2011298813db0a92507825438ad3065db7c341f828bb72ead7"}
Sep 17 22:06:49.902448 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:49.902401    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm" podStartSLOduration=1.9023845590000001 podStartE2EDuration="1.902384559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:06:49.900542403 +0000 UTC m=+1525.718868715" watchObservedRunningTime="2026-09-17 22:06:49.902384559 +0000 UTC m=+1525.720710871"
Sep 17 22:06:58.768149 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:58.768114    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:06:58.768750 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:58.768191    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:06:58.780743 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:58.780716    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:06:58.918848 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:06:58.918809    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:07:24.985299 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:07:24.985268    2564 scope.go:117] "RemoveContainer" containerID="83673a4ad1bd3a298cac7b606483202cffd86e97aff57ee2961e52c5ddd8e23e"
Sep 17 22:07:24.996031 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:07:24.996005    2564 scope.go:117] "RemoveContainer" containerID="040f609f9a9b11765afb437b1cd45c42b93d9a5b68c550eecba5f54b7d96a93e"
Sep 17 22:08:01.366884 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:01.366842    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv"]
Sep 17 22:08:01.370573 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:01.370557    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv"
Sep 17 22:08:01.373611 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:01.373581    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-194000\""
Sep 17 22:08:01.373726 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:01.373585    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-194000-dockercfg-b7rjd\""
Sep 17 22:08:01.385142 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:01.385111    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv"]
Sep 17 22:08:01.458434 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:01.458386    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4323a69d-88a2-400f-aac2-bd076d9a882e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv\" (UID: \"4323a69d-88a2-400f-aac2-bd076d9a882e\") " pod="openshift-ingress/payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv"
Sep 17 22:08:01.458639 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:01.458595    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72bxv\" (UniqueName: \"kubernetes.io/projected/4323a69d-88a2-400f-aac2-bd076d9a882e-kube-api-access-72bxv\") pod \"payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv\" (UID: \"4323a69d-88a2-400f-aac2-bd076d9a882e\") " pod="openshift-ingress/payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv"
Sep 17 22:08:01.560146 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:01.560112    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4323a69d-88a2-400f-aac2-bd076d9a882e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv\" (UID: \"4323a69d-88a2-400f-aac2-bd076d9a882e\") " pod="openshift-ingress/payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv"
Sep 17 22:08:01.560364 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:01.560255    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72bxv\" (UniqueName: \"kubernetes.io/projected/4323a69d-88a2-400f-aac2-bd076d9a882e-kube-api-access-72bxv\") pod \"payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv\" (UID: \"4323a69d-88a2-400f-aac2-bd076d9a882e\") " pod="openshift-ingress/payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv"
Sep 17 22:08:01.560821 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:01.560799    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4323a69d-88a2-400f-aac2-bd076d9a882e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv\" (UID: \"4323a69d-88a2-400f-aac2-bd076d9a882e\") " pod="openshift-ingress/payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv"
Sep 17 22:08:01.570005 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:01.569983    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72bxv\" (UniqueName: \"kubernetes.io/projected/4323a69d-88a2-400f-aac2-bd076d9a882e-kube-api-access-72bxv\") pod \"payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv\" (UID: \"4323a69d-88a2-400f-aac2-bd076d9a882e\") " pod="openshift-ingress/payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv"
Sep 17 22:08:01.681805 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:01.681767    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv"
Sep 17 22:08:01.961558 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:01.961528    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv"]
Sep 17 22:08:01.963114 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:08:01.963091    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4323a69d_88a2_400f_aac2_bd076d9a882e.slice/crio-9bc96c761f2c4198925656c245463a5870f22101974ff153fa84d1193f25406e WatchSource:0}: Error finding container 9bc96c761f2c4198925656c245463a5870f22101974ff153fa84d1193f25406e: Status 404 returned error can't find the container with id 9bc96c761f2c4198925656c245463a5870f22101974ff153fa84d1193f25406e
Sep 17 22:08:02.283681 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:02.283571    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv" event={"ID":"4323a69d-88a2-400f-aac2-bd076d9a882e","Type":"ContainerStarted","Data":"fcf1ac8ca3613942c79b393ff5b75b883739a250c56c616163f3b16a6a8718c5"}
Sep 17 22:08:02.283681 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:02.283640    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv" event={"ID":"4323a69d-88a2-400f-aac2-bd076d9a882e","Type":"ContainerStarted","Data":"9bc96c761f2c4198925656c245463a5870f22101974ff153fa84d1193f25406e"}
Sep 17 22:08:02.283918 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:02.283696    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv"
Sep 17 22:08:02.306158 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:02.306084    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv" podStartSLOduration=1.306066723 podStartE2EDuration="1.306066723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:08:02.303750022 +0000 UTC m=+1598.122076326" watchObservedRunningTime="2026-09-17 22:08:02.306066723 +0000 UTC m=+1598.124393034"
Sep 17 22:08:13.293351 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:13.293311    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv"
Sep 17 22:08:32.912558 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:32.912516    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg"]
Sep 17 22:08:32.916247 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:32.916226    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg"
Sep 17 22:08:32.919359 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:32.919334    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-7dd893-dockercfg-5ld6l\""
Sep 17 22:08:32.919881 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:32.919864    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-7dd893\""
Sep 17 22:08:32.929431 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:32.929409    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg"]
Sep 17 22:08:32.980661 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:32.980627    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cce2093-b133-407b-ad1f-dff334003185-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg\" (UID: \"6cce2093-b133-407b-ad1f-dff334003185\") " pod="openshift-ingress/payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg"
Sep 17 22:08:32.980834 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:32.980680    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgx2j\" (UniqueName: \"kubernetes.io/projected/6cce2093-b133-407b-ad1f-dff334003185-kube-api-access-zgx2j\") pod \"payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg\" (UID: \"6cce2093-b133-407b-ad1f-dff334003185\") " pod="openshift-ingress/payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg"
Sep 17 22:08:33.082256 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:33.082221    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cce2093-b133-407b-ad1f-dff334003185-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg\" (UID: \"6cce2093-b133-407b-ad1f-dff334003185\") " pod="openshift-ingress/payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg"
Sep 17 22:08:33.082435 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:33.082277    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgx2j\" (UniqueName: \"kubernetes.io/projected/6cce2093-b133-407b-ad1f-dff334003185-kube-api-access-zgx2j\") pod \"payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg\" (UID: \"6cce2093-b133-407b-ad1f-dff334003185\") " pod="openshift-ingress/payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg"
Sep 17 22:08:33.082885 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:33.082867    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cce2093-b133-407b-ad1f-dff334003185-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg\" (UID: \"6cce2093-b133-407b-ad1f-dff334003185\") " pod="openshift-ingress/payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg"
Sep 17 22:08:33.096625 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:33.096599    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgx2j\" (UniqueName: \"kubernetes.io/projected/6cce2093-b133-407b-ad1f-dff334003185-kube-api-access-zgx2j\") pod \"payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg\" (UID: \"6cce2093-b133-407b-ad1f-dff334003185\") " pod="openshift-ingress/payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg"
Sep 17 22:08:33.228596 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:33.228501    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg"
Sep 17 22:08:33.508971 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:33.508944    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg"]
Sep 17 22:08:33.511067 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:08:33.511039    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cce2093_b133_407b_ad1f_dff334003185.slice/crio-8b32d48c934563ba9d6f7c93bf604d6c465c05c221765df27d0be72233edaf22 WatchSource:0}: Error finding container 8b32d48c934563ba9d6f7c93bf604d6c465c05c221765df27d0be72233edaf22: Status 404 returned error can't find the container with id 8b32d48c934563ba9d6f7c93bf604d6c465c05c221765df27d0be72233edaf22
Sep 17 22:08:34.476054 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:34.476017    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg" event={"ID":"6cce2093-b133-407b-ad1f-dff334003185","Type":"ContainerStarted","Data":"1011816765cd88f41473fa4e637c8f23dfa59235b91cf3d2508bedf5d00a4026"}
Sep 17 22:08:34.476492 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:34.476118    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg" event={"ID":"6cce2093-b133-407b-ad1f-dff334003185","Type":"ContainerStarted","Data":"8b32d48c934563ba9d6f7c93bf604d6c465c05c221765df27d0be72233edaf22"}
Sep 17 22:08:34.476492 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:34.476150    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg"
Sep 17 22:08:34.497045 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:34.496985    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg" podStartSLOduration=2.496968269 podStartE2EDuration="2.496968269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:08:34.495304743 +0000 UTC m=+1630.313631053" watchObservedRunningTime="2026-09-17 22:08:34.496968269 +0000 UTC m=+1630.315294581"
Sep 17 22:08:45.483141 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:45.483113    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg"
Sep 17 22:08:45.499583 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:45.499553    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f996ae-lxhm4"]
Sep 17 22:08:45.503309 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:45.503291    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f996ae-lxhm4"
Sep 17 22:08:45.506397 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:45.506377    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmdbp\""
Sep 17 22:08:45.506514 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:45.506378    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 22:08:45.519158 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:45.519133    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f996ae-lxhm4"]
Sep 17 22:08:45.612732 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:45.612701    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b05aa63-e60d-4534-88cf-1730da68abe5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f996ae-lxhm4\" (UID: \"0b05aa63-e60d-4534-88cf-1730da68abe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f996ae-lxhm4"
Sep 17 22:08:45.714281 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:45.714249    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b05aa63-e60d-4534-88cf-1730da68abe5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f996ae-lxhm4\" (UID: \"0b05aa63-e60d-4534-88cf-1730da68abe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f996ae-lxhm4"
Sep 17 22:08:45.714873 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:45.714853    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b05aa63-e60d-4534-88cf-1730da68abe5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f996ae-lxhm4\" (UID: \"0b05aa63-e60d-4534-88cf-1730da68abe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f996ae-lxhm4"
Sep 17 22:08:45.814488 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:45.814398    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f996ae-lxhm4"
Sep 17 22:08:46.092248 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:08:46.092153    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b05aa63_e60d_4534_88cf_1730da68abe5.slice/crio-2781a1092708ccd93ff1caa0c78db5aab3d65781546a9bd584a9fa38977752b6 WatchSource:0}: Error finding container 2781a1092708ccd93ff1caa0c78db5aab3d65781546a9bd584a9fa38977752b6: Status 404 returned error can't find the container with id 2781a1092708ccd93ff1caa0c78db5aab3d65781546a9bd584a9fa38977752b6
Sep 17 22:08:46.092792 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:46.092765    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f996ae-lxhm4"]
Sep 17 22:08:46.546274 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:46.546237    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f996ae-lxhm4" event={"ID":"0b05aa63-e60d-4534-88cf-1730da68abe5","Type":"ContainerStarted","Data":"f9ae3ad5040f59367c7cb4a74bcf75e9886ef617f4b116ac2f5c7417a5c7a944"}
Sep 17 22:08:46.546647 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:46.546300    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f996ae-lxhm4" event={"ID":"0b05aa63-e60d-4534-88cf-1730da68abe5","Type":"ContainerStarted","Data":"2781a1092708ccd93ff1caa0c78db5aab3d65781546a9bd584a9fa38977752b6"}
Sep 17 22:08:46.566162 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:46.566107    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f996ae-lxhm4" podStartSLOduration=1.5660925369999998 podStartE2EDuration="1.566092537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:08:46.564550765 +0000 UTC m=+1642.382877089" watchObservedRunningTime="2026-09-17 22:08:46.566092537 +0000 UTC m=+1642.384419022"
Sep 17 22:08:47.553761 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:47.553723    2564 generic.go:356] "Generic (PLEG): container finished" podID="0b05aa63-e60d-4534-88cf-1730da68abe5" containerID="f9ae3ad5040f59367c7cb4a74bcf75e9886ef617f4b116ac2f5c7417a5c7a944" exitCode=0
Sep 17 22:08:47.554137 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:47.553797    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f996ae-lxhm4" event={"ID":"0b05aa63-e60d-4534-88cf-1730da68abe5","Type":"ContainerDied","Data":"f9ae3ad5040f59367c7cb4a74bcf75e9886ef617f4b116ac2f5c7417a5c7a944"}
Sep 17 22:08:48.709512 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:48.709486    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f996ae-lxhm4"
Sep 17 22:08:48.846400 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:48.846319    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b05aa63-e60d-4534-88cf-1730da68abe5-maas-api-service-ca\") pod \"0b05aa63-e60d-4534-88cf-1730da68abe5\" (UID: \"0b05aa63-e60d-4534-88cf-1730da68abe5\") "
Sep 17 22:08:48.846784 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:48.846751    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b05aa63-e60d-4534-88cf-1730da68abe5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0b05aa63-e60d-4534-88cf-1730da68abe5" (UID: "0b05aa63-e60d-4534-88cf-1730da68abe5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:08:48.947676 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:48.947641    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b05aa63-e60d-4534-88cf-1730da68abe5-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:08:49.566915 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:49.566887    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f996ae-lxhm4"
Sep 17 22:08:49.567117 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:49.566914    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f996ae-lxhm4" event={"ID":"0b05aa63-e60d-4534-88cf-1730da68abe5","Type":"ContainerDied","Data":"2781a1092708ccd93ff1caa0c78db5aab3d65781546a9bd584a9fa38977752b6"}
Sep 17 22:08:49.567117 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:49.566976    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2781a1092708ccd93ff1caa0c78db5aab3d65781546a9bd584a9fa38977752b6"
Sep 17 22:08:50.351995 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:50.351955    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"]
Sep 17 22:08:50.366440 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:50.366399    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"]
Sep 17 22:08:50.392242 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:50.392210    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"]
Sep 17 22:08:51.485746 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:51.485709    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-194000-dqq7g"]
Sep 17 22:08:51.486148 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:51.486134    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b05aa63-e60d-4534-88cf-1730da68abe5" containerName="revoke-keys"
Sep 17 22:08:51.486148 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:51.486149    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b05aa63-e60d-4534-88cf-1730da68abe5" containerName="revoke-keys"
Sep 17 22:08:51.486291 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:51.486283    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b05aa63-e60d-4534-88cf-1730da68abe5" containerName="revoke-keys"
Sep 17 22:08:51.489438 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:51.489420    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-194000-dqq7g"
Sep 17 22:08:51.492364 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:51.492345    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 22:08:51.492483 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:51.492420    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmdbp\""
Sep 17 22:08:51.498162 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:51.498139    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-194000-dqq7g"]
Sep 17 22:08:51.573205 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:51.573135    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0890305-418a-4aab-8b38-51dc3049b855-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-194000-dqq7g\" (UID: \"d0890305-418a-4aab-8b38-51dc3049b855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-194000-dqq7g"
Sep 17 22:08:51.674611 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:51.674577    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0890305-418a-4aab-8b38-51dc3049b855-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-194000-dqq7g\" (UID: \"d0890305-418a-4aab-8b38-51dc3049b855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-194000-dqq7g"
Sep 17 22:08:51.675251 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:51.675231    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0890305-418a-4aab-8b38-51dc3049b855-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-194000-dqq7g\" (UID: \"d0890305-418a-4aab-8b38-51dc3049b855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-194000-dqq7g"
Sep 17 22:08:51.802412 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:51.802317    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-194000-dqq7g"
Sep 17 22:08:52.082823 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:52.082795    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-194000-dqq7g"]
Sep 17 22:08:52.084405 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:08:52.084377    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0890305_418a_4aab_8b38_51dc3049b855.slice/crio-a812cdd12690a9255304a20cf314a0b7fc26e954d4474b1384611d5598f2da4e WatchSource:0}: Error finding container a812cdd12690a9255304a20cf314a0b7fc26e954d4474b1384611d5598f2da4e: Status 404 returned error can't find the container with id a812cdd12690a9255304a20cf314a0b7fc26e954d4474b1384611d5598f2da4e
Sep 17 22:08:52.586193 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:52.586143    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-194000-dqq7g" event={"ID":"d0890305-418a-4aab-8b38-51dc3049b855","Type":"ContainerStarted","Data":"a510bfaac3948dd7106b0065eb2c26d11cac361bc1f4d58bf4ed1e28f8bbfe97"}
Sep 17 22:08:52.586688 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:52.586249    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-194000-dqq7g" event={"ID":"d0890305-418a-4aab-8b38-51dc3049b855","Type":"ContainerStarted","Data":"a812cdd12690a9255304a20cf314a0b7fc26e954d4474b1384611d5598f2da4e"}
Sep 17 22:08:52.604010 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:52.603960    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-194000-dqq7g" podStartSLOduration=1.603945212 podStartE2EDuration="1.603945212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:08:52.601655454 +0000 UTC m=+1648.419981766" watchObservedRunningTime="2026-09-17 22:08:52.603945212 +0000 UTC m=+1648.422271524"
Sep 17 22:08:53.593368 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:53.593333    2564 generic.go:356] "Generic (PLEG): container finished" podID="d0890305-418a-4aab-8b38-51dc3049b855" containerID="a510bfaac3948dd7106b0065eb2c26d11cac361bc1f4d58bf4ed1e28f8bbfe97" exitCode=0
Sep 17 22:08:53.593793 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:53.593422    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-194000-dqq7g" event={"ID":"d0890305-418a-4aab-8b38-51dc3049b855","Type":"ContainerDied","Data":"a510bfaac3948dd7106b0065eb2c26d11cac361bc1f4d58bf4ed1e28f8bbfe97"}
Sep 17 22:08:54.751109 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:54.751083    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-194000-dqq7g"
Sep 17 22:08:54.906214 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:54.906163    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0890305-418a-4aab-8b38-51dc3049b855-maas-api-service-ca\") pod \"d0890305-418a-4aab-8b38-51dc3049b855\" (UID: \"d0890305-418a-4aab-8b38-51dc3049b855\") "
Sep 17 22:08:54.906602 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:54.906578    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0890305-418a-4aab-8b38-51dc3049b855-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d0890305-418a-4aab-8b38-51dc3049b855" (UID: "d0890305-418a-4aab-8b38-51dc3049b855"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:08:55.007361 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:55.007322    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0890305-418a-4aab-8b38-51dc3049b855-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:08:55.608305 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:55.608270    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-194000-dqq7g"
Sep 17 22:08:55.608473 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:55.608302    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-194000-dqq7g" event={"ID":"d0890305-418a-4aab-8b38-51dc3049b855","Type":"ContainerDied","Data":"a812cdd12690a9255304a20cf314a0b7fc26e954d4474b1384611d5598f2da4e"}
Sep 17 22:08:55.608473 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:08:55.608384    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a812cdd12690a9255304a20cf314a0b7fc26e954d4474b1384611d5598f2da4e"
Sep 17 22:09:03.660502 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:03.660469    2564 generic.go:356] "Generic (PLEG): container finished" podID="6b923825-e691-4554-aef8-c604b8fa3749" containerID="03f5a5e0832b6f9e684484d9726470a3da140cc1b40b9dfb043693d60b996305" exitCode=137
Sep 17 22:09:03.660928 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:03.660549    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf" event={"ID":"6b923825-e691-4554-aef8-c604b8fa3749","Type":"ContainerDied","Data":"03f5a5e0832b6f9e684484d9726470a3da140cc1b40b9dfb043693d60b996305"}
Sep 17 22:09:03.660928 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:03.660897    2564 scope.go:117] "RemoveContainer" containerID="03f5a5e0832b6f9e684484d9726470a3da140cc1b40b9dfb043693d60b996305"
Sep 17 22:09:04.669318 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:04.669281    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf" event={"ID":"6b923825-e691-4554-aef8-c604b8fa3749","Type":"ContainerStarted","Data":"0ff0a92007b5b90d8169aafb6a281e55a7c5f9ef653ffe1baff4923dded31237"}
Sep 17 22:09:04.669727 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:04.669480    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf"
Sep 17 22:09:05.372799 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.372759    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl" podUID="9c33695c-bc26-49f7-9d90-4b75b5cdaf81" containerName="main" containerID="cri-o://48288f49dee257ed2ad53884c0d34f1e015145fc77c8d7a98d365c2a1872f75c" gracePeriod=45
Sep 17 22:09:05.387300 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.387259    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm" podUID="b9d45cfa-fa11-49b6-8316-47d2ad76c7ed" containerName="main" containerID="cri-o://1d947e950461456cb64668c757a50463db2783ad555ff4a81aaffcebcb48558a" gracePeriod=45
Sep 17 22:09:05.413049 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.413001    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz" podUID="6971204f-1b69-42e6-8327-ebd78192e939" containerName="main" containerID="cri-o://0cff843bf7e98ea17c72f23b0e6aa9b6269a065b241c8f8ef7f8a59c58d7647b" gracePeriod=45
Sep 17 22:09:05.450792 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.450763    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f996ae-29828040-d6qfj"]
Sep 17 22:09:05.457161 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.457134    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f996ae-29828040-d6qfj"]
Sep 17 22:09:05.554135 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.554063    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t"]
Sep 17 22:09:05.555757 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.554845    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t" podUID="6afd63a8-2cb1-4ace-9483-a70b2298929f" containerName="payload-pre-processing" containerID="cri-o://41296f576876ad992e2c6bf8b95a931b72c9b269d7e4d20533eb1797698a5e8d" gracePeriod=30
Sep 17 22:09:05.679627 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.679551    2564 generic.go:356] "Generic (PLEG): container finished" podID="9c33695c-bc26-49f7-9d90-4b75b5cdaf81" containerID="48288f49dee257ed2ad53884c0d34f1e015145fc77c8d7a98d365c2a1872f75c" exitCode=0
Sep 17 22:09:05.681259 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.679702    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl" event={"ID":"9c33695c-bc26-49f7-9d90-4b75b5cdaf81","Type":"ContainerDied","Data":"48288f49dee257ed2ad53884c0d34f1e015145fc77c8d7a98d365c2a1872f75c"}
Sep 17 22:09:05.682967 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.682921    2564 generic.go:356] "Generic (PLEG): container finished" podID="b9d45cfa-fa11-49b6-8316-47d2ad76c7ed" containerID="1d947e950461456cb64668c757a50463db2783ad555ff4a81aaffcebcb48558a" exitCode=0
Sep 17 22:09:05.683068 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.682968    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm" event={"ID":"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed","Type":"ContainerDied","Data":"1d947e950461456cb64668c757a50463db2783ad555ff4a81aaffcebcb48558a"}
Sep 17 22:09:05.685735 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.685354    2564 generic.go:356] "Generic (PLEG): container finished" podID="6971204f-1b69-42e6-8327-ebd78192e939" containerID="0cff843bf7e98ea17c72f23b0e6aa9b6269a065b241c8f8ef7f8a59c58d7647b" exitCode=0
Sep 17 22:09:05.685735 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.685384    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz" event={"ID":"6971204f-1b69-42e6-8327-ebd78192e939","Type":"ContainerDied","Data":"0cff843bf7e98ea17c72f23b0e6aa9b6269a065b241c8f8ef7f8a59c58d7647b"}
Sep 17 22:09:05.725131 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.725046    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:09:05.753238 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.752941    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 22:09:05.757843 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.757772    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:09:05.846500 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.846472    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t"
Sep 17 22:09:05.915465 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.915371    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-tls-certs\") pod \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\" (UID: \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\") "
Sep 17 22:09:05.915638 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.915453    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hx87h\" (UniqueName: \"kubernetes.io/projected/6afd63a8-2cb1-4ace-9483-a70b2298929f-kube-api-access-hx87h\") pod \"6afd63a8-2cb1-4ace-9483-a70b2298929f\" (UID: \"6afd63a8-2cb1-4ace-9483-a70b2298929f\") "
Sep 17 22:09:05.915638 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.915523    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjxhp\" (UniqueName: \"kubernetes.io/projected/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-kube-api-access-xjxhp\") pod \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\" (UID: \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\") "
Sep 17 22:09:05.915638 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.915554    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-model-cache\") pod \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\" (UID: \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\") "
Sep 17 22:09:05.915638 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.915605    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6971204f-1b69-42e6-8327-ebd78192e939-dshm\") pod \"6971204f-1b69-42e6-8327-ebd78192e939\" (UID: \"6971204f-1b69-42e6-8327-ebd78192e939\") "
Sep 17 22:09:05.915856 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.915644    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-home\") pod \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\" (UID: \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\") "
Sep 17 22:09:05.915856 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.915668    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-tls-certs\") pod \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\" (UID: \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\") "
Sep 17 22:09:05.915856 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.915721    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcn4k\" (UniqueName: \"kubernetes.io/projected/6971204f-1b69-42e6-8327-ebd78192e939-kube-api-access-vcn4k\") pod \"6971204f-1b69-42e6-8327-ebd78192e939\" (UID: \"6971204f-1b69-42e6-8327-ebd78192e939\") "
Sep 17 22:09:05.915856 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.915772    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afd63a8-2cb1-4ace-9483-a70b2298929f-plugins-config-volume\") pod \"6afd63a8-2cb1-4ace-9483-a70b2298929f\" (UID: \"6afd63a8-2cb1-4ace-9483-a70b2298929f\") "
Sep 17 22:09:05.916068 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.915855    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c33695c-bc26-49f7-9d90-4b75b5cdaf81" (UID: "9c33695c-bc26-49f7-9d90-4b75b5cdaf81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:09:05.916125 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.916095    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afd63a8-2cb1-4ace-9483-a70b2298929f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6afd63a8-2cb1-4ace-9483-a70b2298929f" (UID: "6afd63a8-2cb1-4ace-9483-a70b2298929f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:09:05.916233 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.916215    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6971204f-1b69-42e6-8327-ebd78192e939-tmp-dir\") pod \"6971204f-1b69-42e6-8327-ebd78192e939\" (UID: \"6971204f-1b69-42e6-8327-ebd78192e939\") "
Sep 17 22:09:05.916299 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.916289    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6971204f-1b69-42e6-8327-ebd78192e939-home\") pod \"6971204f-1b69-42e6-8327-ebd78192e939\" (UID: \"6971204f-1b69-42e6-8327-ebd78192e939\") "
Sep 17 22:09:05.916352 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.916316    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-dshm\") pod \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\" (UID: \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\") "
Sep 17 22:09:05.916537 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.916411    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-home" (OuterVolumeSpecName: "home") pod "9c33695c-bc26-49f7-9d90-4b75b5cdaf81" (UID: "9c33695c-bc26-49f7-9d90-4b75b5cdaf81"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:09:05.916729 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.916636    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6971204f-1b69-42e6-8327-ebd78192e939-home" (OuterVolumeSpecName: "home") pod "6971204f-1b69-42e6-8327-ebd78192e939" (UID: "6971204f-1b69-42e6-8327-ebd78192e939"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:09:05.916913 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.916877    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6971204f-1b69-42e6-8327-ebd78192e939-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6971204f-1b69-42e6-8327-ebd78192e939" (UID: "6971204f-1b69-42e6-8327-ebd78192e939"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:09:05.916913 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.916902    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-dshm\") pod \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\" (UID: \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\") "
Sep 17 22:09:05.917061 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.916937    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ks9kl\" (UniqueName: \"kubernetes.io/projected/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-kube-api-access-ks9kl\") pod \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\" (UID: \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\") "
Sep 17 22:09:05.917061 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.916967    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-model-cache\") pod \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\" (UID: \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\") "
Sep 17 22:09:05.917061 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.916999    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-tmp-dir\") pod \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\" (UID: \"9c33695c-bc26-49f7-9d90-4b75b5cdaf81\") "
Sep 17 22:09:05.917061 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.917023    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6971204f-1b69-42e6-8327-ebd78192e939-model-cache\") pod \"6971204f-1b69-42e6-8327-ebd78192e939\" (UID: \"6971204f-1b69-42e6-8327-ebd78192e939\") "
Sep 17 22:09:05.917293 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.917075    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6971204f-1b69-42e6-8327-ebd78192e939-tls-certs\") pod \"6971204f-1b69-42e6-8327-ebd78192e939\" (UID: \"6971204f-1b69-42e6-8327-ebd78192e939\") "
Sep 17 22:09:05.917293 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.917110    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-tmp-dir\") pod \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\" (UID: \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\") "
Sep 17 22:09:05.917293 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.917148    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-home\") pod \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\" (UID: \"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed\") "
Sep 17 22:09:05.917659 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.917610    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-home\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:05.917659 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.917636    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afd63a8-2cb1-4ace-9483-a70b2298929f-plugins-config-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:05.917659 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.917651    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6971204f-1b69-42e6-8327-ebd78192e939-tmp-dir\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:05.917820 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.917665    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6971204f-1b69-42e6-8327-ebd78192e939-home\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:05.917820 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.917678    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-model-cache\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:05.917920 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.917858    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-home" (OuterVolumeSpecName: "home") pod "b9d45cfa-fa11-49b6-8316-47d2ad76c7ed" (UID: "b9d45cfa-fa11-49b6-8316-47d2ad76c7ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:09:05.918029 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.917985    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b9d45cfa-fa11-49b6-8316-47d2ad76c7ed" (UID: "b9d45cfa-fa11-49b6-8316-47d2ad76c7ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:09:05.918095 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.918036    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c33695c-bc26-49f7-9d90-4b75b5cdaf81" (UID: "9c33695c-bc26-49f7-9d90-4b75b5cdaf81"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:09:05.918296 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.918270    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6971204f-1b69-42e6-8327-ebd78192e939-model-cache" (OuterVolumeSpecName: "model-cache") pod "6971204f-1b69-42e6-8327-ebd78192e939" (UID: "6971204f-1b69-42e6-8327-ebd78192e939"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:09:05.918381 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.918358    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6afd63a8-2cb1-4ace-9483-a70b2298929f-kube-api-access-hx87h" (OuterVolumeSpecName: "kube-api-access-hx87h") pod "6afd63a8-2cb1-4ace-9483-a70b2298929f" (UID: "6afd63a8-2cb1-4ace-9483-a70b2298929f"). InnerVolumeSpecName "kube-api-access-hx87h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:09:05.918443 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.918408    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "b9d45cfa-fa11-49b6-8316-47d2ad76c7ed" (UID: "b9d45cfa-fa11-49b6-8316-47d2ad76c7ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:09:05.918718 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.918674    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6971204f-1b69-42e6-8327-ebd78192e939-kube-api-access-vcn4k" (OuterVolumeSpecName: "kube-api-access-vcn4k") pod "6971204f-1b69-42e6-8327-ebd78192e939" (UID: "6971204f-1b69-42e6-8327-ebd78192e939"). InnerVolumeSpecName "kube-api-access-vcn4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:09:05.918718 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.918697    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c33695c-bc26-49f7-9d90-4b75b5cdaf81" (UID: "9c33695c-bc26-49f7-9d90-4b75b5cdaf81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:09:05.919296 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.919156    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-kube-api-access-xjxhp" (OuterVolumeSpecName: "kube-api-access-xjxhp") pod "9c33695c-bc26-49f7-9d90-4b75b5cdaf81" (UID: "9c33695c-bc26-49f7-9d90-4b75b5cdaf81"). InnerVolumeSpecName "kube-api-access-xjxhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:09:05.920059 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.919974    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6971204f-1b69-42e6-8327-ebd78192e939-dshm" (OuterVolumeSpecName: "dshm") pod "6971204f-1b69-42e6-8327-ebd78192e939" (UID: "6971204f-1b69-42e6-8327-ebd78192e939"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:09:05.920458 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.920425    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-dshm" (OuterVolumeSpecName: "dshm") pod "b9d45cfa-fa11-49b6-8316-47d2ad76c7ed" (UID: "b9d45cfa-fa11-49b6-8316-47d2ad76c7ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:09:05.920581 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.920530    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-kube-api-access-ks9kl" (OuterVolumeSpecName: "kube-api-access-ks9kl") pod "b9d45cfa-fa11-49b6-8316-47d2ad76c7ed" (UID: "b9d45cfa-fa11-49b6-8316-47d2ad76c7ed"). InnerVolumeSpecName "kube-api-access-ks9kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:09:05.920638 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.920577    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6971204f-1b69-42e6-8327-ebd78192e939-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6971204f-1b69-42e6-8327-ebd78192e939" (UID: "6971204f-1b69-42e6-8327-ebd78192e939"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:09:05.920686 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.920637    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b9d45cfa-fa11-49b6-8316-47d2ad76c7ed" (UID: "b9d45cfa-fa11-49b6-8316-47d2ad76c7ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:09:05.921278 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.921255    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-dshm" (OuterVolumeSpecName: "dshm") pod "9c33695c-bc26-49f7-9d90-4b75b5cdaf81" (UID: "9c33695c-bc26-49f7-9d90-4b75b5cdaf81"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:09:05.923191 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.923149    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f996ae-lxhm4"]
Sep 17 22:09:05.931396 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:05.931376    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f996ae-lxhm4"]
Sep 17 22:09:06.018822 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.018786    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-dshm\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:06.018822 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.018819    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-dshm\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:06.019053 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.018833    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks9kl\" (UniqueName: \"kubernetes.io/projected/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-kube-api-access-ks9kl\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:06.019053 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.018848    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-model-cache\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:06.019053 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.018864    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-tmp-dir\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:06.019053 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.018876    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6971204f-1b69-42e6-8327-ebd78192e939-model-cache\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:06.019053 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.018887    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6971204f-1b69-42e6-8327-ebd78192e939-tls-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:06.019053 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.018901    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-tmp-dir\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:06.019053 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.018912    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-home\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:06.019053 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.018923    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-tls-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:06.019053 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.018944    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx87h\" (UniqueName: \"kubernetes.io/projected/6afd63a8-2cb1-4ace-9483-a70b2298929f-kube-api-access-hx87h\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:06.019053 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.018958    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjxhp\" (UniqueName: \"kubernetes.io/projected/9c33695c-bc26-49f7-9d90-4b75b5cdaf81-kube-api-access-xjxhp\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:06.019053 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.018972    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6971204f-1b69-42e6-8327-ebd78192e939-dshm\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:06.019053 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.018985    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed-tls-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:06.019053 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.018998    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcn4k\" (UniqueName: \"kubernetes.io/projected/6971204f-1b69-42e6-8327-ebd78192e939-kube-api-access-vcn4k\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:06.693130 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.693098    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"
Sep 17 22:09:06.693599 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.693105    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm" event={"ID":"b9d45cfa-fa11-49b6-8316-47d2ad76c7ed","Type":"ContainerDied","Data":"5fcaee621d3c9c2011298813db0a92507825438ad3065db7c341f828bb72ead7"}
Sep 17 22:09:06.693599 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.693228    2564 scope.go:117] "RemoveContainer" containerID="1d947e950461456cb64668c757a50463db2783ad555ff4a81aaffcebcb48558a"
Sep 17 22:09:06.694802 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.694685    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"
Sep 17 22:09:06.694802 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.694705    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz" event={"ID":"6971204f-1b69-42e6-8327-ebd78192e939","Type":"ContainerDied","Data":"ff629cbc5ed27fbf9a41c36831c923ea133a04510e3d3cc6e41cfcc88edde8ff"}
Sep 17 22:09:06.696326 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.696305    2564 generic.go:356] "Generic (PLEG): container finished" podID="6afd63a8-2cb1-4ace-9483-a70b2298929f" containerID="41296f576876ad992e2c6bf8b95a931b72c9b269d7e4d20533eb1797698a5e8d" exitCode=0
Sep 17 22:09:06.696429 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.696408    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t"
Sep 17 22:09:06.696429 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.696417    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t" event={"ID":"6afd63a8-2cb1-4ace-9483-a70b2298929f","Type":"ContainerDied","Data":"41296f576876ad992e2c6bf8b95a931b72c9b269d7e4d20533eb1797698a5e8d"}
Sep 17 22:09:06.696558 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.696458    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t" event={"ID":"6afd63a8-2cb1-4ace-9483-a70b2298929f","Type":"ContainerDied","Data":"34df1260251a8a80e3154f33e48b2280c7b2d5ac6c9e3a09871dea648f90c17e"}
Sep 17 22:09:06.698239 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.698216    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl" event={"ID":"9c33695c-bc26-49f7-9d90-4b75b5cdaf81","Type":"ContainerDied","Data":"aeb5058a72ea99f1508fa84c41da75ad1052b560322be58a85f88d8d99cd7d36"}
Sep 17 22:09:06.698332 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.698225    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"
Sep 17 22:09:06.705304 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.705050    2564 scope.go:117] "RemoveContainer" containerID="0cff843bf7e98ea17c72f23b0e6aa9b6269a065b241c8f8ef7f8a59c58d7647b"
Sep 17 22:09:06.716572 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.716531    2564 scope.go:117] "RemoveContainer" containerID="41296f576876ad992e2c6bf8b95a931b72c9b269d7e4d20533eb1797698a5e8d"
Sep 17 22:09:06.725588 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.725570    2564 scope.go:117] "RemoveContainer" containerID="41296f576876ad992e2c6bf8b95a931b72c9b269d7e4d20533eb1797698a5e8d"
Sep 17 22:09:06.725873 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:09:06.725857    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41296f576876ad992e2c6bf8b95a931b72c9b269d7e4d20533eb1797698a5e8d\": container with ID starting with 41296f576876ad992e2c6bf8b95a931b72c9b269d7e4d20533eb1797698a5e8d not found: ID does not exist" containerID="41296f576876ad992e2c6bf8b95a931b72c9b269d7e4d20533eb1797698a5e8d"
Sep 17 22:09:06.725913 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.725883    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41296f576876ad992e2c6bf8b95a931b72c9b269d7e4d20533eb1797698a5e8d"} err="failed to get container status \"41296f576876ad992e2c6bf8b95a931b72c9b269d7e4d20533eb1797698a5e8d\": rpc error: code = NotFound desc = could not find container \"41296f576876ad992e2c6bf8b95a931b72c9b269d7e4d20533eb1797698a5e8d\": container with ID starting with 41296f576876ad992e2c6bf8b95a931b72c9b269d7e4d20533eb1797698a5e8d not found: ID does not exist"
Sep 17 22:09:06.725913 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.725901    2564 scope.go:117] "RemoveContainer" containerID="48288f49dee257ed2ad53884c0d34f1e015145fc77c8d7a98d365c2a1872f75c"
Sep 17 22:09:06.735042 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.735020    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t"]
Sep 17 22:09:06.744840 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.744813    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f996ae-65ffc957cd-xkl7t"]
Sep 17 22:09:06.753933 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.753906    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b05aa63-e60d-4534-88cf-1730da68abe5" path="/var/lib/kubelet/pods/0b05aa63-e60d-4534-88cf-1730da68abe5/volumes"
Sep 17 22:09:06.754235 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.754222    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ca15b6e-4aa8-4347-9556-f2f77814463d" path="/var/lib/kubelet/pods/2ca15b6e-4aa8-4347-9556-f2f77814463d/volumes"
Sep 17 22:09:06.754512 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.754500    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6afd63a8-2cb1-4ace-9483-a70b2298929f" path="/var/lib/kubelet/pods/6afd63a8-2cb1-4ace-9483-a70b2298929f/volumes"
Sep 17 22:09:06.763006 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.762984    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"]
Sep 17 22:09:06.766285 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.766263    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f996ae/distinct-2-w0-f996ae-kserve-b8fb4864d-qsfbl"]
Sep 17 22:09:06.779316 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.779295    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"]
Sep 17 22:09:06.787334 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.787303    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f996ae/simulator-w0-f996ae-kserve-67677f8797-n52gz"]
Sep 17 22:09:06.799984 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.799959    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"]
Sep 17 22:09:06.812568 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:06.812544    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f996ae/embedding-w0-f996ae-kserve-7b7579d988-mp5rm"]
Sep 17 22:09:08.754282 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:08.754250    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6971204f-1b69-42e6-8327-ebd78192e939" path="/var/lib/kubelet/pods/6971204f-1b69-42e6-8327-ebd78192e939/volumes"
Sep 17 22:09:08.754750 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:08.754664    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c33695c-bc26-49f7-9d90-4b75b5cdaf81" path="/var/lib/kubelet/pods/9c33695c-bc26-49f7-9d90-4b75b5cdaf81/volumes"
Sep 17 22:09:08.755024 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:08.755009    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9d45cfa-fa11-49b6-8316-47d2ad76c7ed" path="/var/lib/kubelet/pods/b9d45cfa-fa11-49b6-8316-47d2ad76c7ed/volumes"
Sep 17 22:09:13.837766 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:13.837731    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"]
Sep 17 22:09:13.838268 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:13.837989    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz" podUID="e9febd7d-9eea-4806-8103-09aa77022f2f" containerName="istio-proxy" containerID="cri-o://86215180a22172d251a875fe933975569d247bcbbfdff88702e2bd52d8a3d6f5" gracePeriod=30
Sep 17 22:09:13.981246 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:13.981209    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv"]
Sep 17 22:09:13.981492 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:13.981446    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv" podUID="4323a69d-88a2-400f-aac2-bd076d9a882e" containerName="payload-processing" containerID="cri-o://fcf1ac8ca3613942c79b393ff5b75b883739a250c56c616163f3b16a6a8718c5" gracePeriod=30
Sep 17 22:09:14.250137 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:14.250107    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv"
Sep 17 22:09:14.284995 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:14.284959    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-194000-dqq7g"]
Sep 17 22:09:14.291775 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:14.291744    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-194000-dqq7g"]
Sep 17 22:09:14.301467 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:14.301442    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72bxv\" (UniqueName: \"kubernetes.io/projected/4323a69d-88a2-400f-aac2-bd076d9a882e-kube-api-access-72bxv\") pod \"4323a69d-88a2-400f-aac2-bd076d9a882e\" (UID: \"4323a69d-88a2-400f-aac2-bd076d9a882e\") "
Sep 17 22:09:14.301615 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:14.301476    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4323a69d-88a2-400f-aac2-bd076d9a882e-plugins-config-volume\") pod \"4323a69d-88a2-400f-aac2-bd076d9a882e\" (UID: \"4323a69d-88a2-400f-aac2-bd076d9a882e\") "
Sep 17 22:09:14.301918 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:14.301892    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4323a69d-88a2-400f-aac2-bd076d9a882e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4323a69d-88a2-400f-aac2-bd076d9a882e" (UID: "4323a69d-88a2-400f-aac2-bd076d9a882e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:09:14.303975 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:14.303950    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4323a69d-88a2-400f-aac2-bd076d9a882e-kube-api-access-72bxv" (OuterVolumeSpecName: "kube-api-access-72bxv") pod "4323a69d-88a2-400f-aac2-bd076d9a882e" (UID: "4323a69d-88a2-400f-aac2-bd076d9a882e"). InnerVolumeSpecName "kube-api-access-72bxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:09:14.402591 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:14.402559    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72bxv\" (UniqueName: \"kubernetes.io/projected/4323a69d-88a2-400f-aac2-bd076d9a882e-kube-api-access-72bxv\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:14.402591 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:14.402586    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4323a69d-88a2-400f-aac2-bd076d9a882e-plugins-config-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:14.759722 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:14.759612    2564 generic.go:356] "Generic (PLEG): container finished" podID="4323a69d-88a2-400f-aac2-bd076d9a882e" containerID="fcf1ac8ca3613942c79b393ff5b75b883739a250c56c616163f3b16a6a8718c5" exitCode=0
Sep 17 22:09:14.759886 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:14.759752    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv"
Sep 17 22:09:14.761911 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:14.761884    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0890305-418a-4aab-8b38-51dc3049b855" path="/var/lib/kubelet/pods/d0890305-418a-4aab-8b38-51dc3049b855/volumes"
Sep 17 22:09:14.762409 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:14.762386    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv" event={"ID":"4323a69d-88a2-400f-aac2-bd076d9a882e","Type":"ContainerDied","Data":"fcf1ac8ca3613942c79b393ff5b75b883739a250c56c616163f3b16a6a8718c5"}
Sep 17 22:09:14.762518 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:14.762435    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv" event={"ID":"4323a69d-88a2-400f-aac2-bd076d9a882e","Type":"ContainerDied","Data":"9bc96c761f2c4198925656c245463a5870f22101974ff153fa84d1193f25406e"}
Sep 17 22:09:14.762518 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:14.762461    2564 scope.go:117] "RemoveContainer" containerID="fcf1ac8ca3613942c79b393ff5b75b883739a250c56c616163f3b16a6a8718c5"
Sep 17 22:09:14.772953 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:14.772932    2564 scope.go:117] "RemoveContainer" containerID="fcf1ac8ca3613942c79b393ff5b75b883739a250c56c616163f3b16a6a8718c5"
Sep 17 22:09:14.773361 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:09:14.773342    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcf1ac8ca3613942c79b393ff5b75b883739a250c56c616163f3b16a6a8718c5\": container with ID starting with fcf1ac8ca3613942c79b393ff5b75b883739a250c56c616163f3b16a6a8718c5 not found: ID does not exist" containerID="fcf1ac8ca3613942c79b393ff5b75b883739a250c56c616163f3b16a6a8718c5"
Sep 17 22:09:14.773437 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:14.773370    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcf1ac8ca3613942c79b393ff5b75b883739a250c56c616163f3b16a6a8718c5"} err="failed to get container status \"fcf1ac8ca3613942c79b393ff5b75b883739a250c56c616163f3b16a6a8718c5\": rpc error: code = NotFound desc = could not find container \"fcf1ac8ca3613942c79b393ff5b75b883739a250c56c616163f3b16a6a8718c5\": container with ID starting with fcf1ac8ca3613942c79b393ff5b75b883739a250c56c616163f3b16a6a8718c5 not found: ID does not exist"
Sep 17 22:09:14.801659 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:14.801618    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv"]
Sep 17 22:09:14.813269 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:14.813233    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-194000-7d7c6b9595-n8rzv"]
Sep 17 22:09:15.687855 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:15.687825    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf"
Sep 17 22:09:16.754668 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:16.754632    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4323a69d-88a2-400f-aac2-bd076d9a882e" path="/var/lib/kubelet/pods/4323a69d-88a2-400f-aac2-bd076d9a882e/volumes"
Sep 17 22:09:19.104538 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.104514    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 22:09:19.152359 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.152328    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9febd7d-9eea-4806-8103-09aa77022f2f-istio-token\") pod \"e9febd7d-9eea-4806-8103-09aa77022f2f\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") "
Sep 17 22:09:19.152550 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.152369    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-credential-socket\") pod \"e9febd7d-9eea-4806-8103-09aa77022f2f\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") "
Sep 17 22:09:19.152550 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.152400    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-istio-envoy\") pod \"e9febd7d-9eea-4806-8103-09aa77022f2f\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") "
Sep 17 22:09:19.152550 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.152428    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9febd7d-9eea-4806-8103-09aa77022f2f-istio-podinfo\") pod \"e9febd7d-9eea-4806-8103-09aa77022f2f\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") "
Sep 17 22:09:19.152550 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.152514    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-istio-data\") pod \"e9febd7d-9eea-4806-8103-09aa77022f2f\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") "
Sep 17 22:09:19.152550 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.152545    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-workload-socket\") pod \"e9febd7d-9eea-4806-8103-09aa77022f2f\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") "
Sep 17 22:09:19.152853 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.152570    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdggw\" (UniqueName: \"kubernetes.io/projected/e9febd7d-9eea-4806-8103-09aa77022f2f-kube-api-access-rdggw\") pod \"e9febd7d-9eea-4806-8103-09aa77022f2f\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") "
Sep 17 22:09:19.152853 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.152597    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-workload-certs\") pod \"e9febd7d-9eea-4806-8103-09aa77022f2f\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") "
Sep 17 22:09:19.152853 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.152607    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e9febd7d-9eea-4806-8103-09aa77022f2f" (UID: "e9febd7d-9eea-4806-8103-09aa77022f2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:09:19.152853 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.152641    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9febd7d-9eea-4806-8103-09aa77022f2f-istiod-ca-cert\") pod \"e9febd7d-9eea-4806-8103-09aa77022f2f\" (UID: \"e9febd7d-9eea-4806-8103-09aa77022f2f\") "
Sep 17 22:09:19.153066 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.152898    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e9febd7d-9eea-4806-8103-09aa77022f2f" (UID: "e9febd7d-9eea-4806-8103-09aa77022f2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:09:19.153066 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.153045    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-workload-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:19.153201 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.153105    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-credential-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:19.153344 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.153289    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e9febd7d-9eea-4806-8103-09aa77022f2f" (UID: "e9febd7d-9eea-4806-8103-09aa77022f2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:09:19.153425 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.153388    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-istio-data" (OuterVolumeSpecName: "istio-data") pod "e9febd7d-9eea-4806-8103-09aa77022f2f" (UID: "e9febd7d-9eea-4806-8103-09aa77022f2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:09:19.153425 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.153409    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9febd7d-9eea-4806-8103-09aa77022f2f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e9febd7d-9eea-4806-8103-09aa77022f2f" (UID: "e9febd7d-9eea-4806-8103-09aa77022f2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:09:19.154810 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.154784    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9febd7d-9eea-4806-8103-09aa77022f2f-istio-token" (OuterVolumeSpecName: "istio-token") pod "e9febd7d-9eea-4806-8103-09aa77022f2f" (UID: "e9febd7d-9eea-4806-8103-09aa77022f2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:09:19.154926 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.154862    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9febd7d-9eea-4806-8103-09aa77022f2f-kube-api-access-rdggw" (OuterVolumeSpecName: "kube-api-access-rdggw") pod "e9febd7d-9eea-4806-8103-09aa77022f2f" (UID: "e9febd7d-9eea-4806-8103-09aa77022f2f"). InnerVolumeSpecName "kube-api-access-rdggw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:09:19.155273 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.155248    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9febd7d-9eea-4806-8103-09aa77022f2f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e9febd7d-9eea-4806-8103-09aa77022f2f" (UID: "e9febd7d-9eea-4806-8103-09aa77022f2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 22:09:19.155448 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.155428    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e9febd7d-9eea-4806-8103-09aa77022f2f" (UID: "e9febd7d-9eea-4806-8103-09aa77022f2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:09:19.254284 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.254246    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9febd7d-9eea-4806-8103-09aa77022f2f-istio-token\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:19.254284 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.254277    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-istio-envoy\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:19.254284 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.254286    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9febd7d-9eea-4806-8103-09aa77022f2f-istio-podinfo\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:19.254284 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.254293    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-istio-data\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:19.254570 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.254301    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdggw\" (UniqueName: \"kubernetes.io/projected/e9febd7d-9eea-4806-8103-09aa77022f2f-kube-api-access-rdggw\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:19.254570 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.254311    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9febd7d-9eea-4806-8103-09aa77022f2f-workload-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:19.254570 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.254319    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9febd7d-9eea-4806-8103-09aa77022f2f-istiod-ca-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:19.792280 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.792244    2564 generic.go:356] "Generic (PLEG): container finished" podID="e9febd7d-9eea-4806-8103-09aa77022f2f" containerID="86215180a22172d251a875fe933975569d247bcbbfdff88702e2bd52d8a3d6f5" exitCode=0
Sep 17 22:09:19.792459 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.792317    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"
Sep 17 22:09:19.792459 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.792336    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz" event={"ID":"e9febd7d-9eea-4806-8103-09aa77022f2f","Type":"ContainerDied","Data":"86215180a22172d251a875fe933975569d247bcbbfdff88702e2bd52d8a3d6f5"}
Sep 17 22:09:19.792459 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.792406    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz" event={"ID":"e9febd7d-9eea-4806-8103-09aa77022f2f","Type":"ContainerDied","Data":"c1e92a1bb98a4d6969f08fa27b9876976681012dcddd7906df7f8d3e251478cf"}
Sep 17 22:09:19.792459 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.792437    2564 scope.go:117] "RemoveContainer" containerID="86215180a22172d251a875fe933975569d247bcbbfdff88702e2bd52d8a3d6f5"
Sep 17 22:09:19.803759 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.803742    2564 scope.go:117] "RemoveContainer" containerID="34035e9125a4900c90e3f2cf227155f8e54ecfd6ec2446a771f3e080e757abf3"
Sep 17 22:09:19.812217 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.812195    2564 scope.go:117] "RemoveContainer" containerID="86215180a22172d251a875fe933975569d247bcbbfdff88702e2bd52d8a3d6f5"
Sep 17 22:09:19.812483 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:09:19.812463    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86215180a22172d251a875fe933975569d247bcbbfdff88702e2bd52d8a3d6f5\": container with ID starting with 86215180a22172d251a875fe933975569d247bcbbfdff88702e2bd52d8a3d6f5 not found: ID does not exist" containerID="86215180a22172d251a875fe933975569d247bcbbfdff88702e2bd52d8a3d6f5"
Sep 17 22:09:19.812569 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.812499    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86215180a22172d251a875fe933975569d247bcbbfdff88702e2bd52d8a3d6f5"} err="failed to get container status \"86215180a22172d251a875fe933975569d247bcbbfdff88702e2bd52d8a3d6f5\": rpc error: code = NotFound desc = could not find container \"86215180a22172d251a875fe933975569d247bcbbfdff88702e2bd52d8a3d6f5\": container with ID starting with 86215180a22172d251a875fe933975569d247bcbbfdff88702e2bd52d8a3d6f5 not found: ID does not exist"
Sep 17 22:09:19.812569 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.812525    2564 scope.go:117] "RemoveContainer" containerID="34035e9125a4900c90e3f2cf227155f8e54ecfd6ec2446a771f3e080e757abf3"
Sep 17 22:09:19.812807 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:09:19.812790    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34035e9125a4900c90e3f2cf227155f8e54ecfd6ec2446a771f3e080e757abf3\": container with ID starting with 34035e9125a4900c90e3f2cf227155f8e54ecfd6ec2446a771f3e080e757abf3 not found: ID does not exist" containerID="34035e9125a4900c90e3f2cf227155f8e54ecfd6ec2446a771f3e080e757abf3"
Sep 17 22:09:19.812850 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.812813    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34035e9125a4900c90e3f2cf227155f8e54ecfd6ec2446a771f3e080e757abf3"} err="failed to get container status \"34035e9125a4900c90e3f2cf227155f8e54ecfd6ec2446a771f3e080e757abf3\": rpc error: code = NotFound desc = could not find container \"34035e9125a4900c90e3f2cf227155f8e54ecfd6ec2446a771f3e080e757abf3\": container with ID starting with 34035e9125a4900c90e3f2cf227155f8e54ecfd6ec2446a771f3e080e757abf3 not found: ID does not exist"
Sep 17 22:09:19.820689 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.820661    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"]
Sep 17 22:09:19.825490 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:19.825466    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f996ae-openshift-default-8bcc8c97b-p9njz"]
Sep 17 22:09:20.757239 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:20.757202    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9febd7d-9eea-4806-8103-09aa77022f2f" path="/var/lib/kubelet/pods/e9febd7d-9eea-4806-8103-09aa77022f2f/volumes"
Sep 17 22:09:24.373091 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.373060    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7dd893-s6nq5"]
Sep 17 22:09:24.373539 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.373524    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6971204f-1b69-42e6-8327-ebd78192e939" containerName="main"
Sep 17 22:09:24.373539 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.373540    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6971204f-1b69-42e6-8327-ebd78192e939" containerName="main"
Sep 17 22:09:24.373619 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.373553    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9febd7d-9eea-4806-8103-09aa77022f2f" containerName="istio-proxy"
Sep 17 22:09:24.373619 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.373559    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9febd7d-9eea-4806-8103-09aa77022f2f" containerName="istio-proxy"
Sep 17 22:09:24.373619 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.373568    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6afd63a8-2cb1-4ace-9483-a70b2298929f" containerName="payload-pre-processing"
Sep 17 22:09:24.373619 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.373574    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6afd63a8-2cb1-4ace-9483-a70b2298929f" containerName="payload-pre-processing"
Sep 17 22:09:24.373619 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.373581    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c33695c-bc26-49f7-9d90-4b75b5cdaf81" containerName="main"
Sep 17 22:09:24.373619 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.373586    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c33695c-bc26-49f7-9d90-4b75b5cdaf81" containerName="main"
Sep 17 22:09:24.373619 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.373601    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0890305-418a-4aab-8b38-51dc3049b855" containerName="revoke-keys"
Sep 17 22:09:24.373619 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.373607    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0890305-418a-4aab-8b38-51dc3049b855" containerName="revoke-keys"
Sep 17 22:09:24.373619 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.373616    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d45cfa-fa11-49b6-8316-47d2ad76c7ed" containerName="main"
Sep 17 22:09:24.373619 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.373621    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d45cfa-fa11-49b6-8316-47d2ad76c7ed" containerName="main"
Sep 17 22:09:24.373932 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.373628    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4323a69d-88a2-400f-aac2-bd076d9a882e" containerName="payload-processing"
Sep 17 22:09:24.373932 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.373633    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4323a69d-88a2-400f-aac2-bd076d9a882e" containerName="payload-processing"
Sep 17 22:09:24.373932 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.373689    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6971204f-1b69-42e6-8327-ebd78192e939" containerName="main"
Sep 17 22:09:24.373932 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.373697    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9febd7d-9eea-4806-8103-09aa77022f2f" containerName="istio-proxy"
Sep 17 22:09:24.373932 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.373704    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9febd7d-9eea-4806-8103-09aa77022f2f" containerName="istio-proxy"
Sep 17 22:09:24.373932 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.373710    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6afd63a8-2cb1-4ace-9483-a70b2298929f" containerName="payload-pre-processing"
Sep 17 22:09:24.373932 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.373716    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9d45cfa-fa11-49b6-8316-47d2ad76c7ed" containerName="main"
Sep 17 22:09:24.373932 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.373720    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0890305-418a-4aab-8b38-51dc3049b855" containerName="revoke-keys"
Sep 17 22:09:24.373932 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.373726    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c33695c-bc26-49f7-9d90-4b75b5cdaf81" containerName="main"
Sep 17 22:09:24.373932 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.373731    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4323a69d-88a2-400f-aac2-bd076d9a882e" containerName="payload-processing"
Sep 17 22:09:24.378136 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.378121    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7dd893-s6nq5"
Sep 17 22:09:24.391001 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.390980    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmdbp\""
Sep 17 22:09:24.391001 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.390991    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 22:09:24.410536 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.410504    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7dd893-s6nq5"]
Sep 17 22:09:24.505298 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.505262    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/926b50de-da13-4ae0-9134-d9e48536981f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7dd893-s6nq5\" (UID: \"926b50de-da13-4ae0-9134-d9e48536981f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7dd893-s6nq5"
Sep 17 22:09:24.606266 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.606232    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/926b50de-da13-4ae0-9134-d9e48536981f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7dd893-s6nq5\" (UID: \"926b50de-da13-4ae0-9134-d9e48536981f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7dd893-s6nq5"
Sep 17 22:09:24.611790 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.611772    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 22:09:24.617685 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.617663    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/926b50de-da13-4ae0-9134-d9e48536981f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7dd893-s6nq5\" (UID: \"926b50de-da13-4ae0-9134-d9e48536981f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7dd893-s6nq5"
Sep 17 22:09:24.691746 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.691705    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmdbp\""
Sep 17 22:09:24.700032 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:24.699999    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7dd893-s6nq5"
Sep 17 22:09:25.001087 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:25.001045    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7dd893-s6nq5"]
Sep 17 22:09:25.002347 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:09:25.002316    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926b50de_da13_4ae0_9134_d9e48536981f.slice/crio-29ae6982aa78cfca80e91096a014fe9433d16d78705b2a278d54ff1077e19f91 WatchSource:0}: Error finding container 29ae6982aa78cfca80e91096a014fe9433d16d78705b2a278d54ff1077e19f91: Status 404 returned error can't find the container with id 29ae6982aa78cfca80e91096a014fe9433d16d78705b2a278d54ff1077e19f91
Sep 17 22:09:25.033464 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:25.033436    2564 scope.go:117] "RemoveContainer" containerID="8da09c52221a0b8db7a630197f2d7b5504dd97e13813cfed78d7c8db35fae41d"
Sep 17 22:09:25.050454 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:25.050431    2564 scope.go:117] "RemoveContainer" containerID="1df7f6675730fc35cb5d5d38cf9437666009cdcb0729d0b002460db09058fba6"
Sep 17 22:09:25.059579 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:25.059557    2564 scope.go:117] "RemoveContainer" containerID="c1ec83e01bd4e2dafd0acba19806dedeb75f9738280d556506fa9c0b715bb7b5"
Sep 17 22:09:25.830716 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:25.830679    2564 generic.go:356] "Generic (PLEG): container finished" podID="926b50de-da13-4ae0-9134-d9e48536981f" containerID="b665c3afa6be94fca578925f83b1ce80c22c098d32f46eb930d0df3e1b765c2e" exitCode=0
Sep 17 22:09:25.831112 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:25.830766    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7dd893-s6nq5" event={"ID":"926b50de-da13-4ae0-9134-d9e48536981f","Type":"ContainerDied","Data":"b665c3afa6be94fca578925f83b1ce80c22c098d32f46eb930d0df3e1b765c2e"}
Sep 17 22:09:25.831112 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:25.830817    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7dd893-s6nq5" event={"ID":"926b50de-da13-4ae0-9134-d9e48536981f","Type":"ContainerStarted","Data":"29ae6982aa78cfca80e91096a014fe9433d16d78705b2a278d54ff1077e19f91"}
Sep 17 22:09:26.982300 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:26.982275    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7dd893-s6nq5"
Sep 17 22:09:27.130288 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:27.130199    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/926b50de-da13-4ae0-9134-d9e48536981f-maas-api-service-ca\") pod \"926b50de-da13-4ae0-9134-d9e48536981f\" (UID: \"926b50de-da13-4ae0-9134-d9e48536981f\") "
Sep 17 22:09:27.130592 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:27.130565    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/926b50de-da13-4ae0-9134-d9e48536981f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "926b50de-da13-4ae0-9134-d9e48536981f" (UID: "926b50de-da13-4ae0-9134-d9e48536981f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:09:27.231417 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:27.231382    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/926b50de-da13-4ae0-9134-d9e48536981f-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:27.843310 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:27.843281    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7dd893-s6nq5"
Sep 17 22:09:27.843508 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:27.843316    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7dd893-s6nq5" event={"ID":"926b50de-da13-4ae0-9134-d9e48536981f","Type":"ContainerDied","Data":"29ae6982aa78cfca80e91096a014fe9433d16d78705b2a278d54ff1077e19f91"}
Sep 17 22:09:27.843508 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:27.843373    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29ae6982aa78cfca80e91096a014fe9433d16d78705b2a278d54ff1077e19f91"
Sep 17 22:09:40.319226 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:40.319191    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg"]
Sep 17 22:09:40.320569 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:40.319452    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg" podUID="6cce2093-b133-407b-ad1f-dff334003185" containerName="payload-processing" containerID="cri-o://1011816765cd88f41473fa4e637c8f23dfa59235b91cf3d2508bedf5d00a4026" gracePeriod=30
Sep 17 22:09:40.574019 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:40.573950    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7dd893-s6nq5"]
Sep 17 22:09:40.578540 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:40.578517    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7dd893-s6nq5"]
Sep 17 22:09:40.583757 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:40.583733    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg"
Sep 17 22:09:40.754440 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:40.754409    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="926b50de-da13-4ae0-9134-d9e48536981f" path="/var/lib/kubelet/pods/926b50de-da13-4ae0-9134-d9e48536981f/volumes"
Sep 17 22:09:40.767668 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:40.767639    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cce2093-b133-407b-ad1f-dff334003185-plugins-config-volume\") pod \"6cce2093-b133-407b-ad1f-dff334003185\" (UID: \"6cce2093-b133-407b-ad1f-dff334003185\") "
Sep 17 22:09:40.767827 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:40.767723    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgx2j\" (UniqueName: \"kubernetes.io/projected/6cce2093-b133-407b-ad1f-dff334003185-kube-api-access-zgx2j\") pod \"6cce2093-b133-407b-ad1f-dff334003185\" (UID: \"6cce2093-b133-407b-ad1f-dff334003185\") "
Sep 17 22:09:40.767994 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:40.767972    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cce2093-b133-407b-ad1f-dff334003185-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6cce2093-b133-407b-ad1f-dff334003185" (UID: "6cce2093-b133-407b-ad1f-dff334003185"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:09:40.769971 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:40.769951    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cce2093-b133-407b-ad1f-dff334003185-kube-api-access-zgx2j" (OuterVolumeSpecName: "kube-api-access-zgx2j") pod "6cce2093-b133-407b-ad1f-dff334003185" (UID: "6cce2093-b133-407b-ad1f-dff334003185"). InnerVolumeSpecName "kube-api-access-zgx2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:09:40.868730 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:40.868641    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgx2j\" (UniqueName: \"kubernetes.io/projected/6cce2093-b133-407b-ad1f-dff334003185-kube-api-access-zgx2j\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:40.868730 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:40.868673    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cce2093-b133-407b-ad1f-dff334003185-plugins-config-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:40.917442 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:40.917404    2564 generic.go:356] "Generic (PLEG): container finished" podID="6cce2093-b133-407b-ad1f-dff334003185" containerID="1011816765cd88f41473fa4e637c8f23dfa59235b91cf3d2508bedf5d00a4026" exitCode=0
Sep 17 22:09:40.917637 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:40.917458    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg"
Sep 17 22:09:40.917637 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:40.917503    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg" event={"ID":"6cce2093-b133-407b-ad1f-dff334003185","Type":"ContainerDied","Data":"1011816765cd88f41473fa4e637c8f23dfa59235b91cf3d2508bedf5d00a4026"}
Sep 17 22:09:40.917637 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:40.917559    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg" event={"ID":"6cce2093-b133-407b-ad1f-dff334003185","Type":"ContainerDied","Data":"8b32d48c934563ba9d6f7c93bf604d6c465c05c221765df27d0be72233edaf22"}
Sep 17 22:09:40.917637 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:40.917584    2564 scope.go:117] "RemoveContainer" containerID="1011816765cd88f41473fa4e637c8f23dfa59235b91cf3d2508bedf5d00a4026"
Sep 17 22:09:40.928160 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:40.928141    2564 scope.go:117] "RemoveContainer" containerID="1011816765cd88f41473fa4e637c8f23dfa59235b91cf3d2508bedf5d00a4026"
Sep 17 22:09:40.928492 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:09:40.928473    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1011816765cd88f41473fa4e637c8f23dfa59235b91cf3d2508bedf5d00a4026\": container with ID starting with 1011816765cd88f41473fa4e637c8f23dfa59235b91cf3d2508bedf5d00a4026 not found: ID does not exist" containerID="1011816765cd88f41473fa4e637c8f23dfa59235b91cf3d2508bedf5d00a4026"
Sep 17 22:09:40.928545 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:40.928501    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1011816765cd88f41473fa4e637c8f23dfa59235b91cf3d2508bedf5d00a4026"} err="failed to get container status \"1011816765cd88f41473fa4e637c8f23dfa59235b91cf3d2508bedf5d00a4026\": rpc error: code = NotFound desc = could not find container \"1011816765cd88f41473fa4e637c8f23dfa59235b91cf3d2508bedf5d00a4026\": container with ID starting with 1011816765cd88f41473fa4e637c8f23dfa59235b91cf3d2508bedf5d00a4026 not found: ID does not exist"
Sep 17 22:09:40.951661 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:40.951630    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg"]
Sep 17 22:09:40.963864 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:40.963834    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7dd893-5fff5d7c7f-nzkbg"]
Sep 17 22:09:42.754522 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:42.754488    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cce2093-b133-407b-ad1f-dff334003185" path="/var/lib/kubelet/pods/6cce2093-b133-407b-ad1f-dff334003185/volumes"
Sep 17 22:09:49.113684 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:49.113601    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a23afc-x75rx"]
Sep 17 22:09:49.114063 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:49.113994    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cce2093-b133-407b-ad1f-dff334003185" containerName="payload-processing"
Sep 17 22:09:49.114063 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:49.114005    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cce2093-b133-407b-ad1f-dff334003185" containerName="payload-processing"
Sep 17 22:09:49.114063 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:49.114014    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9febd7d-9eea-4806-8103-09aa77022f2f" containerName="istio-proxy"
Sep 17 22:09:49.114063 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:49.114020    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9febd7d-9eea-4806-8103-09aa77022f2f" containerName="istio-proxy"
Sep 17 22:09:49.114063 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:49.114040    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="926b50de-da13-4ae0-9134-d9e48536981f" containerName="revoke-keys"
Sep 17 22:09:49.114063 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:49.114045    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="926b50de-da13-4ae0-9134-d9e48536981f" containerName="revoke-keys"
Sep 17 22:09:49.114285 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:49.114112    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="926b50de-da13-4ae0-9134-d9e48536981f" containerName="revoke-keys"
Sep 17 22:09:49.114285 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:49.114121    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cce2093-b133-407b-ad1f-dff334003185" containerName="payload-processing"
Sep 17 22:09:49.122489 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:49.122456    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a23afc-x75rx"
Sep 17 22:09:49.125549 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:49.125526    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 22:09:49.126444 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:49.126424    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmdbp\""
Sep 17 22:09:49.133596 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:49.133567    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a23afc-x75rx"]
Sep 17 22:09:49.145831 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:49.145799    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c53777-a780-4055-929e-c7d30691a2ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a23afc-x75rx\" (UID: \"d3c53777-a780-4055-929e-c7d30691a2ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a23afc-x75rx"
Sep 17 22:09:49.246357 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:49.246315    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c53777-a780-4055-929e-c7d30691a2ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a23afc-x75rx\" (UID: \"d3c53777-a780-4055-929e-c7d30691a2ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a23afc-x75rx"
Sep 17 22:09:49.247018 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:49.246995    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c53777-a780-4055-929e-c7d30691a2ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a23afc-x75rx\" (UID: \"d3c53777-a780-4055-929e-c7d30691a2ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a23afc-x75rx"
Sep 17 22:09:49.433748 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:49.433700    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a23afc-x75rx"
Sep 17 22:09:49.720394 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:49.720366    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a23afc-x75rx"]
Sep 17 22:09:49.720690 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:09:49.720668    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c53777_a780_4055_929e_c7d30691a2ed.slice/crio-7cf3391c03f42259f151fbdf53dd32cac2f71d5a2c7436c4d8d1701dfc7b5a9c WatchSource:0}: Error finding container 7cf3391c03f42259f151fbdf53dd32cac2f71d5a2c7436c4d8d1701dfc7b5a9c: Status 404 returned error can't find the container with id 7cf3391c03f42259f151fbdf53dd32cac2f71d5a2c7436c4d8d1701dfc7b5a9c
Sep 17 22:09:49.975049 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:49.974960    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a23afc-x75rx" event={"ID":"d3c53777-a780-4055-929e-c7d30691a2ed","Type":"ContainerStarted","Data":"7cf3391c03f42259f151fbdf53dd32cac2f71d5a2c7436c4d8d1701dfc7b5a9c"}
Sep 17 22:09:50.981928 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:50.981831    2564 generic.go:356] "Generic (PLEG): container finished" podID="d3c53777-a780-4055-929e-c7d30691a2ed" containerID="478f2443fcaf87194804fd06a28b0adb9501fd02bf8e7aeff1de44dbb3bad459" exitCode=0
Sep 17 22:09:50.981928 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:50.981900    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a23afc-x75rx" event={"ID":"d3c53777-a780-4055-929e-c7d30691a2ed","Type":"ContainerDied","Data":"478f2443fcaf87194804fd06a28b0adb9501fd02bf8e7aeff1de44dbb3bad459"}
Sep 17 22:09:52.126158 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:52.126133    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a23afc-x75rx"
Sep 17 22:09:52.172273 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:52.172231    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c53777-a780-4055-929e-c7d30691a2ed-maas-api-service-ca\") pod \"d3c53777-a780-4055-929e-c7d30691a2ed\" (UID: \"d3c53777-a780-4055-929e-c7d30691a2ed\") "
Sep 17 22:09:52.172595 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:52.172572    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c53777-a780-4055-929e-c7d30691a2ed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d3c53777-a780-4055-929e-c7d30691a2ed" (UID: "d3c53777-a780-4055-929e-c7d30691a2ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:09:52.273123 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:52.273029    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c53777-a780-4055-929e-c7d30691a2ed-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:09:52.995146 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:52.995115    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a23afc-x75rx"
Sep 17 22:09:52.995339 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:52.995106    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a23afc-x75rx" event={"ID":"d3c53777-a780-4055-929e-c7d30691a2ed","Type":"ContainerDied","Data":"7cf3391c03f42259f151fbdf53dd32cac2f71d5a2c7436c4d8d1701dfc7b5a9c"}
Sep 17 22:09:52.995339 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:52.995234    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cf3391c03f42259f151fbdf53dd32cac2f71d5a2c7436c4d8d1701dfc7b5a9c"
Sep 17 22:09:59.925236 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:59.925197    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz"]
Sep 17 22:09:59.925703 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:09:59.925492    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz" podUID="5835d2b2-3580-4255-8ccc-27812592b696" containerName="payload-pre-processing" containerID="cri-o://287e244af6a6fba15e742e50ae3f76858f66a2900b6ada7accca33131e92b93c" gracePeriod=30
Sep 17 22:10:00.124915 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:00.124883    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a23afc-x75rx"]
Sep 17 22:10:00.129727 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:00.129691    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a23afc-x75rx"]
Sep 17 22:10:00.189437 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:00.189372    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz"
Sep 17 22:10:00.248549 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:00.248511    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-756mk\" (UniqueName: \"kubernetes.io/projected/5835d2b2-3580-4255-8ccc-27812592b696-kube-api-access-756mk\") pod \"5835d2b2-3580-4255-8ccc-27812592b696\" (UID: \"5835d2b2-3580-4255-8ccc-27812592b696\") "
Sep 17 22:10:00.248738 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:00.248697    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5835d2b2-3580-4255-8ccc-27812592b696-plugins-config-volume\") pod \"5835d2b2-3580-4255-8ccc-27812592b696\" (UID: \"5835d2b2-3580-4255-8ccc-27812592b696\") "
Sep 17 22:10:00.249219 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:00.249151    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5835d2b2-3580-4255-8ccc-27812592b696-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5835d2b2-3580-4255-8ccc-27812592b696" (UID: "5835d2b2-3580-4255-8ccc-27812592b696"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:10:00.249598 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:00.249578    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5835d2b2-3580-4255-8ccc-27812592b696-plugins-config-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:10:00.251526 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:00.251492    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5835d2b2-3580-4255-8ccc-27812592b696-kube-api-access-756mk" (OuterVolumeSpecName: "kube-api-access-756mk") pod "5835d2b2-3580-4255-8ccc-27812592b696" (UID: "5835d2b2-3580-4255-8ccc-27812592b696"). InnerVolumeSpecName "kube-api-access-756mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:10:00.351093 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:00.351056    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-756mk\" (UniqueName: \"kubernetes.io/projected/5835d2b2-3580-4255-8ccc-27812592b696-kube-api-access-756mk\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:10:00.753333 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:00.753301    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3c53777-a780-4055-929e-c7d30691a2ed" path="/var/lib/kubelet/pods/d3c53777-a780-4055-929e-c7d30691a2ed/volumes"
Sep 17 22:10:01.039621 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:01.039532    2564 generic.go:356] "Generic (PLEG): container finished" podID="5835d2b2-3580-4255-8ccc-27812592b696" containerID="287e244af6a6fba15e742e50ae3f76858f66a2900b6ada7accca33131e92b93c" exitCode=0
Sep 17 22:10:01.039621 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:01.039578    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz"
Sep 17 22:10:01.039621 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:01.039610    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz" event={"ID":"5835d2b2-3580-4255-8ccc-27812592b696","Type":"ContainerDied","Data":"287e244af6a6fba15e742e50ae3f76858f66a2900b6ada7accca33131e92b93c"}
Sep 17 22:10:01.040124 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:01.039661    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz" event={"ID":"5835d2b2-3580-4255-8ccc-27812592b696","Type":"ContainerDied","Data":"d1f51e0e246978078513bf6a8624a37c30292a66450101f33b3bf95cbcd3947b"}
Sep 17 22:10:01.040124 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:01.039684    2564 scope.go:117] "RemoveContainer" containerID="287e244af6a6fba15e742e50ae3f76858f66a2900b6ada7accca33131e92b93c"
Sep 17 22:10:01.048716 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:01.048694    2564 scope.go:117] "RemoveContainer" containerID="287e244af6a6fba15e742e50ae3f76858f66a2900b6ada7accca33131e92b93c"
Sep 17 22:10:01.049002 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:10:01.048987    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"287e244af6a6fba15e742e50ae3f76858f66a2900b6ada7accca33131e92b93c\": container with ID starting with 287e244af6a6fba15e742e50ae3f76858f66a2900b6ada7accca33131e92b93c not found: ID does not exist" containerID="287e244af6a6fba15e742e50ae3f76858f66a2900b6ada7accca33131e92b93c"
Sep 17 22:10:01.049045 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:01.049011    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287e244af6a6fba15e742e50ae3f76858f66a2900b6ada7accca33131e92b93c"} err="failed to get container status \"287e244af6a6fba15e742e50ae3f76858f66a2900b6ada7accca33131e92b93c\": rpc error: code = NotFound desc = could not find container \"287e244af6a6fba15e742e50ae3f76858f66a2900b6ada7accca33131e92b93c\": container with ID starting with 287e244af6a6fba15e742e50ae3f76858f66a2900b6ada7accca33131e92b93c not found: ID does not exist"
Sep 17 22:10:01.081244 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:01.081213    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz"]
Sep 17 22:10:01.089680 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:01.089650    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a23afc-5c9c6b64f4-mz6zz"]
Sep 17 22:10:02.754361 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:02.754326    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5835d2b2-3580-4255-8ccc-27812592b696" path="/var/lib/kubelet/pods/5835d2b2-3580-4255-8ccc-27812592b696/volumes"
Sep 17 22:10:07.857767 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:07.857729    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-12d586-wr464"]
Sep 17 22:10:07.858202 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:07.858160    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c53777-a780-4055-929e-c7d30691a2ed" containerName="revoke-keys"
Sep 17 22:10:07.858202 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:07.858196    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c53777-a780-4055-929e-c7d30691a2ed" containerName="revoke-keys"
Sep 17 22:10:07.858302 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:07.858210    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5835d2b2-3580-4255-8ccc-27812592b696" containerName="payload-pre-processing"
Sep 17 22:10:07.858302 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:07.858216    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5835d2b2-3580-4255-8ccc-27812592b696" containerName="payload-pre-processing"
Sep 17 22:10:07.858302 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:07.858282    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5835d2b2-3580-4255-8ccc-27812592b696" containerName="payload-pre-processing"
Sep 17 22:10:07.858302 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:07.858296    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c53777-a780-4055-929e-c7d30691a2ed" containerName="revoke-keys"
Sep 17 22:10:07.860828 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:07.860812    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-12d586-wr464"
Sep 17 22:10:07.863695 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:07.863674    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmdbp\""
Sep 17 22:10:07.864123 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:07.864110    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 22:10:07.870024 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:07.870000    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-12d586-wr464"]
Sep 17 22:10:07.928838 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:07.928808    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f9209a-f25a-4fa5-a09c-c1c6d56b38d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-12d586-wr464\" (UID: \"c0f9209a-f25a-4fa5-a09c-c1c6d56b38d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-12d586-wr464"
Sep 17 22:10:08.029684 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:08.029644    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f9209a-f25a-4fa5-a09c-c1c6d56b38d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-12d586-wr464\" (UID: \"c0f9209a-f25a-4fa5-a09c-c1c6d56b38d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-12d586-wr464"
Sep 17 22:10:08.030332 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:08.030312    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f9209a-f25a-4fa5-a09c-c1c6d56b38d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-12d586-wr464\" (UID: \"c0f9209a-f25a-4fa5-a09c-c1c6d56b38d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-12d586-wr464"
Sep 17 22:10:08.173469 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:08.173430    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-12d586-wr464"
Sep 17 22:10:08.451311 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:08.451286    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-12d586-wr464"]
Sep 17 22:10:08.453527 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:10:08.453501    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f9209a_f25a_4fa5_a09c_c1c6d56b38d5.slice/crio-967bda3729ce8012d36b3178db8591d84dcdc9f201ac1c57c796ba30545522fa WatchSource:0}: Error finding container 967bda3729ce8012d36b3178db8591d84dcdc9f201ac1c57c796ba30545522fa: Status 404 returned error can't find the container with id 967bda3729ce8012d36b3178db8591d84dcdc9f201ac1c57c796ba30545522fa
Sep 17 22:10:09.084616 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:09.084514    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-12d586-wr464" event={"ID":"c0f9209a-f25a-4fa5-a09c-c1c6d56b38d5","Type":"ContainerStarted","Data":"37f1286042fd98b785cc57f15e31d1023bbd9b42fbdb6cb0fa826af6d84eb0a2"}
Sep 17 22:10:09.084616 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:09.084581    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-12d586-wr464" event={"ID":"c0f9209a-f25a-4fa5-a09c-c1c6d56b38d5","Type":"ContainerStarted","Data":"967bda3729ce8012d36b3178db8591d84dcdc9f201ac1c57c796ba30545522fa"}
Sep 17 22:10:09.102058 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:09.102003    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-12d586-wr464" podStartSLOduration=2.101986395 podStartE2EDuration="2.101986395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:10:09.100209663 +0000 UTC m=+1724.918535976" watchObservedRunningTime="2026-09-17 22:10:09.101986395 +0000 UTC m=+1724.920312702"
Sep 17 22:10:10.091639 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:10.091598    2564 generic.go:356] "Generic (PLEG): container finished" podID="c0f9209a-f25a-4fa5-a09c-c1c6d56b38d5" containerID="37f1286042fd98b785cc57f15e31d1023bbd9b42fbdb6cb0fa826af6d84eb0a2" exitCode=0
Sep 17 22:10:10.092116 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:10.091673    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-12d586-wr464" event={"ID":"c0f9209a-f25a-4fa5-a09c-c1c6d56b38d5","Type":"ContainerDied","Data":"37f1286042fd98b785cc57f15e31d1023bbd9b42fbdb6cb0fa826af6d84eb0a2"}
Sep 17 22:10:11.244089 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:11.244061    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-12d586-wr464"
Sep 17 22:10:11.362502 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:11.362416    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f9209a-f25a-4fa5-a09c-c1c6d56b38d5-maas-api-service-ca\") pod \"c0f9209a-f25a-4fa5-a09c-c1c6d56b38d5\" (UID: \"c0f9209a-f25a-4fa5-a09c-c1c6d56b38d5\") "
Sep 17 22:10:11.362776 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:11.362752    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0f9209a-f25a-4fa5-a09c-c1c6d56b38d5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c0f9209a-f25a-4fa5-a09c-c1c6d56b38d5" (UID: "c0f9209a-f25a-4fa5-a09c-c1c6d56b38d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:10:11.463908 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:11.463878    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f9209a-f25a-4fa5-a09c-c1c6d56b38d5-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:10:12.104936 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:12.104907    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-12d586-wr464"
Sep 17 22:10:12.104936 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:12.104925    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-12d586-wr464" event={"ID":"c0f9209a-f25a-4fa5-a09c-c1c6d56b38d5","Type":"ContainerDied","Data":"967bda3729ce8012d36b3178db8591d84dcdc9f201ac1c57c796ba30545522fa"}
Sep 17 22:10:12.105155 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:12.104981    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="967bda3729ce8012d36b3178db8591d84dcdc9f201ac1c57c796ba30545522fa"
Sep 17 22:10:18.875978 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:18.875941    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg"]
Sep 17 22:10:18.876448 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:18.876229    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg" podUID="f8a94467-affb-4833-a0e7-d916a4f71f06" containerName="payload-processing" containerID="cri-o://50ec18670a436a6b1cff622a182a8ca420cba389c4f352b80f423669415e1ddb" gracePeriod=30
Sep 17 22:10:18.883544 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:18.883514    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8"]
Sep 17 22:10:18.883797 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:18.883751    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8" podUID="d73be6ba-2cd5-436e-a330-10dde63bcaa7" containerName="payload-pre-processing" containerID="cri-o://7136aa5ad6fcf0b14b5aaee0aed6ae45bec95301a021c88f6894412e610cfa14" gracePeriod=30
Sep 17 22:10:19.145890 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:19.145857    2564 generic.go:356] "Generic (PLEG): container finished" podID="d73be6ba-2cd5-436e-a330-10dde63bcaa7" containerID="7136aa5ad6fcf0b14b5aaee0aed6ae45bec95301a021c88f6894412e610cfa14" exitCode=0
Sep 17 22:10:19.146040 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:19.145927    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8" event={"ID":"d73be6ba-2cd5-436e-a330-10dde63bcaa7","Type":"ContainerDied","Data":"7136aa5ad6fcf0b14b5aaee0aed6ae45bec95301a021c88f6894412e610cfa14"}
Sep 17 22:10:19.147496 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:19.147471    2564 generic.go:356] "Generic (PLEG): container finished" podID="f8a94467-affb-4833-a0e7-d916a4f71f06" containerID="50ec18670a436a6b1cff622a182a8ca420cba389c4f352b80f423669415e1ddb" exitCode=0
Sep 17 22:10:19.147593 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:19.147543    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg" event={"ID":"f8a94467-affb-4833-a0e7-d916a4f71f06","Type":"ContainerDied","Data":"50ec18670a436a6b1cff622a182a8ca420cba389c4f352b80f423669415e1ddb"}
Sep 17 22:10:19.174951 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:19.174930    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg"
Sep 17 22:10:19.175281 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:19.175263    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-12d586-wr464"]
Sep 17 22:10:19.179310 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:19.179162    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8"
Sep 17 22:10:19.181535 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:19.181511    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-12d586-wr464"]
Sep 17 22:10:19.340048 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:19.340011    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a94467-affb-4833-a0e7-d916a4f71f06-plugins-config-volume\") pod \"f8a94467-affb-4833-a0e7-d916a4f71f06\" (UID: \"f8a94467-affb-4833-a0e7-d916a4f71f06\") "
Sep 17 22:10:19.340254 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:19.340091    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2x2g\" (UniqueName: \"kubernetes.io/projected/d73be6ba-2cd5-436e-a330-10dde63bcaa7-kube-api-access-l2x2g\") pod \"d73be6ba-2cd5-436e-a330-10dde63bcaa7\" (UID: \"d73be6ba-2cd5-436e-a330-10dde63bcaa7\") "
Sep 17 22:10:19.340254 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:19.340134    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73be6ba-2cd5-436e-a330-10dde63bcaa7-plugins-config-volume\") pod \"d73be6ba-2cd5-436e-a330-10dde63bcaa7\" (UID: \"d73be6ba-2cd5-436e-a330-10dde63bcaa7\") "
Sep 17 22:10:19.340337 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:19.340251    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7crp7\" (UniqueName: \"kubernetes.io/projected/f8a94467-affb-4833-a0e7-d916a4f71f06-kube-api-access-7crp7\") pod \"f8a94467-affb-4833-a0e7-d916a4f71f06\" (UID: \"f8a94467-affb-4833-a0e7-d916a4f71f06\") "
Sep 17 22:10:19.340454 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:19.340427    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a94467-affb-4833-a0e7-d916a4f71f06-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f8a94467-affb-4833-a0e7-d916a4f71f06" (UID: "f8a94467-affb-4833-a0e7-d916a4f71f06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:10:19.340586 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:19.340558    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73be6ba-2cd5-436e-a330-10dde63bcaa7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d73be6ba-2cd5-436e-a330-10dde63bcaa7" (UID: "d73be6ba-2cd5-436e-a330-10dde63bcaa7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:10:19.340586 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:19.340579    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a94467-affb-4833-a0e7-d916a4f71f06-plugins-config-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:10:19.342533 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:19.342509    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73be6ba-2cd5-436e-a330-10dde63bcaa7-kube-api-access-l2x2g" (OuterVolumeSpecName: "kube-api-access-l2x2g") pod "d73be6ba-2cd5-436e-a330-10dde63bcaa7" (UID: "d73be6ba-2cd5-436e-a330-10dde63bcaa7"). InnerVolumeSpecName "kube-api-access-l2x2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:10:19.342598 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:19.342556    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a94467-affb-4833-a0e7-d916a4f71f06-kube-api-access-7crp7" (OuterVolumeSpecName: "kube-api-access-7crp7") pod "f8a94467-affb-4833-a0e7-d916a4f71f06" (UID: "f8a94467-affb-4833-a0e7-d916a4f71f06"). InnerVolumeSpecName "kube-api-access-7crp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:10:19.441735 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:19.441638    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7crp7\" (UniqueName: \"kubernetes.io/projected/f8a94467-affb-4833-a0e7-d916a4f71f06-kube-api-access-7crp7\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:10:19.441735 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:19.441672    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2x2g\" (UniqueName: \"kubernetes.io/projected/d73be6ba-2cd5-436e-a330-10dde63bcaa7-kube-api-access-l2x2g\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:10:19.441735 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:19.441682    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73be6ba-2cd5-436e-a330-10dde63bcaa7-plugins-config-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:10:20.153964 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:20.153867    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8"
Sep 17 22:10:20.153964 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:20.153869    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8" event={"ID":"d73be6ba-2cd5-436e-a330-10dde63bcaa7","Type":"ContainerDied","Data":"c9739b5a779cea87592d72393fd63c96f259322536c4e05b6aeb37f9e2d33e6e"}
Sep 17 22:10:20.154513 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:20.154017    2564 scope.go:117] "RemoveContainer" containerID="7136aa5ad6fcf0b14b5aaee0aed6ae45bec95301a021c88f6894412e610cfa14"
Sep 17 22:10:20.155639 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:20.155520    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg"
Sep 17 22:10:20.155639 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:20.155545    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg" event={"ID":"f8a94467-affb-4833-a0e7-d916a4f71f06","Type":"ContainerDied","Data":"3d2605cca450681eb4542ff99328749264e954a625fccdd5b12cd475c30192f7"}
Sep 17 22:10:20.165298 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:20.165278    2564 scope.go:117] "RemoveContainer" containerID="50ec18670a436a6b1cff622a182a8ca420cba389c4f352b80f423669415e1ddb"
Sep 17 22:10:20.207824 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:20.207788    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg"]
Sep 17 22:10:20.222138 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:20.222102    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-12d586-547cd9789f-smxzg"]
Sep 17 22:10:20.272701 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:20.272667    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8"]
Sep 17 22:10:20.276865 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:20.276835    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-12d586-5446dd85b6-mm2t8"]
Sep 17 22:10:20.755300 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:20.755270    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0f9209a-f25a-4fa5-a09c-c1c6d56b38d5" path="/var/lib/kubelet/pods/c0f9209a-f25a-4fa5-a09c-c1c6d56b38d5/volumes"
Sep 17 22:10:20.755611 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:20.755598    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d73be6ba-2cd5-436e-a330-10dde63bcaa7" path="/var/lib/kubelet/pods/d73be6ba-2cd5-436e-a330-10dde63bcaa7/volumes"
Sep 17 22:10:20.755913 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:20.755902    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8a94467-affb-4833-a0e7-d916a4f71f06" path="/var/lib/kubelet/pods/f8a94467-affb-4833-a0e7-d916a4f71f06/volumes"
Sep 17 22:10:37.638611 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:37.638572    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dbe8a7-2gpzl"]
Sep 17 22:10:37.639001 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:37.638956    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a94467-affb-4833-a0e7-d916a4f71f06" containerName="payload-processing"
Sep 17 22:10:37.639001 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:37.638968    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a94467-affb-4833-a0e7-d916a4f71f06" containerName="payload-processing"
Sep 17 22:10:37.639001 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:37.638978    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0f9209a-f25a-4fa5-a09c-c1c6d56b38d5" containerName="revoke-keys"
Sep 17 22:10:37.639001 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:37.638984    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0f9209a-f25a-4fa5-a09c-c1c6d56b38d5" containerName="revoke-keys"
Sep 17 22:10:37.639132 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:37.639007    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d73be6ba-2cd5-436e-a330-10dde63bcaa7" containerName="payload-pre-processing"
Sep 17 22:10:37.639132 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:37.639012    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d73be6ba-2cd5-436e-a330-10dde63bcaa7" containerName="payload-pre-processing"
Sep 17 22:10:37.639132 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:37.639066    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d73be6ba-2cd5-436e-a330-10dde63bcaa7" containerName="payload-pre-processing"
Sep 17 22:10:37.639132 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:37.639075    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0f9209a-f25a-4fa5-a09c-c1c6d56b38d5" containerName="revoke-keys"
Sep 17 22:10:37.639132 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:37.639082    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a94467-affb-4833-a0e7-d916a4f71f06" containerName="payload-processing"
Sep 17 22:10:37.641967 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:37.641951    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dbe8a7-2gpzl"
Sep 17 22:10:37.645161 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:37.645138    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmdbp\""
Sep 17 22:10:37.645366 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:37.645352    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 22:10:37.656433 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:37.656410    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dbe8a7-2gpzl"]
Sep 17 22:10:37.705040 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:37.705007    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/518da771-8c03-43e0-afad-421f961d07e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-dbe8a7-2gpzl\" (UID: \"518da771-8c03-43e0-afad-421f961d07e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dbe8a7-2gpzl"
Sep 17 22:10:37.806537 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:37.806496    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/518da771-8c03-43e0-afad-421f961d07e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-dbe8a7-2gpzl\" (UID: \"518da771-8c03-43e0-afad-421f961d07e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dbe8a7-2gpzl"
Sep 17 22:10:37.807137 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:37.807116    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/518da771-8c03-43e0-afad-421f961d07e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-dbe8a7-2gpzl\" (UID: \"518da771-8c03-43e0-afad-421f961d07e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dbe8a7-2gpzl"
Sep 17 22:10:37.952374 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:37.952337    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dbe8a7-2gpzl"
Sep 17 22:10:38.231653 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:38.231624    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dbe8a7-2gpzl"]
Sep 17 22:10:38.233629 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:10:38.233588    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod518da771_8c03_43e0_afad_421f961d07e2.slice/crio-43463366c4c6ef9f30fe0e6ae1ffde28c10f06eb4eaed5ca83b5f68809e0dadc WatchSource:0}: Error finding container 43463366c4c6ef9f30fe0e6ae1ffde28c10f06eb4eaed5ca83b5f68809e0dadc: Status 404 returned error can't find the container with id 43463366c4c6ef9f30fe0e6ae1ffde28c10f06eb4eaed5ca83b5f68809e0dadc
Sep 17 22:10:38.258792 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:38.258758    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dbe8a7-2gpzl" event={"ID":"518da771-8c03-43e0-afad-421f961d07e2","Type":"ContainerStarted","Data":"43463366c4c6ef9f30fe0e6ae1ffde28c10f06eb4eaed5ca83b5f68809e0dadc"}
Sep 17 22:10:39.266441 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:39.266355    2564 generic.go:356] "Generic (PLEG): container finished" podID="518da771-8c03-43e0-afad-421f961d07e2" containerID="cb347de8a1699672e3264a604e9e982dea6d586d1b48ef33b7903b49e6ed3a79" exitCode=0
Sep 17 22:10:39.266832 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:39.266455    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dbe8a7-2gpzl" event={"ID":"518da771-8c03-43e0-afad-421f961d07e2","Type":"ContainerDied","Data":"cb347de8a1699672e3264a604e9e982dea6d586d1b48ef33b7903b49e6ed3a79"}
Sep 17 22:10:40.416304 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:40.416279    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dbe8a7-2gpzl"
Sep 17 22:10:40.431024 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:40.430998    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/518da771-8c03-43e0-afad-421f961d07e2-maas-api-service-ca\") pod \"518da771-8c03-43e0-afad-421f961d07e2\" (UID: \"518da771-8c03-43e0-afad-421f961d07e2\") "
Sep 17 22:10:40.431359 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:40.431339    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/518da771-8c03-43e0-afad-421f961d07e2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "518da771-8c03-43e0-afad-421f961d07e2" (UID: "518da771-8c03-43e0-afad-421f961d07e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:10:40.532240 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:40.532142    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/518da771-8c03-43e0-afad-421f961d07e2-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:10:41.277713 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:41.277679    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dbe8a7-2gpzl" event={"ID":"518da771-8c03-43e0-afad-421f961d07e2","Type":"ContainerDied","Data":"43463366c4c6ef9f30fe0e6ae1ffde28c10f06eb4eaed5ca83b5f68809e0dadc"}
Sep 17 22:10:41.277713 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:41.277714    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dbe8a7-2gpzl"
Sep 17 22:10:41.277948 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:41.277734    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43463366c4c6ef9f30fe0e6ae1ffde28c10f06eb4eaed5ca83b5f68809e0dadc"
Sep 17 22:10:42.490918 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:42.490874    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"]
Sep 17 22:10:42.515903 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:42.515872    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"]
Sep 17 22:10:53.654664 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:53.654626    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5"]
Sep 17 22:10:53.655158 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:53.654860    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5" podUID="3e041119-b2c9-4c79-8e6a-13076bd4a91f" containerName="payload-pre-processing" containerID="cri-o://515d15d20a032576dce192490db16194e8e69041de0007159d13246f2c11576f" gracePeriod=30
Sep 17 22:10:53.871398 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:53.871364    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dbe8a7-2gpzl"]
Sep 17 22:10:53.881485 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:53.881459    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dbe8a7-2gpzl"]
Sep 17 22:10:53.911189 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:53.911105    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5"
Sep 17 22:10:53.959638 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:53.959606    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e041119-b2c9-4c79-8e6a-13076bd4a91f-plugins-config-volume\") pod \"3e041119-b2c9-4c79-8e6a-13076bd4a91f\" (UID: \"3e041119-b2c9-4c79-8e6a-13076bd4a91f\") "
Sep 17 22:10:53.959833 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:53.959656    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhjsz\" (UniqueName: \"kubernetes.io/projected/3e041119-b2c9-4c79-8e6a-13076bd4a91f-kube-api-access-nhjsz\") pod \"3e041119-b2c9-4c79-8e6a-13076bd4a91f\" (UID: \"3e041119-b2c9-4c79-8e6a-13076bd4a91f\") "
Sep 17 22:10:53.960049 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:53.960025    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e041119-b2c9-4c79-8e6a-13076bd4a91f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e041119-b2c9-4c79-8e6a-13076bd4a91f" (UID: "3e041119-b2c9-4c79-8e6a-13076bd4a91f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:10:53.961880 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:53.961860    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e041119-b2c9-4c79-8e6a-13076bd4a91f-kube-api-access-nhjsz" (OuterVolumeSpecName: "kube-api-access-nhjsz") pod "3e041119-b2c9-4c79-8e6a-13076bd4a91f" (UID: "3e041119-b2c9-4c79-8e6a-13076bd4a91f"). InnerVolumeSpecName "kube-api-access-nhjsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:10:54.061291 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:54.061255    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e041119-b2c9-4c79-8e6a-13076bd4a91f-plugins-config-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:10:54.061291 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:54.061285    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhjsz\" (UniqueName: \"kubernetes.io/projected/3e041119-b2c9-4c79-8e6a-13076bd4a91f-kube-api-access-nhjsz\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:10:54.352027 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:54.351938    2564 generic.go:356] "Generic (PLEG): container finished" podID="3e041119-b2c9-4c79-8e6a-13076bd4a91f" containerID="515d15d20a032576dce192490db16194e8e69041de0007159d13246f2c11576f" exitCode=0
Sep 17 22:10:54.352027 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:54.351987    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5"
Sep 17 22:10:54.352027 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:54.351992    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5" event={"ID":"3e041119-b2c9-4c79-8e6a-13076bd4a91f","Type":"ContainerDied","Data":"515d15d20a032576dce192490db16194e8e69041de0007159d13246f2c11576f"}
Sep 17 22:10:54.352274 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:54.352046    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5" event={"ID":"3e041119-b2c9-4c79-8e6a-13076bd4a91f","Type":"ContainerDied","Data":"be7fea490e143bd7b6c6a3c70e4980d5d39b31ded12be9254ca440066e04809c"}
Sep 17 22:10:54.352274 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:54.352076    2564 scope.go:117] "RemoveContainer" containerID="515d15d20a032576dce192490db16194e8e69041de0007159d13246f2c11576f"
Sep 17 22:10:54.361486 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:54.361467    2564 scope.go:117] "RemoveContainer" containerID="515d15d20a032576dce192490db16194e8e69041de0007159d13246f2c11576f"
Sep 17 22:10:54.361783 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:10:54.361764    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515d15d20a032576dce192490db16194e8e69041de0007159d13246f2c11576f\": container with ID starting with 515d15d20a032576dce192490db16194e8e69041de0007159d13246f2c11576f not found: ID does not exist" containerID="515d15d20a032576dce192490db16194e8e69041de0007159d13246f2c11576f"
Sep 17 22:10:54.361845 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:54.361797    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515d15d20a032576dce192490db16194e8e69041de0007159d13246f2c11576f"} err="failed to get container status \"515d15d20a032576dce192490db16194e8e69041de0007159d13246f2c11576f\": rpc error: code = NotFound desc = could not find container \"515d15d20a032576dce192490db16194e8e69041de0007159d13246f2c11576f\": container with ID starting with 515d15d20a032576dce192490db16194e8e69041de0007159d13246f2c11576f not found: ID does not exist"
Sep 17 22:10:54.396344 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:54.396305    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5"]
Sep 17 22:10:54.398362 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:54.398334    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-dbe8a7-58ff4c79f8-zfxf5"]
Sep 17 22:10:54.754270 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:54.754228    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e041119-b2c9-4c79-8e6a-13076bd4a91f" path="/var/lib/kubelet/pods/3e041119-b2c9-4c79-8e6a-13076bd4a91f/volumes"
Sep 17 22:10:54.754769 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:54.754726    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="518da771-8c03-43e0-afad-421f961d07e2" path="/var/lib/kubelet/pods/518da771-8c03-43e0-afad-421f961d07e2/volumes"
Sep 17 22:10:57.511370 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.511319    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl" podUID="96dccd7c-c657-4888-ad95-d65a36d269ca" containerName="main" containerID="cri-o://da5a36765e27fca7962b5c13fa0566f902dc21160c5d7e327b18e78c6aca370a" gracePeriod=45
Sep 17 22:10:57.536119 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.536083    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4" podUID="0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46" containerName="main" containerID="cri-o://a850f05e42f48018c9864c05a05ce72e36320bb3135cb642a8dbbf58f3ca2dc0" gracePeriod=45
Sep 17 22:10:57.820862 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.820837    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:10:57.825269 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.825240    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 22:10:57.899281 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.899254    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96dccd7c-c657-4888-ad95-d65a36d269ca-tls-certs\") pod \"96dccd7c-c657-4888-ad95-d65a36d269ca\" (UID: \"96dccd7c-c657-4888-ad95-d65a36d269ca\") "
Sep 17 22:10:57.899437 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.899318    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96dccd7c-c657-4888-ad95-d65a36d269ca-tmp-dir\") pod \"96dccd7c-c657-4888-ad95-d65a36d269ca\" (UID: \"96dccd7c-c657-4888-ad95-d65a36d269ca\") "
Sep 17 22:10:57.899437 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.899349    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-tls-certs\") pod \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\" (UID: \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\") "
Sep 17 22:10:57.899437 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.899396    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-dshm\") pod \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\" (UID: \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\") "
Sep 17 22:10:57.899616 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.899452    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96dccd7c-c657-4888-ad95-d65a36d269ca-home\") pod \"96dccd7c-c657-4888-ad95-d65a36d269ca\" (UID: \"96dccd7c-c657-4888-ad95-d65a36d269ca\") "
Sep 17 22:10:57.899616 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.899485    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-tmp-dir\") pod \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\" (UID: \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\") "
Sep 17 22:10:57.899616 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.899520    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcvpj\" (UniqueName: \"kubernetes.io/projected/96dccd7c-c657-4888-ad95-d65a36d269ca-kube-api-access-mcvpj\") pod \"96dccd7c-c657-4888-ad95-d65a36d269ca\" (UID: \"96dccd7c-c657-4888-ad95-d65a36d269ca\") "
Sep 17 22:10:57.899616 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.899546    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96dccd7c-c657-4888-ad95-d65a36d269ca-model-cache\") pod \"96dccd7c-c657-4888-ad95-d65a36d269ca\" (UID: \"96dccd7c-c657-4888-ad95-d65a36d269ca\") "
Sep 17 22:10:57.899616 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.899569    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-home\") pod \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\" (UID: \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\") "
Sep 17 22:10:57.899616 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.899595    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96dccd7c-c657-4888-ad95-d65a36d269ca-dshm\") pod \"96dccd7c-c657-4888-ad95-d65a36d269ca\" (UID: \"96dccd7c-c657-4888-ad95-d65a36d269ca\") "
Sep 17 22:10:57.899908 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.899632    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96dccd7c-c657-4888-ad95-d65a36d269ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "96dccd7c-c657-4888-ad95-d65a36d269ca" (UID: "96dccd7c-c657-4888-ad95-d65a36d269ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:10:57.899908 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.899644    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ddw8\" (UniqueName: \"kubernetes.io/projected/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-kube-api-access-8ddw8\") pod \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\" (UID: \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\") "
Sep 17 22:10:57.899908 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.899728    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-model-cache\") pod \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\" (UID: \"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46\") "
Sep 17 22:10:57.899908 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.899891    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46" (UID: "0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:10:57.900132 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.900112    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46" (UID: "0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:10:57.900291 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.900269    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-tmp-dir\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:10:57.900379 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.900294    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-model-cache\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:10:57.900379 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.900309    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96dccd7c-c657-4888-ad95-d65a36d269ca-tmp-dir\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:10:57.901081 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.900828    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96dccd7c-c657-4888-ad95-d65a36d269ca-home" (OuterVolumeSpecName: "home") pod "96dccd7c-c657-4888-ad95-d65a36d269ca" (UID: "96dccd7c-c657-4888-ad95-d65a36d269ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:10:57.901742 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.901325    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96dccd7c-c657-4888-ad95-d65a36d269ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "96dccd7c-c657-4888-ad95-d65a36d269ca" (UID: "96dccd7c-c657-4888-ad95-d65a36d269ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:10:57.901742 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.901521    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-home" (OuterVolumeSpecName: "home") pod "0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46" (UID: "0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:10:57.902702 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.902675    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-kube-api-access-8ddw8" (OuterVolumeSpecName: "kube-api-access-8ddw8") pod "0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46" (UID: "0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46"). InnerVolumeSpecName "kube-api-access-8ddw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:10:57.902824 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.902775    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96dccd7c-c657-4888-ad95-d65a36d269ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "96dccd7c-c657-4888-ad95-d65a36d269ca" (UID: "96dccd7c-c657-4888-ad95-d65a36d269ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:10:57.903113 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.903090    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96dccd7c-c657-4888-ad95-d65a36d269ca-kube-api-access-mcvpj" (OuterVolumeSpecName: "kube-api-access-mcvpj") pod "96dccd7c-c657-4888-ad95-d65a36d269ca" (UID: "96dccd7c-c657-4888-ad95-d65a36d269ca"). InnerVolumeSpecName "kube-api-access-mcvpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:10:57.903647 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.903626    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46" (UID: "0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:10:57.904379 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.904362    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96dccd7c-c657-4888-ad95-d65a36d269ca-dshm" (OuterVolumeSpecName: "dshm") pod "96dccd7c-c657-4888-ad95-d65a36d269ca" (UID: "96dccd7c-c657-4888-ad95-d65a36d269ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:10:57.904785 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:57.904772    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-dshm" (OuterVolumeSpecName: "dshm") pod "0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46" (UID: "0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:10:58.001364 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.001332    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-dshm\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:10:58.001364 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.001356    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96dccd7c-c657-4888-ad95-d65a36d269ca-home\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:10:58.001364 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.001366    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcvpj\" (UniqueName: \"kubernetes.io/projected/96dccd7c-c657-4888-ad95-d65a36d269ca-kube-api-access-mcvpj\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:10:58.001608 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.001375    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96dccd7c-c657-4888-ad95-d65a36d269ca-model-cache\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:10:58.001608 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.001385    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-home\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:10:58.001608 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.001392    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96dccd7c-c657-4888-ad95-d65a36d269ca-dshm\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:10:58.001608 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.001399    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ddw8\" (UniqueName: \"kubernetes.io/projected/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-kube-api-access-8ddw8\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:10:58.001608 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.001407    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96dccd7c-c657-4888-ad95-d65a36d269ca-tls-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:10:58.001608 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.001414    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46-tls-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:10:58.377374 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.377344    2564 generic.go:356] "Generic (PLEG): container finished" podID="0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46" containerID="a850f05e42f48018c9864c05a05ce72e36320bb3135cb642a8dbbf58f3ca2dc0" exitCode=0
Sep 17 22:10:58.377581 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.377387    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"
Sep 17 22:10:58.377581 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.377430    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4" event={"ID":"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46","Type":"ContainerDied","Data":"a850f05e42f48018c9864c05a05ce72e36320bb3135cb642a8dbbf58f3ca2dc0"}
Sep 17 22:10:58.377581 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.377483    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4" event={"ID":"0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46","Type":"ContainerDied","Data":"ac0fba64aab8b66f9320e36e6f91d1409a5eef10892306bb60bffd2084305575"}
Sep 17 22:10:58.377581 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.377508    2564 scope.go:117] "RemoveContainer" containerID="a850f05e42f48018c9864c05a05ce72e36320bb3135cb642a8dbbf58f3ca2dc0"
Sep 17 22:10:58.378819 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.378800    2564 generic.go:356] "Generic (PLEG): container finished" podID="96dccd7c-c657-4888-ad95-d65a36d269ca" containerID="da5a36765e27fca7962b5c13fa0566f902dc21160c5d7e327b18e78c6aca370a" exitCode=0
Sep 17 22:10:58.378920 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.378839    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl" event={"ID":"96dccd7c-c657-4888-ad95-d65a36d269ca","Type":"ContainerDied","Data":"da5a36765e27fca7962b5c13fa0566f902dc21160c5d7e327b18e78c6aca370a"}
Sep 17 22:10:58.378920 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.378849    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"
Sep 17 22:10:58.378920 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.378868    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl" event={"ID":"96dccd7c-c657-4888-ad95-d65a36d269ca","Type":"ContainerDied","Data":"f08335b85f614e61ac0a1a4b155fb6d7dd90cb182f86c5e2d46b13942d98234b"}
Sep 17 22:10:58.387015 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.386997    2564 scope.go:117] "RemoveContainer" containerID="a850f05e42f48018c9864c05a05ce72e36320bb3135cb642a8dbbf58f3ca2dc0"
Sep 17 22:10:58.387493 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:10:58.387466    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a850f05e42f48018c9864c05a05ce72e36320bb3135cb642a8dbbf58f3ca2dc0\": container with ID starting with a850f05e42f48018c9864c05a05ce72e36320bb3135cb642a8dbbf58f3ca2dc0 not found: ID does not exist" containerID="a850f05e42f48018c9864c05a05ce72e36320bb3135cb642a8dbbf58f3ca2dc0"
Sep 17 22:10:58.387621 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.387504    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a850f05e42f48018c9864c05a05ce72e36320bb3135cb642a8dbbf58f3ca2dc0"} err="failed to get container status \"a850f05e42f48018c9864c05a05ce72e36320bb3135cb642a8dbbf58f3ca2dc0\": rpc error: code = NotFound desc = could not find container \"a850f05e42f48018c9864c05a05ce72e36320bb3135cb642a8dbbf58f3ca2dc0\": container with ID starting with a850f05e42f48018c9864c05a05ce72e36320bb3135cb642a8dbbf58f3ca2dc0 not found: ID does not exist"
Sep 17 22:10:58.387621 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.387528    2564 scope.go:117] "RemoveContainer" containerID="da5a36765e27fca7962b5c13fa0566f902dc21160c5d7e327b18e78c6aca370a"
Sep 17 22:10:58.396790 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.396767    2564 scope.go:117] "RemoveContainer" containerID="da5a36765e27fca7962b5c13fa0566f902dc21160c5d7e327b18e78c6aca370a"
Sep 17 22:10:58.397079 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:10:58.397060    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da5a36765e27fca7962b5c13fa0566f902dc21160c5d7e327b18e78c6aca370a\": container with ID starting with da5a36765e27fca7962b5c13fa0566f902dc21160c5d7e327b18e78c6aca370a not found: ID does not exist" containerID="da5a36765e27fca7962b5c13fa0566f902dc21160c5d7e327b18e78c6aca370a"
Sep 17 22:10:58.397148 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.397086    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5a36765e27fca7962b5c13fa0566f902dc21160c5d7e327b18e78c6aca370a"} err="failed to get container status \"da5a36765e27fca7962b5c13fa0566f902dc21160c5d7e327b18e78c6aca370a\": rpc error: code = NotFound desc = could not find container \"da5a36765e27fca7962b5c13fa0566f902dc21160c5d7e327b18e78c6aca370a\": container with ID starting with da5a36765e27fca7962b5c13fa0566f902dc21160c5d7e327b18e78c6aca370a not found: ID does not exist"
Sep 17 22:10:58.411944 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.411917    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"]
Sep 17 22:10:58.420111 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.420088    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-dbe8a7/distinct-2-w4-dbe8a7-kserve-64c5659cbc-b9jtl"]
Sep 17 22:10:58.434613 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.434590    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"]
Sep 17 22:10:58.442783 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.442762    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-dbe8a7/simulator-w4-dbe8a7-kserve-599b998f44-mzvq4"]
Sep 17 22:10:58.755003 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.754969    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46" path="/var/lib/kubelet/pods/0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46/volumes"
Sep 17 22:10:58.755433 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:10:58.755382    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96dccd7c-c657-4888-ad95-d65a36d269ca" path="/var/lib/kubelet/pods/96dccd7c-c657-4888-ad95-d65a36d269ca/volumes"
Sep 17 22:11:24.789314 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:24.789283    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdcnw_6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2/ovn-acl-logging/0.log"
Sep 17 22:11:24.792328 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:24.792302    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdcnw_6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2/ovn-acl-logging/0.log"
Sep 17 22:11:25.137300 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:25.137217    2564 scope.go:117] "RemoveContainer" containerID="cb141663b0fd36bf7076ab7b840f1cb7f169c182034b28738bf32c00a9bd9ac4"
Sep 17 22:11:38.009769 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:38.009735    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7823-bed488"]
Sep 17 22:11:38.010161 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:38.010107    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e041119-b2c9-4c79-8e6a-13076bd4a91f" containerName="payload-pre-processing"
Sep 17 22:11:38.010161 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:38.010119    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e041119-b2c9-4c79-8e6a-13076bd4a91f" containerName="payload-pre-processing"
Sep 17 22:11:38.010161 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:38.010126    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="518da771-8c03-43e0-afad-421f961d07e2" containerName="revoke-keys"
Sep 17 22:11:38.010161 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:38.010133    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="518da771-8c03-43e0-afad-421f961d07e2" containerName="revoke-keys"
Sep 17 22:11:38.010161 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:38.010149    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46" containerName="main"
Sep 17 22:11:38.010161 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:38.010154    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46" containerName="main"
Sep 17 22:11:38.010161 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:38.010162    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96dccd7c-c657-4888-ad95-d65a36d269ca" containerName="main"
Sep 17 22:11:38.010161 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:38.010182    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="96dccd7c-c657-4888-ad95-d65a36d269ca" containerName="main"
Sep 17 22:11:38.010447 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:38.010232    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="96dccd7c-c657-4888-ad95-d65a36d269ca" containerName="main"
Sep 17 22:11:38.010447 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:38.010242    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d02dd9f-3fb4-4f4b-88e2-c2ae9240fc46" containerName="main"
Sep 17 22:11:38.010447 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:38.010248    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="518da771-8c03-43e0-afad-421f961d07e2" containerName="revoke-keys"
Sep 17 22:11:38.010447 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:38.010254    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e041119-b2c9-4c79-8e6a-13076bd4a91f" containerName="payload-pre-processing"
Sep 17 22:11:38.013165 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:38.013147    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7823-bed488"
Sep 17 22:11:38.015824 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:38.015803    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lbp7g\""
Sep 17 22:11:38.020112 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:38.020083    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7823-bed488"]
Sep 17 22:11:38.069507 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:38.069474    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkqw5\" (UniqueName: \"kubernetes.io/projected/1784b293-6ea2-42f9-a391-0c282c24246a-kube-api-access-wkqw5\") pod \"test-curl-7823-bed488\" (UID: \"1784b293-6ea2-42f9-a391-0c282c24246a\") " pod="opendatahub/test-curl-7823-bed488"
Sep 17 22:11:38.170431 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:38.170389    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkqw5\" (UniqueName: \"kubernetes.io/projected/1784b293-6ea2-42f9-a391-0c282c24246a-kube-api-access-wkqw5\") pod \"test-curl-7823-bed488\" (UID: \"1784b293-6ea2-42f9-a391-0c282c24246a\") " pod="opendatahub/test-curl-7823-bed488"
Sep 17 22:11:38.178787 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:38.178762    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkqw5\" (UniqueName: \"kubernetes.io/projected/1784b293-6ea2-42f9-a391-0c282c24246a-kube-api-access-wkqw5\") pod \"test-curl-7823-bed488\" (UID: \"1784b293-6ea2-42f9-a391-0c282c24246a\") " pod="opendatahub/test-curl-7823-bed488"
Sep 17 22:11:38.323835 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:38.323748    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7823-bed488"
Sep 17 22:11:38.597326 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:38.597300    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7823-bed488"]
Sep 17 22:11:38.599911 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:11:38.599885    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1784b293_6ea2_42f9_a391_0c282c24246a.slice/crio-50ac21e8f6ae7987aa94bb2ac0f39e9996b56bc92c523ef78d2af1295dce277b WatchSource:0}: Error finding container 50ac21e8f6ae7987aa94bb2ac0f39e9996b56bc92c523ef78d2af1295dce277b: Status 404 returned error can't find the container with id 50ac21e8f6ae7987aa94bb2ac0f39e9996b56bc92c523ef78d2af1295dce277b
Sep 17 22:11:38.611383 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:38.611359    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7823-bed488" event={"ID":"1784b293-6ea2-42f9-a391-0c282c24246a","Type":"ContainerStarted","Data":"50ac21e8f6ae7987aa94bb2ac0f39e9996b56bc92c523ef78d2af1295dce277b"}
Sep 17 22:11:39.617921 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:39.617885    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7823-bed488" event={"ID":"1784b293-6ea2-42f9-a391-0c282c24246a","Type":"ContainerStarted","Data":"98b9613dc30f41212bbbbbfca6d7eed71c0fdd5e0240233cf5b84734db919cfc"}
Sep 17 22:11:39.633806 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:39.633749    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7823-bed488" podStartSLOduration=2.633733816 podStartE2EDuration="2.633733816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:11:39.633567396 +0000 UTC m=+1815.451893710" watchObservedRunningTime="2026-09-17 22:11:39.633733816 +0000 UTC m=+1815.452060129"
Sep 17 22:11:40.260607 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:40.260559    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7823-bed488"]
Sep 17 22:11:40.266906 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:40.266880    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7823-bed488"]
Sep 17 22:11:40.623276 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:11:40.623247    2564 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7823-bed488" container="test-curl-7823-bed488"
Sep 17 22:11:41.627882 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:41.627818    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7823-bed488" podUID="1784b293-6ea2-42f9-a391-0c282c24246a" containerName="test-curl-7823-bed488" containerID="cri-o://98b9613dc30f41212bbbbbfca6d7eed71c0fdd5e0240233cf5b84734db919cfc" gracePeriod=2
Sep 17 22:11:43.808608 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:43.808584    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7823-bed488"
Sep 17 22:11:43.922260 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:43.922224    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkqw5\" (UniqueName: \"kubernetes.io/projected/1784b293-6ea2-42f9-a391-0c282c24246a-kube-api-access-wkqw5\") pod \"1784b293-6ea2-42f9-a391-0c282c24246a\" (UID: \"1784b293-6ea2-42f9-a391-0c282c24246a\") "
Sep 17 22:11:43.924543 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:43.924518    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1784b293-6ea2-42f9-a391-0c282c24246a-kube-api-access-wkqw5" (OuterVolumeSpecName: "kube-api-access-wkqw5") pod "1784b293-6ea2-42f9-a391-0c282c24246a" (UID: "1784b293-6ea2-42f9-a391-0c282c24246a"). InnerVolumeSpecName "kube-api-access-wkqw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:11:44.023431 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:44.023392    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkqw5\" (UniqueName: \"kubernetes.io/projected/1784b293-6ea2-42f9-a391-0c282c24246a-kube-api-access-wkqw5\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:11:44.643920 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:44.643886    2564 generic.go:356] "Generic (PLEG): container finished" podID="1784b293-6ea2-42f9-a391-0c282c24246a" containerID="98b9613dc30f41212bbbbbfca6d7eed71c0fdd5e0240233cf5b84734db919cfc" exitCode=137
Sep 17 22:11:44.644147 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:44.643942    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7823-bed488"
Sep 17 22:11:44.644147 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:44.643974    2564 scope.go:117] "RemoveContainer" containerID="98b9613dc30f41212bbbbbfca6d7eed71c0fdd5e0240233cf5b84734db919cfc"
Sep 17 22:11:44.653711 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:44.653693    2564 scope.go:117] "RemoveContainer" containerID="98b9613dc30f41212bbbbbfca6d7eed71c0fdd5e0240233cf5b84734db919cfc"
Sep 17 22:11:44.653965 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:11:44.653947    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98b9613dc30f41212bbbbbfca6d7eed71c0fdd5e0240233cf5b84734db919cfc\": container with ID starting with 98b9613dc30f41212bbbbbfca6d7eed71c0fdd5e0240233cf5b84734db919cfc not found: ID does not exist" containerID="98b9613dc30f41212bbbbbfca6d7eed71c0fdd5e0240233cf5b84734db919cfc"
Sep 17 22:11:44.654018 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:44.653975    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b9613dc30f41212bbbbbfca6d7eed71c0fdd5e0240233cf5b84734db919cfc"} err="failed to get container status \"98b9613dc30f41212bbbbbfca6d7eed71c0fdd5e0240233cf5b84734db919cfc\": rpc error: code = NotFound desc = could not find container \"98b9613dc30f41212bbbbbfca6d7eed71c0fdd5e0240233cf5b84734db919cfc\": container with ID starting with 98b9613dc30f41212bbbbbfca6d7eed71c0fdd5e0240233cf5b84734db919cfc not found: ID does not exist"
Sep 17 22:11:44.657003 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:11:44.656981    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7823-bed488\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="1784b293-6ea2-42f9-a391-0c282c24246a" pod="opendatahub/test-curl-7823-bed488"
Sep 17 22:11:44.757395 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:11:44.757359    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7823-bed488\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="1784b293-6ea2-42f9-a391-0c282c24246a" pod="opendatahub/test-curl-7823-bed488"
Sep 17 22:11:44.759285 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:44.759266    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1784b293-6ea2-42f9-a391-0c282c24246a" path="/var/lib/kubelet/pods/1784b293-6ea2-42f9-a391-0c282c24246a/volumes"
Sep 17 22:11:53.216340 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:11:53.216306    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"]
Sep 17 22:12:08.236462 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.236417    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp" podUID="c12d0714-cb27-46cf-bdca-25dec1611ce7" containerName="main" containerID="cri-o://494b0048ca3e4d06b8fbb8e36a8ca40cdd440a65c72412e7f47c9678a3d1486d" gracePeriod=45
Sep 17 22:12:08.490495 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.490438    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:12:08.557413 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.557375    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6pw2\" (UniqueName: \"kubernetes.io/projected/c12d0714-cb27-46cf-bdca-25dec1611ce7-kube-api-access-r6pw2\") pod \"c12d0714-cb27-46cf-bdca-25dec1611ce7\" (UID: \"c12d0714-cb27-46cf-bdca-25dec1611ce7\") "
Sep 17 22:12:08.557413 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.557413    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12d0714-cb27-46cf-bdca-25dec1611ce7-model-cache\") pod \"c12d0714-cb27-46cf-bdca-25dec1611ce7\" (UID: \"c12d0714-cb27-46cf-bdca-25dec1611ce7\") "
Sep 17 22:12:08.557676 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.557435    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12d0714-cb27-46cf-bdca-25dec1611ce7-tls-certs\") pod \"c12d0714-cb27-46cf-bdca-25dec1611ce7\" (UID: \"c12d0714-cb27-46cf-bdca-25dec1611ce7\") "
Sep 17 22:12:08.557676 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.557458    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12d0714-cb27-46cf-bdca-25dec1611ce7-dshm\") pod \"c12d0714-cb27-46cf-bdca-25dec1611ce7\" (UID: \"c12d0714-cb27-46cf-bdca-25dec1611ce7\") "
Sep 17 22:12:08.557676 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.557477    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12d0714-cb27-46cf-bdca-25dec1611ce7-tmp-dir\") pod \"c12d0714-cb27-46cf-bdca-25dec1611ce7\" (UID: \"c12d0714-cb27-46cf-bdca-25dec1611ce7\") "
Sep 17 22:12:08.557676 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.557497    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12d0714-cb27-46cf-bdca-25dec1611ce7-home\") pod \"c12d0714-cb27-46cf-bdca-25dec1611ce7\" (UID: \"c12d0714-cb27-46cf-bdca-25dec1611ce7\") "
Sep 17 22:12:08.557866 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.557766    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12d0714-cb27-46cf-bdca-25dec1611ce7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c12d0714-cb27-46cf-bdca-25dec1611ce7" (UID: "c12d0714-cb27-46cf-bdca-25dec1611ce7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:12:08.557866 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.557764    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12d0714-cb27-46cf-bdca-25dec1611ce7-model-cache" (OuterVolumeSpecName: "model-cache") pod "c12d0714-cb27-46cf-bdca-25dec1611ce7" (UID: "c12d0714-cb27-46cf-bdca-25dec1611ce7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:12:08.557866 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.557804    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12d0714-cb27-46cf-bdca-25dec1611ce7-home" (OuterVolumeSpecName: "home") pod "c12d0714-cb27-46cf-bdca-25dec1611ce7" (UID: "c12d0714-cb27-46cf-bdca-25dec1611ce7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:12:08.559849 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.559816    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12d0714-cb27-46cf-bdca-25dec1611ce7-dshm" (OuterVolumeSpecName: "dshm") pod "c12d0714-cb27-46cf-bdca-25dec1611ce7" (UID: "c12d0714-cb27-46cf-bdca-25dec1611ce7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:12:08.559964 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.559875    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c12d0714-cb27-46cf-bdca-25dec1611ce7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c12d0714-cb27-46cf-bdca-25dec1611ce7" (UID: "c12d0714-cb27-46cf-bdca-25dec1611ce7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:12:08.559964 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.559904    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12d0714-cb27-46cf-bdca-25dec1611ce7-kube-api-access-r6pw2" (OuterVolumeSpecName: "kube-api-access-r6pw2") pod "c12d0714-cb27-46cf-bdca-25dec1611ce7" (UID: "c12d0714-cb27-46cf-bdca-25dec1611ce7"). InnerVolumeSpecName "kube-api-access-r6pw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:12:08.659101 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.659064    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6pw2\" (UniqueName: \"kubernetes.io/projected/c12d0714-cb27-46cf-bdca-25dec1611ce7-kube-api-access-r6pw2\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:12:08.659101 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.659095    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12d0714-cb27-46cf-bdca-25dec1611ce7-model-cache\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:12:08.659332 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.659125    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12d0714-cb27-46cf-bdca-25dec1611ce7-tls-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:12:08.659332 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.659133    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12d0714-cb27-46cf-bdca-25dec1611ce7-dshm\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:12:08.659332 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.659140    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12d0714-cb27-46cf-bdca-25dec1611ce7-tmp-dir\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:12:08.659332 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.659147    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12d0714-cb27-46cf-bdca-25dec1611ce7-home\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:12:08.770027 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.769998    2564 generic.go:356] "Generic (PLEG): container finished" podID="c12d0714-cb27-46cf-bdca-25dec1611ce7" containerID="494b0048ca3e4d06b8fbb8e36a8ca40cdd440a65c72412e7f47c9678a3d1486d" exitCode=0
Sep 17 22:12:08.770157 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.770028    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"
Sep 17 22:12:08.770157 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.770071    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp" event={"ID":"c12d0714-cb27-46cf-bdca-25dec1611ce7","Type":"ContainerDied","Data":"494b0048ca3e4d06b8fbb8e36a8ca40cdd440a65c72412e7f47c9678a3d1486d"}
Sep 17 22:12:08.770157 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.770110    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp" event={"ID":"c12d0714-cb27-46cf-bdca-25dec1611ce7","Type":"ContainerDied","Data":"146caaa7d97fa49daac69c6fa86b1f550fc68abb5ce4e45babb4729814c5c20c"}
Sep 17 22:12:08.770157 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.770130    2564 scope.go:117] "RemoveContainer" containerID="494b0048ca3e4d06b8fbb8e36a8ca40cdd440a65c72412e7f47c9678a3d1486d"
Sep 17 22:12:08.784796 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.779527    2564 scope.go:117] "RemoveContainer" containerID="494b0048ca3e4d06b8fbb8e36a8ca40cdd440a65c72412e7f47c9678a3d1486d"
Sep 17 22:12:08.784943 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:12:08.784918    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"494b0048ca3e4d06b8fbb8e36a8ca40cdd440a65c72412e7f47c9678a3d1486d\": container with ID starting with 494b0048ca3e4d06b8fbb8e36a8ca40cdd440a65c72412e7f47c9678a3d1486d not found: ID does not exist" containerID="494b0048ca3e4d06b8fbb8e36a8ca40cdd440a65c72412e7f47c9678a3d1486d"
Sep 17 22:12:08.785009 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.784956    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"494b0048ca3e4d06b8fbb8e36a8ca40cdd440a65c72412e7f47c9678a3d1486d"} err="failed to get container status \"494b0048ca3e4d06b8fbb8e36a8ca40cdd440a65c72412e7f47c9678a3d1486d\": rpc error: code = NotFound desc = could not find container \"494b0048ca3e4d06b8fbb8e36a8ca40cdd440a65c72412e7f47c9678a3d1486d\": container with ID starting with 494b0048ca3e4d06b8fbb8e36a8ca40cdd440a65c72412e7f47c9678a3d1486d not found: ID does not exist"
Sep 17 22:12:08.789793 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.789769    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"]
Sep 17 22:12:08.796670 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:08.796649    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f8dad/auth-test-model-8f8dad-kserve-7f4967c9ff-zsjfp"]
Sep 17 22:12:10.753972 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:10.753929    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c12d0714-cb27-46cf-bdca-25dec1611ce7" path="/var/lib/kubelet/pods/c12d0714-cb27-46cf-bdca-25dec1611ce7/volumes"
Sep 17 22:12:22.867358 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:22.867317    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"]
Sep 17 22:12:22.868845 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:22.868819    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1784b293-6ea2-42f9-a391-0c282c24246a" containerName="test-curl-7823-bed488"
Sep 17 22:12:22.869002 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:22.868991    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1784b293-6ea2-42f9-a391-0c282c24246a" containerName="test-curl-7823-bed488"
Sep 17 22:12:22.869110 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:22.869100    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c12d0714-cb27-46cf-bdca-25dec1611ce7" containerName="main"
Sep 17 22:12:22.869214 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:22.869203    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c12d0714-cb27-46cf-bdca-25dec1611ce7" containerName="main"
Sep 17 22:12:22.869495 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:22.869480    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c12d0714-cb27-46cf-bdca-25dec1611ce7" containerName="main"
Sep 17 22:12:22.869600 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:22.869590    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1784b293-6ea2-42f9-a391-0c282c24246a" containerName="test-curl-7823-bed488"
Sep 17 22:12:22.874813 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:22.874790    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:12:22.877617 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:22.877588    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"]
Sep 17 22:12:22.878109 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:22.877663    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8f8dad\"/\"openshift-service-ca.crt\""
Sep 17 22:12:22.879030 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:22.878999    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8f8dad\"/\"sub-test-model-8f8dad-kserve-self-signed-certs\""
Sep 17 22:12:22.879030 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:22.879002    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8f8dad\"/\"kube-root-ca.crt\""
Sep 17 22:12:22.879227 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:22.879049    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8f8dad\"/\"default-dockercfg-q655d\""
Sep 17 22:12:23.011302 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:23.011268    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78cd147f-323e-4882-8571-bbd40cff6def-tmp-dir\") pod \"sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5\" (UID: \"78cd147f-323e-4882-8571-bbd40cff6def\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:12:23.011488 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:23.011324    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78cd147f-323e-4882-8571-bbd40cff6def-home\") pod \"sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5\" (UID: \"78cd147f-323e-4882-8571-bbd40cff6def\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:12:23.011488 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:23.011386    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78cd147f-323e-4882-8571-bbd40cff6def-tls-certs\") pod \"sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5\" (UID: \"78cd147f-323e-4882-8571-bbd40cff6def\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:12:23.011488 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:23.011409    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78cd147f-323e-4882-8571-bbd40cff6def-model-cache\") pod \"sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5\" (UID: \"78cd147f-323e-4882-8571-bbd40cff6def\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:12:23.011488 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:23.011436    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbwsh\" (UniqueName: \"kubernetes.io/projected/78cd147f-323e-4882-8571-bbd40cff6def-kube-api-access-mbwsh\") pod \"sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5\" (UID: \"78cd147f-323e-4882-8571-bbd40cff6def\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:12:23.011488 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:23.011464    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78cd147f-323e-4882-8571-bbd40cff6def-dshm\") pod \"sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5\" (UID: \"78cd147f-323e-4882-8571-bbd40cff6def\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:12:23.112108 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:23.112065    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78cd147f-323e-4882-8571-bbd40cff6def-model-cache\") pod \"sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5\" (UID: \"78cd147f-323e-4882-8571-bbd40cff6def\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:12:23.112108 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:23.112107    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbwsh\" (UniqueName: \"kubernetes.io/projected/78cd147f-323e-4882-8571-bbd40cff6def-kube-api-access-mbwsh\") pod \"sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5\" (UID: \"78cd147f-323e-4882-8571-bbd40cff6def\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:12:23.112388 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:23.112139    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78cd147f-323e-4882-8571-bbd40cff6def-dshm\") pod \"sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5\" (UID: \"78cd147f-323e-4882-8571-bbd40cff6def\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:12:23.112388 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:23.112189    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78cd147f-323e-4882-8571-bbd40cff6def-tmp-dir\") pod \"sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5\" (UID: \"78cd147f-323e-4882-8571-bbd40cff6def\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:12:23.112388 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:23.112222    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78cd147f-323e-4882-8571-bbd40cff6def-home\") pod \"sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5\" (UID: \"78cd147f-323e-4882-8571-bbd40cff6def\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:12:23.112388 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:23.112257    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78cd147f-323e-4882-8571-bbd40cff6def-tls-certs\") pod \"sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5\" (UID: \"78cd147f-323e-4882-8571-bbd40cff6def\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:12:23.112605 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:23.112562    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78cd147f-323e-4882-8571-bbd40cff6def-model-cache\") pod \"sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5\" (UID: \"78cd147f-323e-4882-8571-bbd40cff6def\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:12:23.112664 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:23.112604    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78cd147f-323e-4882-8571-bbd40cff6def-tmp-dir\") pod \"sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5\" (UID: \"78cd147f-323e-4882-8571-bbd40cff6def\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:12:23.112706 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:23.112666    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78cd147f-323e-4882-8571-bbd40cff6def-home\") pod \"sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5\" (UID: \"78cd147f-323e-4882-8571-bbd40cff6def\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:12:23.114587 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:23.114568    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78cd147f-323e-4882-8571-bbd40cff6def-dshm\") pod \"sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5\" (UID: \"78cd147f-323e-4882-8571-bbd40cff6def\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:12:23.114918 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:23.114891    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78cd147f-323e-4882-8571-bbd40cff6def-tls-certs\") pod \"sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5\" (UID: \"78cd147f-323e-4882-8571-bbd40cff6def\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:12:23.122406 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:23.122354    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbwsh\" (UniqueName: \"kubernetes.io/projected/78cd147f-323e-4882-8571-bbd40cff6def-kube-api-access-mbwsh\") pod \"sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5\" (UID: \"78cd147f-323e-4882-8571-bbd40cff6def\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:12:23.186007 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:23.185973    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:12:23.460104 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:23.460080    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"]
Sep 17 22:12:23.461819 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:12:23.461793    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78cd147f_323e_4882_8571_bbd40cff6def.slice/crio-7d85bedb42e0bb3cc78eb28cb5c3fc84bb3c42cd92a9c4ae21fae555104d0eda WatchSource:0}: Error finding container 7d85bedb42e0bb3cc78eb28cb5c3fc84bb3c42cd92a9c4ae21fae555104d0eda: Status 404 returned error can't find the container with id 7d85bedb42e0bb3cc78eb28cb5c3fc84bb3c42cd92a9c4ae21fae555104d0eda
Sep 17 22:12:23.853299 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:23.853207    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5" event={"ID":"78cd147f-323e-4882-8571-bbd40cff6def","Type":"ContainerStarted","Data":"47164d683f237751995feaa2a7e965befa6504022742246c81059224979b6e5d"}
Sep 17 22:12:23.853299 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:23.853260    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5" event={"ID":"78cd147f-323e-4882-8571-bbd40cff6def","Type":"ContainerStarted","Data":"7d85bedb42e0bb3cc78eb28cb5c3fc84bb3c42cd92a9c4ae21fae555104d0eda"}
Sep 17 22:12:23.875380 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:23.875328    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5" podStartSLOduration=1.87530974 podStartE2EDuration="1.87530974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:12:23.873029793 +0000 UTC m=+1859.691356104" watchObservedRunningTime="2026-09-17 22:12:23.87530974 +0000 UTC m=+1859.693636051"
Sep 17 22:12:33.186117 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:33.186072    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:12:33.186117 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:33.186130    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:12:33.199426 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:33.199395    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:12:33.917913 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:12:33.917882    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:13:03.073852 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.073772    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"]
Sep 17 22:13:03.077136 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.077118    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:13:03.082128 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.082101    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-35b181\"/\"kube-root-ca.crt\""
Sep 17 22:13:03.082302 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.082103    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-35b181\"/\"default-dockercfg-r8vjk\""
Sep 17 22:13:03.082302 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.082103    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-35b181\"/\"sub-test-model-35b181-kserve-self-signed-certs\""
Sep 17 22:13:03.082596 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.082581    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-35b181\"/\"openshift-service-ca.crt\""
Sep 17 22:13:03.090851 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.090823    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"]
Sep 17 22:13:03.183121 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.183087    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-dshm\") pod \"sub-test-model-35b181-kserve-65cd8ff755-x4gw5\" (UID: \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\") " pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:13:03.183325 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.183203    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-tmp-dir\") pod \"sub-test-model-35b181-kserve-65cd8ff755-x4gw5\" (UID: \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\") " pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:13:03.183325 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.183252    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-home\") pod \"sub-test-model-35b181-kserve-65cd8ff755-x4gw5\" (UID: \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\") " pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:13:03.183421 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.183327    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-model-cache\") pod \"sub-test-model-35b181-kserve-65cd8ff755-x4gw5\" (UID: \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\") " pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:13:03.183421 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.183355    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dppm\" (UniqueName: \"kubernetes.io/projected/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-kube-api-access-7dppm\") pod \"sub-test-model-35b181-kserve-65cd8ff755-x4gw5\" (UID: \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\") " pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:13:03.183421 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.183400    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-tls-certs\") pod \"sub-test-model-35b181-kserve-65cd8ff755-x4gw5\" (UID: \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\") " pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:13:03.284936 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.284892    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-dshm\") pod \"sub-test-model-35b181-kserve-65cd8ff755-x4gw5\" (UID: \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\") " pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:13:03.285141 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.284977    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-tmp-dir\") pod \"sub-test-model-35b181-kserve-65cd8ff755-x4gw5\" (UID: \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\") " pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:13:03.285141 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.285008    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-home\") pod \"sub-test-model-35b181-kserve-65cd8ff755-x4gw5\" (UID: \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\") " pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:13:03.285141 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.285046    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-model-cache\") pod \"sub-test-model-35b181-kserve-65cd8ff755-x4gw5\" (UID: \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\") " pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:13:03.285141 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.285072    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dppm\" (UniqueName: \"kubernetes.io/projected/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-kube-api-access-7dppm\") pod \"sub-test-model-35b181-kserve-65cd8ff755-x4gw5\" (UID: \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\") " pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:13:03.285141 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.285107    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-tls-certs\") pod \"sub-test-model-35b181-kserve-65cd8ff755-x4gw5\" (UID: \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\") " pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:13:03.285475 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.285452    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-home\") pod \"sub-test-model-35b181-kserve-65cd8ff755-x4gw5\" (UID: \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\") " pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:13:03.285539 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.285491    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-tmp-dir\") pod \"sub-test-model-35b181-kserve-65cd8ff755-x4gw5\" (UID: \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\") " pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:13:03.285579 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.285556    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-model-cache\") pod \"sub-test-model-35b181-kserve-65cd8ff755-x4gw5\" (UID: \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\") " pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:13:03.287874 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.287842    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-dshm\") pod \"sub-test-model-35b181-kserve-65cd8ff755-x4gw5\" (UID: \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\") " pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:13:03.288281 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.288260    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-tls-certs\") pod \"sub-test-model-35b181-kserve-65cd8ff755-x4gw5\" (UID: \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\") " pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:13:03.294357 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.294332    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dppm\" (UniqueName: \"kubernetes.io/projected/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-kube-api-access-7dppm\") pod \"sub-test-model-35b181-kserve-65cd8ff755-x4gw5\" (UID: \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\") " pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:13:03.386263 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.386131    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:13:03.662142 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:03.662111    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"]
Sep 17 22:13:03.664270 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:13:03.664243    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7ff7f97_1dee_4fd0_8f6f_e0502e13076c.slice/crio-a24bcb8f1eaad12f2be1f9cdf5d39acff0e0fa2aa5f9751793a4c8de7f8cc416 WatchSource:0}: Error finding container a24bcb8f1eaad12f2be1f9cdf5d39acff0e0fa2aa5f9751793a4c8de7f8cc416: Status 404 returned error can't find the container with id a24bcb8f1eaad12f2be1f9cdf5d39acff0e0fa2aa5f9751793a4c8de7f8cc416
Sep 17 22:13:04.058619 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:04.058524    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5" event={"ID":"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c","Type":"ContainerStarted","Data":"c62c62a354603df146a6f1f3d588de8e9ef096ce34a46928b0f8590084fabb7f"}
Sep 17 22:13:04.058619 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:04.058566    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5" event={"ID":"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c","Type":"ContainerStarted","Data":"a24bcb8f1eaad12f2be1f9cdf5d39acff0e0fa2aa5f9751793a4c8de7f8cc416"}
Sep 17 22:13:04.078637 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:04.078583    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5" podStartSLOduration=1.078566843 podStartE2EDuration="1.078566843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:13:04.076897475 +0000 UTC m=+1899.895223789" watchObservedRunningTime="2026-09-17 22:13:04.078566843 +0000 UTC m=+1899.896893155"
Sep 17 22:13:13.386460 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:13.386416    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:13:13.387006 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:13.386512    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:13:13.402258 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:13.402228    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:13:14.123232 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:14.123203    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:13:51.015795 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:51.015757    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"]
Sep 17 22:13:52.040815 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:13:52.040780    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"]
Sep 17 22:14:06.035123 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.035059    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5" podUID="78cd147f-323e-4882-8571-bbd40cff6def" containerName="main" containerID="cri-o://47164d683f237751995feaa2a7e965befa6504022742246c81059224979b6e5d" gracePeriod=45
Sep 17 22:14:06.280595 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.280561    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:14:06.362053 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.361964    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78cd147f-323e-4882-8571-bbd40cff6def-tls-certs\") pod \"78cd147f-323e-4882-8571-bbd40cff6def\" (UID: \"78cd147f-323e-4882-8571-bbd40cff6def\") "
Sep 17 22:14:06.362053 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.362034    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78cd147f-323e-4882-8571-bbd40cff6def-tmp-dir\") pod \"78cd147f-323e-4882-8571-bbd40cff6def\" (UID: \"78cd147f-323e-4882-8571-bbd40cff6def\") "
Sep 17 22:14:06.362053 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.362054    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78cd147f-323e-4882-8571-bbd40cff6def-home\") pod \"78cd147f-323e-4882-8571-bbd40cff6def\" (UID: \"78cd147f-323e-4882-8571-bbd40cff6def\") "
Sep 17 22:14:06.362413 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.362078    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbwsh\" (UniqueName: \"kubernetes.io/projected/78cd147f-323e-4882-8571-bbd40cff6def-kube-api-access-mbwsh\") pod \"78cd147f-323e-4882-8571-bbd40cff6def\" (UID: \"78cd147f-323e-4882-8571-bbd40cff6def\") "
Sep 17 22:14:06.362413 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.362097    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78cd147f-323e-4882-8571-bbd40cff6def-dshm\") pod \"78cd147f-323e-4882-8571-bbd40cff6def\" (UID: \"78cd147f-323e-4882-8571-bbd40cff6def\") "
Sep 17 22:14:06.362413 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.362142    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78cd147f-323e-4882-8571-bbd40cff6def-model-cache\") pod \"78cd147f-323e-4882-8571-bbd40cff6def\" (UID: \"78cd147f-323e-4882-8571-bbd40cff6def\") "
Sep 17 22:14:06.362413 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.362350    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78cd147f-323e-4882-8571-bbd40cff6def-home" (OuterVolumeSpecName: "home") pod "78cd147f-323e-4882-8571-bbd40cff6def" (UID: "78cd147f-323e-4882-8571-bbd40cff6def"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:14:06.362643 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.362462    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78cd147f-323e-4882-8571-bbd40cff6def-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "78cd147f-323e-4882-8571-bbd40cff6def" (UID: "78cd147f-323e-4882-8571-bbd40cff6def"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:14:06.362643 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.362490    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78cd147f-323e-4882-8571-bbd40cff6def-home\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:14:06.362643 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.362494    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78cd147f-323e-4882-8571-bbd40cff6def-model-cache" (OuterVolumeSpecName: "model-cache") pod "78cd147f-323e-4882-8571-bbd40cff6def" (UID: "78cd147f-323e-4882-8571-bbd40cff6def"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:14:06.364436 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.364410    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78cd147f-323e-4882-8571-bbd40cff6def-dshm" (OuterVolumeSpecName: "dshm") pod "78cd147f-323e-4882-8571-bbd40cff6def" (UID: "78cd147f-323e-4882-8571-bbd40cff6def"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:14:06.364546 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.364477    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78cd147f-323e-4882-8571-bbd40cff6def-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "78cd147f-323e-4882-8571-bbd40cff6def" (UID: "78cd147f-323e-4882-8571-bbd40cff6def"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:14:06.364546 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.364492    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78cd147f-323e-4882-8571-bbd40cff6def-kube-api-access-mbwsh" (OuterVolumeSpecName: "kube-api-access-mbwsh") pod "78cd147f-323e-4882-8571-bbd40cff6def" (UID: "78cd147f-323e-4882-8571-bbd40cff6def"). InnerVolumeSpecName "kube-api-access-mbwsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:14:06.391502 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.391473    2564 generic.go:356] "Generic (PLEG): container finished" podID="78cd147f-323e-4882-8571-bbd40cff6def" containerID="47164d683f237751995feaa2a7e965befa6504022742246c81059224979b6e5d" exitCode=0
Sep 17 22:14:06.391647 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.391519    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"
Sep 17 22:14:06.391647 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.391521    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5" event={"ID":"78cd147f-323e-4882-8571-bbd40cff6def","Type":"ContainerDied","Data":"47164d683f237751995feaa2a7e965befa6504022742246c81059224979b6e5d"}
Sep 17 22:14:06.391647 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.391637    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5" event={"ID":"78cd147f-323e-4882-8571-bbd40cff6def","Type":"ContainerDied","Data":"7d85bedb42e0bb3cc78eb28cb5c3fc84bb3c42cd92a9c4ae21fae555104d0eda"}
Sep 17 22:14:06.391754 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.391657    2564 scope.go:117] "RemoveContainer" containerID="47164d683f237751995feaa2a7e965befa6504022742246c81059224979b6e5d"
Sep 17 22:14:06.402420 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.402399    2564 scope.go:117] "RemoveContainer" containerID="47164d683f237751995feaa2a7e965befa6504022742246c81059224979b6e5d"
Sep 17 22:14:06.402721 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:14:06.402704    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47164d683f237751995feaa2a7e965befa6504022742246c81059224979b6e5d\": container with ID starting with 47164d683f237751995feaa2a7e965befa6504022742246c81059224979b6e5d not found: ID does not exist" containerID="47164d683f237751995feaa2a7e965befa6504022742246c81059224979b6e5d"
Sep 17 22:14:06.402788 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.402728    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47164d683f237751995feaa2a7e965befa6504022742246c81059224979b6e5d"} err="failed to get container status \"47164d683f237751995feaa2a7e965befa6504022742246c81059224979b6e5d\": rpc error: code = NotFound desc = could not find container \"47164d683f237751995feaa2a7e965befa6504022742246c81059224979b6e5d\": container with ID starting with 47164d683f237751995feaa2a7e965befa6504022742246c81059224979b6e5d not found: ID does not exist"
Sep 17 22:14:06.414939 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.414905    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"]
Sep 17 22:14:06.421298 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.421273    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f8dad/sub-test-model-8f8dad-kserve-59d5d7ddd9-j6ll5"]
Sep 17 22:14:06.463681 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.463651    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78cd147f-323e-4882-8571-bbd40cff6def-tls-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:14:06.463681 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.463677    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78cd147f-323e-4882-8571-bbd40cff6def-tmp-dir\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:14:06.463681 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.463686    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbwsh\" (UniqueName: \"kubernetes.io/projected/78cd147f-323e-4882-8571-bbd40cff6def-kube-api-access-mbwsh\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:14:06.463888 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.463695    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78cd147f-323e-4882-8571-bbd40cff6def-dshm\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:14:06.463888 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.463705    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78cd147f-323e-4882-8571-bbd40cff6def-model-cache\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:14:06.754237 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:06.754202    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78cd147f-323e-4882-8571-bbd40cff6def" path="/var/lib/kubelet/pods/78cd147f-323e-4882-8571-bbd40cff6def/volumes"
Sep 17 22:14:07.061431 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.061321    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5" podUID="d7ff7f97-1dee-4fd0-8f6f-e0502e13076c" containerName="main" containerID="cri-o://c62c62a354603df146a6f1f3d588de8e9ef096ce34a46928b0f8590084fabb7f" gracePeriod=45
Sep 17 22:14:07.317394 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.317323    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:14:07.372028 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.371997    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-tls-certs\") pod \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\" (UID: \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\") "
Sep 17 22:14:07.372218 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.372049    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-tmp-dir\") pod \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\" (UID: \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\") "
Sep 17 22:14:07.372218 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.372080    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-dshm\") pod \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\" (UID: \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\") "
Sep 17 22:14:07.372218 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.372198    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-home\") pod \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\" (UID: \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\") "
Sep 17 22:14:07.372396 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.372256    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-model-cache\") pod \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\" (UID: \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\") "
Sep 17 22:14:07.372396 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.372284    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dppm\" (UniqueName: \"kubernetes.io/projected/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-kube-api-access-7dppm\") pod \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\" (UID: \"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c\") "
Sep 17 22:14:07.372396 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.372356    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d7ff7f97-1dee-4fd0-8f6f-e0502e13076c" (UID: "d7ff7f97-1dee-4fd0-8f6f-e0502e13076c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:14:07.372551 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.372485    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-home" (OuterVolumeSpecName: "home") pod "d7ff7f97-1dee-4fd0-8f6f-e0502e13076c" (UID: "d7ff7f97-1dee-4fd0-8f6f-e0502e13076c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:14:07.372551 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.372498    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-model-cache" (OuterVolumeSpecName: "model-cache") pod "d7ff7f97-1dee-4fd0-8f6f-e0502e13076c" (UID: "d7ff7f97-1dee-4fd0-8f6f-e0502e13076c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:14:07.372667 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.372647    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-tmp-dir\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:14:07.372721 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.372672    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-home\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:14:07.372721 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.372686    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-model-cache\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:14:07.374460 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.374440    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d7ff7f97-1dee-4fd0-8f6f-e0502e13076c" (UID: "d7ff7f97-1dee-4fd0-8f6f-e0502e13076c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:14:07.374542 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.374462    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-dshm" (OuterVolumeSpecName: "dshm") pod "d7ff7f97-1dee-4fd0-8f6f-e0502e13076c" (UID: "d7ff7f97-1dee-4fd0-8f6f-e0502e13076c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:14:07.374582 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.374548    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-kube-api-access-7dppm" (OuterVolumeSpecName: "kube-api-access-7dppm") pod "d7ff7f97-1dee-4fd0-8f6f-e0502e13076c" (UID: "d7ff7f97-1dee-4fd0-8f6f-e0502e13076c"). InnerVolumeSpecName "kube-api-access-7dppm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:14:07.399729 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.399688    2564 generic.go:356] "Generic (PLEG): container finished" podID="d7ff7f97-1dee-4fd0-8f6f-e0502e13076c" containerID="c62c62a354603df146a6f1f3d588de8e9ef096ce34a46928b0f8590084fabb7f" exitCode=0
Sep 17 22:14:07.399729 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.399733    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"
Sep 17 22:14:07.399929 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.399744    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5" event={"ID":"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c","Type":"ContainerDied","Data":"c62c62a354603df146a6f1f3d588de8e9ef096ce34a46928b0f8590084fabb7f"}
Sep 17 22:14:07.399929 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.399782    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5" event={"ID":"d7ff7f97-1dee-4fd0-8f6f-e0502e13076c","Type":"ContainerDied","Data":"a24bcb8f1eaad12f2be1f9cdf5d39acff0e0fa2aa5f9751793a4c8de7f8cc416"}
Sep 17 22:14:07.399929 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.399806    2564 scope.go:117] "RemoveContainer" containerID="c62c62a354603df146a6f1f3d588de8e9ef096ce34a46928b0f8590084fabb7f"
Sep 17 22:14:07.409388 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.409370    2564 scope.go:117] "RemoveContainer" containerID="c62c62a354603df146a6f1f3d588de8e9ef096ce34a46928b0f8590084fabb7f"
Sep 17 22:14:07.409648 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:14:07.409630    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c62c62a354603df146a6f1f3d588de8e9ef096ce34a46928b0f8590084fabb7f\": container with ID starting with c62c62a354603df146a6f1f3d588de8e9ef096ce34a46928b0f8590084fabb7f not found: ID does not exist" containerID="c62c62a354603df146a6f1f3d588de8e9ef096ce34a46928b0f8590084fabb7f"
Sep 17 22:14:07.409691 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.409657    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c62c62a354603df146a6f1f3d588de8e9ef096ce34a46928b0f8590084fabb7f"} err="failed to get container status \"c62c62a354603df146a6f1f3d588de8e9ef096ce34a46928b0f8590084fabb7f\": rpc error: code = NotFound desc = could not find container \"c62c62a354603df146a6f1f3d588de8e9ef096ce34a46928b0f8590084fabb7f\": container with ID starting with c62c62a354603df146a6f1f3d588de8e9ef096ce34a46928b0f8590084fabb7f not found: ID does not exist"
Sep 17 22:14:07.422186 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.422143    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"]
Sep 17 22:14:07.428378 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.428353    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-35b181/sub-test-model-35b181-kserve-65cd8ff755-x4gw5"]
Sep 17 22:14:07.473226 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.473160    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-tls-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:14:07.473226 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.473221    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-dshm\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:14:07.473226 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:07.473231    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dppm\" (UniqueName: \"kubernetes.io/projected/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c-kube-api-access-7dppm\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:14:08.754030 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:08.753995    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7ff7f97-1dee-4fd0-8f6f-e0502e13076c" path="/var/lib/kubelet/pods/d7ff7f97-1dee-4fd0-8f6f-e0502e13076c/volumes"
Sep 17 22:14:09.857087 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.857048    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"]
Sep 17 22:14:09.857623 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.857604    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7ff7f97-1dee-4fd0-8f6f-e0502e13076c" containerName="main"
Sep 17 22:14:09.857705 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.857625    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7ff7f97-1dee-4fd0-8f6f-e0502e13076c" containerName="main"
Sep 17 22:14:09.857705 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.857647    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78cd147f-323e-4882-8571-bbd40cff6def" containerName="main"
Sep 17 22:14:09.857705 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.857655    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="78cd147f-323e-4882-8571-bbd40cff6def" containerName="main"
Sep 17 22:14:09.857875 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.857739    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="78cd147f-323e-4882-8571-bbd40cff6def" containerName="main"
Sep 17 22:14:09.857875 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.857752    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7ff7f97-1dee-4fd0-8f6f-e0502e13076c" containerName="main"
Sep 17 22:14:09.862036 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.862013    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:14:09.864973 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.864958    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8f8dad\"/\"kube-root-ca.crt\""
Sep 17 22:14:09.866362 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.866342    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8f8dad\"/\"default-dockercfg-q655d\""
Sep 17 22:14:09.866484 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.866468    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8f8dad\"/\"openshift-service-ca.crt\""
Sep 17 22:14:09.866484 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.866475    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8f8dad\"/\"rate-test-model-8f8dad-kserve-self-signed-certs\""
Sep 17 22:14:09.870572 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.870549    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"]
Sep 17 22:14:09.896850 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.896821    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35ee99a4-6c69-4429-92c2-9d793f198d02-tls-certs\") pod \"rate-test-model-8f8dad-kserve-5df4b66767-r2tfh\" (UID: \"35ee99a4-6c69-4429-92c2-9d793f198d02\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:14:09.896850 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.896864    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbv4r\" (UniqueName: \"kubernetes.io/projected/35ee99a4-6c69-4429-92c2-9d793f198d02-kube-api-access-jbv4r\") pod \"rate-test-model-8f8dad-kserve-5df4b66767-r2tfh\" (UID: \"35ee99a4-6c69-4429-92c2-9d793f198d02\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:14:09.897084 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.896983    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35ee99a4-6c69-4429-92c2-9d793f198d02-tmp-dir\") pod \"rate-test-model-8f8dad-kserve-5df4b66767-r2tfh\" (UID: \"35ee99a4-6c69-4429-92c2-9d793f198d02\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:14:09.897084 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.897021    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35ee99a4-6c69-4429-92c2-9d793f198d02-model-cache\") pod \"rate-test-model-8f8dad-kserve-5df4b66767-r2tfh\" (UID: \"35ee99a4-6c69-4429-92c2-9d793f198d02\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:14:09.897213 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.897091    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35ee99a4-6c69-4429-92c2-9d793f198d02-home\") pod \"rate-test-model-8f8dad-kserve-5df4b66767-r2tfh\" (UID: \"35ee99a4-6c69-4429-92c2-9d793f198d02\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:14:09.897213 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.897161    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35ee99a4-6c69-4429-92c2-9d793f198d02-dshm\") pod \"rate-test-model-8f8dad-kserve-5df4b66767-r2tfh\" (UID: \"35ee99a4-6c69-4429-92c2-9d793f198d02\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:14:09.998130 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.998096    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35ee99a4-6c69-4429-92c2-9d793f198d02-dshm\") pod \"rate-test-model-8f8dad-kserve-5df4b66767-r2tfh\" (UID: \"35ee99a4-6c69-4429-92c2-9d793f198d02\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:14:09.998499 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.998443    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35ee99a4-6c69-4429-92c2-9d793f198d02-tls-certs\") pod \"rate-test-model-8f8dad-kserve-5df4b66767-r2tfh\" (UID: \"35ee99a4-6c69-4429-92c2-9d793f198d02\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:14:09.998622 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.998537    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbv4r\" (UniqueName: \"kubernetes.io/projected/35ee99a4-6c69-4429-92c2-9d793f198d02-kube-api-access-jbv4r\") pod \"rate-test-model-8f8dad-kserve-5df4b66767-r2tfh\" (UID: \"35ee99a4-6c69-4429-92c2-9d793f198d02\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:14:09.998687 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.998624    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35ee99a4-6c69-4429-92c2-9d793f198d02-tmp-dir\") pod \"rate-test-model-8f8dad-kserve-5df4b66767-r2tfh\" (UID: \"35ee99a4-6c69-4429-92c2-9d793f198d02\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:14:09.998687 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.998650    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35ee99a4-6c69-4429-92c2-9d793f198d02-model-cache\") pod \"rate-test-model-8f8dad-kserve-5df4b66767-r2tfh\" (UID: \"35ee99a4-6c69-4429-92c2-9d793f198d02\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:14:09.998861 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.998689    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35ee99a4-6c69-4429-92c2-9d793f198d02-home\") pod \"rate-test-model-8f8dad-kserve-5df4b66767-r2tfh\" (UID: \"35ee99a4-6c69-4429-92c2-9d793f198d02\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:14:09.999089 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.999067    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35ee99a4-6c69-4429-92c2-9d793f198d02-home\") pod \"rate-test-model-8f8dad-kserve-5df4b66767-r2tfh\" (UID: \"35ee99a4-6c69-4429-92c2-9d793f198d02\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:14:09.999253 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.999151    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35ee99a4-6c69-4429-92c2-9d793f198d02-tmp-dir\") pod \"rate-test-model-8f8dad-kserve-5df4b66767-r2tfh\" (UID: \"35ee99a4-6c69-4429-92c2-9d793f198d02\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:14:09.999620 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:09.999600    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35ee99a4-6c69-4429-92c2-9d793f198d02-model-cache\") pod \"rate-test-model-8f8dad-kserve-5df4b66767-r2tfh\" (UID: \"35ee99a4-6c69-4429-92c2-9d793f198d02\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:14:10.002504 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:10.002459    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35ee99a4-6c69-4429-92c2-9d793f198d02-tls-certs\") pod \"rate-test-model-8f8dad-kserve-5df4b66767-r2tfh\" (UID: \"35ee99a4-6c69-4429-92c2-9d793f198d02\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:14:10.003994 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:10.003967    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35ee99a4-6c69-4429-92c2-9d793f198d02-dshm\") pod \"rate-test-model-8f8dad-kserve-5df4b66767-r2tfh\" (UID: \"35ee99a4-6c69-4429-92c2-9d793f198d02\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:14:10.008924 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:10.008897    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbv4r\" (UniqueName: \"kubernetes.io/projected/35ee99a4-6c69-4429-92c2-9d793f198d02-kube-api-access-jbv4r\") pod \"rate-test-model-8f8dad-kserve-5df4b66767-r2tfh\" (UID: \"35ee99a4-6c69-4429-92c2-9d793f198d02\") " pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:14:10.172894 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:10.172858    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:14:10.448050 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:10.448018    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"]
Sep 17 22:14:10.449606 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:14:10.449580    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35ee99a4_6c69_4429_92c2_9d793f198d02.slice/crio-1199cd9475fbed73d616a8e1523668f98cc2167a21a3d95570bb26e65474b291 WatchSource:0}: Error finding container 1199cd9475fbed73d616a8e1523668f98cc2167a21a3d95570bb26e65474b291: Status 404 returned error can't find the container with id 1199cd9475fbed73d616a8e1523668f98cc2167a21a3d95570bb26e65474b291
Sep 17 22:14:11.424584 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:11.424543    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh" event={"ID":"35ee99a4-6c69-4429-92c2-9d793f198d02","Type":"ContainerStarted","Data":"059e73caf24e4cb50660147db902e0681b2bbcb696ff78f0fdd1cb43ac7b21e1"}
Sep 17 22:14:11.425016 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:11.424597    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh" event={"ID":"35ee99a4-6c69-4429-92c2-9d793f198d02","Type":"ContainerStarted","Data":"1199cd9475fbed73d616a8e1523668f98cc2167a21a3d95570bb26e65474b291"}
Sep 17 22:14:11.444757 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:11.444705    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh" podStartSLOduration=2.444687753 podStartE2EDuration="2.444687753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:14:11.443277988 +0000 UTC m=+1967.261604301" watchObservedRunningTime="2026-09-17 22:14:11.444687753 +0000 UTC m=+1967.263014090"
Sep 17 22:14:20.173389 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:20.173289    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:14:20.173389 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:20.173377    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:14:20.186089 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:20.186064    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:14:20.484269 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:14:20.484189    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:15:00.140724 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:00.140688    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8f8dad-29828055-v8dxk"]
Sep 17 22:15:00.144069 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:00.144053    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-35b181-29828055-v5dl5"]
Sep 17 22:15:00.144193 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:00.144160    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8f8dad-29828055-v8dxk"
Sep 17 22:15:00.147202 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:00.147164    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-35b181-29828055-v5dl5"
Sep 17 22:15:00.155690 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:00.155667    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmdbp\""
Sep 17 22:15:00.161065 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:00.161049    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-35b181-29828055-v5dl5"
Sep 17 22:15:00.164399 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:00.164160    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8f8dad-29828055-v8dxk"]
Sep 17 22:15:00.164399 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:00.164202    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8f8dad-29828055-v8dxk"
Sep 17 22:15:00.168937 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:00.168657    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-35b181-29828055-v5dl5"]
Sep 17 22:15:00.444317 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:00.444287    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-35b181-29828055-v5dl5"]
Sep 17 22:15:00.445378 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:15:00.445351    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93756527_cf7c_4fb6_a6d0_4b8cad8fc89a.slice/crio-ff978719d97bd991d01b82033386adde318fc48a8e36f243db7976636e84af43 WatchSource:0}: Error finding container ff978719d97bd991d01b82033386adde318fc48a8e36f243db7976636e84af43: Status 404 returned error can't find the container with id ff978719d97bd991d01b82033386adde318fc48a8e36f243db7976636e84af43
Sep 17 22:15:00.449645 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:00.449622    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8f8dad-29828055-v8dxk"]
Sep 17 22:15:00.450260 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:15:00.450239    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode987c5ac_a0f3_4fbc_b4e3_a7eb1ea01899.slice/crio-807fee881d915e7e5ecca6dc4d854dcc08bf44afd049a3ab9ee45fb144f5591d WatchSource:0}: Error finding container 807fee881d915e7e5ecca6dc4d854dcc08bf44afd049a3ab9ee45fb144f5591d: Status 404 returned error can't find the container with id 807fee881d915e7e5ecca6dc4d854dcc08bf44afd049a3ab9ee45fb144f5591d
Sep 17 22:15:00.688350 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:00.688315    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-35b181-29828055-v5dl5" event={"ID":"93756527-cf7c-4fb6-a6d0-4b8cad8fc89a","Type":"ContainerStarted","Data":"ff978719d97bd991d01b82033386adde318fc48a8e36f243db7976636e84af43"}
Sep 17 22:15:00.689859 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:00.689834    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8f8dad-29828055-v8dxk" event={"ID":"e987c5ac-a0f3-4fbc-b4e3-a7eb1ea01899","Type":"ContainerStarted","Data":"807fee881d915e7e5ecca6dc4d854dcc08bf44afd049a3ab9ee45fb144f5591d"}
Sep 17 22:15:01.696194 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:01.696087    2564 generic.go:356] "Generic (PLEG): container finished" podID="93756527-cf7c-4fb6-a6d0-4b8cad8fc89a" containerID="2ca8a3a494fc20bd5661bc07164e971656a1a37f3e5a6ec5a0bf9ab25e7d4375" exitCode=0
Sep 17 22:15:01.696598 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:01.696198    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-35b181-29828055-v5dl5" event={"ID":"93756527-cf7c-4fb6-a6d0-4b8cad8fc89a","Type":"ContainerDied","Data":"2ca8a3a494fc20bd5661bc07164e971656a1a37f3e5a6ec5a0bf9ab25e7d4375"}
Sep 17 22:15:01.697892 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:01.697865    2564 generic.go:356] "Generic (PLEG): container finished" podID="e987c5ac-a0f3-4fbc-b4e3-a7eb1ea01899" containerID="a76c6114261abfbde2eea1cbc5d59e7906cc0396eae905ae2ba1424cbc36972f" exitCode=0
Sep 17 22:15:01.697988 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:01.697903    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8f8dad-29828055-v8dxk" event={"ID":"e987c5ac-a0f3-4fbc-b4e3-a7eb1ea01899","Type":"ContainerDied","Data":"a76c6114261abfbde2eea1cbc5d59e7906cc0396eae905ae2ba1424cbc36972f"}
Sep 17 22:15:02.849293 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:02.849271    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-35b181-29828055-v5dl5"
Sep 17 22:15:02.852318 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:02.852300    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8f8dad-29828055-v8dxk"
Sep 17 22:15:03.709228 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:03.709199    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8f8dad-29828055-v8dxk"
Sep 17 22:15:03.709228 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:03.709221    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8f8dad-29828055-v8dxk" event={"ID":"e987c5ac-a0f3-4fbc-b4e3-a7eb1ea01899","Type":"ContainerDied","Data":"807fee881d915e7e5ecca6dc4d854dcc08bf44afd049a3ab9ee45fb144f5591d"}
Sep 17 22:15:03.709488 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:03.709277    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="807fee881d915e7e5ecca6dc4d854dcc08bf44afd049a3ab9ee45fb144f5591d"
Sep 17 22:15:03.710610 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:03.710588    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-35b181-29828055-v5dl5"
Sep 17 22:15:03.710713 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:03.710590    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-35b181-29828055-v5dl5" event={"ID":"93756527-cf7c-4fb6-a6d0-4b8cad8fc89a","Type":"ContainerDied","Data":"ff978719d97bd991d01b82033386adde318fc48a8e36f243db7976636e84af43"}
Sep 17 22:15:03.710713 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:03.710699    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff978719d97bd991d01b82033386adde318fc48a8e36f243db7976636e84af43"
Sep 17 22:15:11.324509 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:11.324470    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"]
Sep 17 22:15:13.145820 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.145784    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"]
Sep 17 22:15:13.146435 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.146350    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e987c5ac-a0f3-4fbc-b4e3-a7eb1ea01899" containerName="cleanup"
Sep 17 22:15:13.146435 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.146365    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e987c5ac-a0f3-4fbc-b4e3-a7eb1ea01899" containerName="cleanup"
Sep 17 22:15:13.146435 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.146373    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93756527-cf7c-4fb6-a6d0-4b8cad8fc89a" containerName="cleanup"
Sep 17 22:15:13.146435 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.146378    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="93756527-cf7c-4fb6-a6d0-4b8cad8fc89a" containerName="cleanup"
Sep 17 22:15:13.146654 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.146446    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="93756527-cf7c-4fb6-a6d0-4b8cad8fc89a" containerName="cleanup"
Sep 17 22:15:13.146654 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.146457    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e987c5ac-a0f3-4fbc-b4e3-a7eb1ea01899" containerName="cleanup"
Sep 17 22:15:13.149719 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.149700    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.153302 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.153280    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a71836-openshift-default-dockercfg-jzh7x\""
Sep 17 22:15:13.165801 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.165778    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"]
Sep 17 22:15:13.279416 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.279384    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istio-token\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.279416 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.279420    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-credential-socket\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.279692 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.279444    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptk4c\" (UniqueName: \"kubernetes.io/projected/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-kube-api-access-ptk4c\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.279692 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.279518    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istio-envoy\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.279692 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.279564    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-workload-socket\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.279692 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.279605    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istio-podinfo\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.279692 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.279678    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-workload-certs\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.279885 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.279705    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istiod-ca-cert\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.279885 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.279766    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istio-data\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.380706 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.380664    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istio-token\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.380893 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.380717    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-credential-socket\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.380893 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.380753    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptk4c\" (UniqueName: \"kubernetes.io/projected/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-kube-api-access-ptk4c\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.380893 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.380790    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istio-envoy\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.380893 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.380817    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-workload-socket\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.380893 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.380855    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istio-podinfo\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.381201 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.380895    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-workload-certs\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.381201 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.380922    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istiod-ca-cert\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.381201 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.380958    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istio-data\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.381348 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.381259    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-credential-socket\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.381348 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.381301    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-workload-socket\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.381433 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.381411    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-workload-certs\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.381494 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.381431    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istio-data\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.381635 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.381617    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istiod-ca-cert\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.383307 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.383286    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istio-envoy\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.383589 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.383570    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istio-podinfo\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.394922 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.394892    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptk4c\" (UniqueName: \"kubernetes.io/projected/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-kube-api-access-ptk4c\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.396096 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.396046    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istio-token\") pod \"e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") " pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.459673 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.459641    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:13.744379 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.744354    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"]
Sep 17 22:15:13.746057 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:15:13.746032    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d2101c_fae4_4ddc_bbb2_0dd4543905f9.slice/crio-5bf9e8ca458b834a945b82d9971e8a1d0d9034443e0300ec3cddfa4fa4eb7ee1 WatchSource:0}: Error finding container 5bf9e8ca458b834a945b82d9971e8a1d0d9034443e0300ec3cddfa4fa4eb7ee1: Status 404 returned error can't find the container with id 5bf9e8ca458b834a945b82d9971e8a1d0d9034443e0300ec3cddfa4fa4eb7ee1
Sep 17 22:15:13.748262 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.748226    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 22:15:13.748370 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.748325    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 22:15:13.748431 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.748377    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 22:15:13.767601 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:13.767577    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd" event={"ID":"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9","Type":"ContainerStarted","Data":"5bf9e8ca458b834a945b82d9971e8a1d0d9034443e0300ec3cddfa4fa4eb7ee1"}
Sep 17 22:15:14.773666 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:14.773633    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd" event={"ID":"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9","Type":"ContainerStarted","Data":"742ebde603f4f05e09d34816b833eca1e7ce237a45bbce2b306ab3c5baa5d481"}
Sep 17 22:15:14.825762 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:14.825710    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd" podStartSLOduration=1.825695831 podStartE2EDuration="1.825695831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:15:14.825080673 +0000 UTC m=+2030.643406984" watchObservedRunningTime="2026-09-17 22:15:14.825695831 +0000 UTC m=+2030.644022142"
Sep 17 22:15:15.459998 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:15.459961    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:15.465067 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:15.465044    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:15.778862 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:15.778771    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:15.779795 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:15.779779    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:25.210875 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:25.210836    2564 scope.go:117] "RemoveContainer" containerID="a510bfaac3948dd7106b0065eb2c26d11cac361bc1f4d58bf4ed1e28f8bbfe97"
Sep 17 22:15:25.219396 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:25.219376    2564 scope.go:117] "RemoveContainer" containerID="b665c3afa6be94fca578925f83b1ce80c22c098d32f46eb930d0df3e1b765c2e"
Sep 17 22:15:25.226691 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:25.226673    2564 scope.go:117] "RemoveContainer" containerID="f9ae3ad5040f59367c7cb4a74bcf75e9886ef617f4b116ac2f5c7417a5c7a944"
Sep 17 22:15:26.345023 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.344954    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh" podUID="35ee99a4-6c69-4429-92c2-9d793f198d02" containerName="main" containerID="cri-o://059e73caf24e4cb50660147db902e0681b2bbcb696ff78f0fdd1cb43ac7b21e1" gracePeriod=45
Sep 17 22:15:26.638483 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.638460    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:15:26.713000 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.712488    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35ee99a4-6c69-4429-92c2-9d793f198d02-tmp-dir\") pod \"35ee99a4-6c69-4429-92c2-9d793f198d02\" (UID: \"35ee99a4-6c69-4429-92c2-9d793f198d02\") "
Sep 17 22:15:26.713000 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.712551    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35ee99a4-6c69-4429-92c2-9d793f198d02-tls-certs\") pod \"35ee99a4-6c69-4429-92c2-9d793f198d02\" (UID: \"35ee99a4-6c69-4429-92c2-9d793f198d02\") "
Sep 17 22:15:26.713000 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.712624    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbv4r\" (UniqueName: \"kubernetes.io/projected/35ee99a4-6c69-4429-92c2-9d793f198d02-kube-api-access-jbv4r\") pod \"35ee99a4-6c69-4429-92c2-9d793f198d02\" (UID: \"35ee99a4-6c69-4429-92c2-9d793f198d02\") "
Sep 17 22:15:26.713000 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.712665    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35ee99a4-6c69-4429-92c2-9d793f198d02-dshm\") pod \"35ee99a4-6c69-4429-92c2-9d793f198d02\" (UID: \"35ee99a4-6c69-4429-92c2-9d793f198d02\") "
Sep 17 22:15:26.713000 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.712746    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35ee99a4-6c69-4429-92c2-9d793f198d02-home\") pod \"35ee99a4-6c69-4429-92c2-9d793f198d02\" (UID: \"35ee99a4-6c69-4429-92c2-9d793f198d02\") "
Sep 17 22:15:26.713000 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.712782    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ee99a4-6c69-4429-92c2-9d793f198d02-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "35ee99a4-6c69-4429-92c2-9d793f198d02" (UID: "35ee99a4-6c69-4429-92c2-9d793f198d02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:15:26.713000 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.712803    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35ee99a4-6c69-4429-92c2-9d793f198d02-model-cache\") pod \"35ee99a4-6c69-4429-92c2-9d793f198d02\" (UID: \"35ee99a4-6c69-4429-92c2-9d793f198d02\") "
Sep 17 22:15:26.713550 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.713127    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35ee99a4-6c69-4429-92c2-9d793f198d02-tmp-dir\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:15:26.713732 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.713694    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ee99a4-6c69-4429-92c2-9d793f198d02-home" (OuterVolumeSpecName: "home") pod "35ee99a4-6c69-4429-92c2-9d793f198d02" (UID: "35ee99a4-6c69-4429-92c2-9d793f198d02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:15:26.714026 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.713926    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ee99a4-6c69-4429-92c2-9d793f198d02-model-cache" (OuterVolumeSpecName: "model-cache") pod "35ee99a4-6c69-4429-92c2-9d793f198d02" (UID: "35ee99a4-6c69-4429-92c2-9d793f198d02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:15:26.716083 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.716053    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ee99a4-6c69-4429-92c2-9d793f198d02-dshm" (OuterVolumeSpecName: "dshm") pod "35ee99a4-6c69-4429-92c2-9d793f198d02" (UID: "35ee99a4-6c69-4429-92c2-9d793f198d02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:15:26.716575 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.716544    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35ee99a4-6c69-4429-92c2-9d793f198d02-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35ee99a4-6c69-4429-92c2-9d793f198d02" (UID: "35ee99a4-6c69-4429-92c2-9d793f198d02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:15:26.716575 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.716550    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35ee99a4-6c69-4429-92c2-9d793f198d02-kube-api-access-jbv4r" (OuterVolumeSpecName: "kube-api-access-jbv4r") pod "35ee99a4-6c69-4429-92c2-9d793f198d02" (UID: "35ee99a4-6c69-4429-92c2-9d793f198d02"). InnerVolumeSpecName "kube-api-access-jbv4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:15:26.813907 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.813861    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35ee99a4-6c69-4429-92c2-9d793f198d02-dshm\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:15:26.813907 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.813893    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35ee99a4-6c69-4429-92c2-9d793f198d02-home\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:15:26.813907 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.813902    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35ee99a4-6c69-4429-92c2-9d793f198d02-model-cache\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:15:26.813907 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.813913    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35ee99a4-6c69-4429-92c2-9d793f198d02-tls-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:15:26.813907 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.813920    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbv4r\" (UniqueName: \"kubernetes.io/projected/35ee99a4-6c69-4429-92c2-9d793f198d02-kube-api-access-jbv4r\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:15:26.842546 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.842517    2564 generic.go:356] "Generic (PLEG): container finished" podID="35ee99a4-6c69-4429-92c2-9d793f198d02" containerID="059e73caf24e4cb50660147db902e0681b2bbcb696ff78f0fdd1cb43ac7b21e1" exitCode=0
Sep 17 22:15:26.842737 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.842560    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh" event={"ID":"35ee99a4-6c69-4429-92c2-9d793f198d02","Type":"ContainerDied","Data":"059e73caf24e4cb50660147db902e0681b2bbcb696ff78f0fdd1cb43ac7b21e1"}
Sep 17 22:15:26.842737 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.842571    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"
Sep 17 22:15:26.842737 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.842586    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh" event={"ID":"35ee99a4-6c69-4429-92c2-9d793f198d02","Type":"ContainerDied","Data":"1199cd9475fbed73d616a8e1523668f98cc2167a21a3d95570bb26e65474b291"}
Sep 17 22:15:26.842737 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.842610    2564 scope.go:117] "RemoveContainer" containerID="059e73caf24e4cb50660147db902e0681b2bbcb696ff78f0fdd1cb43ac7b21e1"
Sep 17 22:15:26.851360 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.851343    2564 scope.go:117] "RemoveContainer" containerID="059e73caf24e4cb50660147db902e0681b2bbcb696ff78f0fdd1cb43ac7b21e1"
Sep 17 22:15:26.851638 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:15:26.851620    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"059e73caf24e4cb50660147db902e0681b2bbcb696ff78f0fdd1cb43ac7b21e1\": container with ID starting with 059e73caf24e4cb50660147db902e0681b2bbcb696ff78f0fdd1cb43ac7b21e1 not found: ID does not exist" containerID="059e73caf24e4cb50660147db902e0681b2bbcb696ff78f0fdd1cb43ac7b21e1"
Sep 17 22:15:26.851708 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.851649    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"059e73caf24e4cb50660147db902e0681b2bbcb696ff78f0fdd1cb43ac7b21e1"} err="failed to get container status \"059e73caf24e4cb50660147db902e0681b2bbcb696ff78f0fdd1cb43ac7b21e1\": rpc error: code = NotFound desc = could not find container \"059e73caf24e4cb50660147db902e0681b2bbcb696ff78f0fdd1cb43ac7b21e1\": container with ID starting with 059e73caf24e4cb50660147db902e0681b2bbcb696ff78f0fdd1cb43ac7b21e1 not found: ID does not exist"
Sep 17 22:15:26.860452 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.860427    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"]
Sep 17 22:15:26.864870 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:26.864843    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f8dad/rate-test-model-8f8dad-kserve-5df4b66767-r2tfh"]
Sep 17 22:15:28.753952 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:28.753918    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35ee99a4-6c69-4429-92c2-9d793f198d02" path="/var/lib/kubelet/pods/35ee99a4-6c69-4429-92c2-9d793f198d02/volumes"
Sep 17 22:15:34.888809 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:34.888725    2564 generic.go:356] "Generic (PLEG): container finished" podID="a7d2101c-fae4-4ddc-bbb2-0dd4543905f9" containerID="742ebde603f4f05e09d34816b833eca1e7ce237a45bbce2b306ab3c5baa5d481" exitCode=0
Sep 17 22:15:34.889226 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:34.888800    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd" event={"ID":"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9","Type":"ContainerDied","Data":"742ebde603f4f05e09d34816b833eca1e7ce237a45bbce2b306ab3c5baa5d481"}
Sep 17 22:15:34.889226 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:34.889123    2564 scope.go:117] "RemoveContainer" containerID="742ebde603f4f05e09d34816b833eca1e7ce237a45bbce2b306ab3c5baa5d481"
Sep 17 22:15:34.890322 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:34.890289    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 22:15:34.890422 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:34.890355    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 22:15:34.890422 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:34.890386    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 22:15:35.905513 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:35.904556    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd" event={"ID":"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9","Type":"ContainerStarted","Data":"c69118b5ca47d70cf1a68b68874843e7297a58cdd46073b08ebcd89166d0c4e5"}
Sep 17 22:15:36.460394 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:36.460356    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:36.460574 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:36.460549    2564 patch_prober.go:28] interesting pod/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Sep 17 22:15:36.460618 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:36.460596    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd" podUID="a7d2101c-fae4-4ddc-bbb2-0dd4543905f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Sep 17 22:15:37.460292 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:37.460254    2564 patch_prober.go:28] interesting pod/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Sep 17 22:15:37.460764 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:37.460323    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd" podUID="a7d2101c-fae4-4ddc-bbb2-0dd4543905f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Sep 17 22:15:38.460604 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:38.460569    2564 patch_prober.go:28] interesting pod/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Sep 17 22:15:38.460984 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:38.460642    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd" podUID="a7d2101c-fae4-4ddc-bbb2-0dd4543905f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Sep 17 22:15:39.460501 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:39.460465    2564 patch_prober.go:28] interesting pod/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Sep 17 22:15:39.460690 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:39.460551    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd" podUID="a7d2101c-fae4-4ddc-bbb2-0dd4543905f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Sep 17 22:15:40.460777 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:40.460743    2564 patch_prober.go:28] interesting pod/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Sep 17 22:15:40.461203 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:40.460799    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd" podUID="a7d2101c-fae4-4ddc-bbb2-0dd4543905f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Sep 17 22:15:41.466107 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:41.466074    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:41.466610 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:41.466483    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:41.467242 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:41.467222    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:15:47.960567 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:47.960492    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"]
Sep 17 22:15:47.960990 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:47.960869    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35ee99a4-6c69-4429-92c2-9d793f198d02" containerName="main"
Sep 17 22:15:47.960990 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:47.960879    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="35ee99a4-6c69-4429-92c2-9d793f198d02" containerName="main"
Sep 17 22:15:47.960990 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:47.960947    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="35ee99a4-6c69-4429-92c2-9d793f198d02" containerName="main"
Sep 17 22:15:47.964108 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:47.964089    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:47.967017 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:47.966921    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-813cee-openshift-default-dockercfg-h92zx\""
Sep 17 22:15:47.977640 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:47.977620    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"]
Sep 17 22:15:48.007388 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.007347    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78p5b\" (UniqueName: \"kubernetes.io/projected/746564fc-cdcf-47e9-8f05-d6530fb5769e-kube-api-access-78p5b\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.007388 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.007391    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-workload-certs\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.007626 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.007425    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-istio-data\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.007626 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.007458    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-credential-socket\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.007626 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.007483    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-istio-envoy\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.007626 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.007530    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746564fc-cdcf-47e9-8f05-d6530fb5769e-istiod-ca-cert\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.007626 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.007610    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746564fc-cdcf-47e9-8f05-d6530fb5769e-istio-podinfo\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.007909 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.007654    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746564fc-cdcf-47e9-8f05-d6530fb5769e-istio-token\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.007909 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.007674    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-workload-socket\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.108932 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.108895    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746564fc-cdcf-47e9-8f05-d6530fb5769e-istiod-ca-cert\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.109131 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.108945    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746564fc-cdcf-47e9-8f05-d6530fb5769e-istio-podinfo\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.109131 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.108986    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746564fc-cdcf-47e9-8f05-d6530fb5769e-istio-token\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.109131 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.109002    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-workload-socket\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.109131 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.109031    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78p5b\" (UniqueName: \"kubernetes.io/projected/746564fc-cdcf-47e9-8f05-d6530fb5769e-kube-api-access-78p5b\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.109131 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.109051    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-workload-certs\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.109131 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.109083    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-istio-data\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.109467 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.109248    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-credential-socket\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.109467 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.109300    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-istio-envoy\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.109773 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.109743    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-workload-certs\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.109773 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.109765    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-workload-socket\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.109906 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.109752    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746564fc-cdcf-47e9-8f05-d6530fb5769e-istiod-ca-cert\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.109906 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.109857    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-istio-data\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.110000 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.109982    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-credential-socket\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.111945 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.111925    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-istio-envoy\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.112252 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.112229    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746564fc-cdcf-47e9-8f05-d6530fb5769e-istio-podinfo\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.124470 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.124446    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78p5b\" (UniqueName: \"kubernetes.io/projected/746564fc-cdcf-47e9-8f05-d6530fb5769e-kube-api-access-78p5b\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.125513 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.125484    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746564fc-cdcf-47e9-8f05-d6530fb5769e-istio-token\") pod \"e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") " pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.276877 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.276773    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:48.558405 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.558375    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"]
Sep 17 22:15:48.560275 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:15:48.560235    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746564fc_cdcf_47e9_8f05_d6530fb5769e.slice/crio-a59bd45070ac764fe69e088431eef4b5884504d776cf7d29f80472cb40fdbfdd WatchSource:0}: Error finding container a59bd45070ac764fe69e088431eef4b5884504d776cf7d29f80472cb40fdbfdd: Status 404 returned error can't find the container with id a59bd45070ac764fe69e088431eef4b5884504d776cf7d29f80472cb40fdbfdd
Sep 17 22:15:48.562388 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.562353    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 22:15:48.562462 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.562434    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 22:15:48.562500 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.562461    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 22:15:48.979049 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.979010    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn" event={"ID":"746564fc-cdcf-47e9-8f05-d6530fb5769e","Type":"ContainerStarted","Data":"4a840a182776ab6f3e0a80b1df90f0bb5c13d2e230b9d15d18477d71d19aad54"}
Sep 17 22:15:48.979561 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:48.979070    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn" event={"ID":"746564fc-cdcf-47e9-8f05-d6530fb5769e","Type":"ContainerStarted","Data":"a59bd45070ac764fe69e088431eef4b5884504d776cf7d29f80472cb40fdbfdd"}
Sep 17 22:15:49.004431 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:49.004372    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn" podStartSLOduration=2.004351405 podStartE2EDuration="2.004351405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:15:49.001336434 +0000 UTC m=+2064.819662746" watchObservedRunningTime="2026-09-17 22:15:49.004351405 +0000 UTC m=+2064.822677719"
Sep 17 22:15:49.277710 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:49.277627    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:49.282628 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:49.282604    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:49.984740 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:49.984710    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:49.985795 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:49.985770    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:15:54.824126 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:54.824094    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l"]
Sep 17 22:15:54.827553 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:54.827538    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l"
Sep 17 22:15:54.831446 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:54.831426    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-813cee\""
Sep 17 22:15:54.834301 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:54.834269    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-813cee-dockercfg-v8xcf\""
Sep 17 22:15:54.845898 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:54.845872    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l"]
Sep 17 22:15:54.982617 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:54.982586    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55f475f-04aa-4317-bddc-6b7f0b887ca4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l\" (UID: \"a55f475f-04aa-4317-bddc-6b7f0b887ca4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l"
Sep 17 22:15:54.982617 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:54.982618    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz688\" (UniqueName: \"kubernetes.io/projected/a55f475f-04aa-4317-bddc-6b7f0b887ca4-kube-api-access-nz688\") pod \"payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l\" (UID: \"a55f475f-04aa-4317-bddc-6b7f0b887ca4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l"
Sep 17 22:15:55.083873 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:55.083785    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55f475f-04aa-4317-bddc-6b7f0b887ca4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l\" (UID: \"a55f475f-04aa-4317-bddc-6b7f0b887ca4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l"
Sep 17 22:15:55.083873 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:55.083822    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz688\" (UniqueName: \"kubernetes.io/projected/a55f475f-04aa-4317-bddc-6b7f0b887ca4-kube-api-access-nz688\") pod \"payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l\" (UID: \"a55f475f-04aa-4317-bddc-6b7f0b887ca4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l"
Sep 17 22:15:55.084501 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:55.084479    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55f475f-04aa-4317-bddc-6b7f0b887ca4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l\" (UID: \"a55f475f-04aa-4317-bddc-6b7f0b887ca4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l"
Sep 17 22:15:55.097016 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:55.096991    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz688\" (UniqueName: \"kubernetes.io/projected/a55f475f-04aa-4317-bddc-6b7f0b887ca4-kube-api-access-nz688\") pod \"payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l\" (UID: \"a55f475f-04aa-4317-bddc-6b7f0b887ca4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l"
Sep 17 22:15:55.137597 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:55.137573    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l"
Sep 17 22:15:55.439962 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:55.439935    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l"]
Sep 17 22:15:55.441226 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:15:55.441166    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda55f475f_04aa_4317_bddc_6b7f0b887ca4.slice/crio-ed058c050d367527d85a6b8f110c712266a54b7768255700532ce5eae6d53cd2 WatchSource:0}: Error finding container ed058c050d367527d85a6b8f110c712266a54b7768255700532ce5eae6d53cd2: Status 404 returned error can't find the container with id ed058c050d367527d85a6b8f110c712266a54b7768255700532ce5eae6d53cd2
Sep 17 22:15:56.021223 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:56.021158    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l" event={"ID":"a55f475f-04aa-4317-bddc-6b7f0b887ca4","Type":"ContainerStarted","Data":"00501276aa6b2dec41eabd5393b3702e6682963528e964aad780c300aaf527ca"}
Sep 17 22:15:56.021643 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:56.021236    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l" event={"ID":"a55f475f-04aa-4317-bddc-6b7f0b887ca4","Type":"ContainerStarted","Data":"ed058c050d367527d85a6b8f110c712266a54b7768255700532ce5eae6d53cd2"}
Sep 17 22:15:56.021643 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:56.021348    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l"
Sep 17 22:15:56.046560 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:15:56.046501    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l" podStartSLOduration=2.046484367 podStartE2EDuration="2.046484367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:15:56.046111459 +0000 UTC m=+2071.864437773" watchObservedRunningTime="2026-09-17 22:15:56.046484367 +0000 UTC m=+2071.864810678"
Sep 17 22:16:07.028888 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:07.028856    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l"
Sep 17 22:16:09.092385 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:09.092347    2564 generic.go:356] "Generic (PLEG): container finished" podID="746564fc-cdcf-47e9-8f05-d6530fb5769e" containerID="4a840a182776ab6f3e0a80b1df90f0bb5c13d2e230b9d15d18477d71d19aad54" exitCode=0
Sep 17 22:16:09.092906 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:09.092409    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn" event={"ID":"746564fc-cdcf-47e9-8f05-d6530fb5769e","Type":"ContainerDied","Data":"4a840a182776ab6f3e0a80b1df90f0bb5c13d2e230b9d15d18477d71d19aad54"}
Sep 17 22:16:09.092979 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:09.092910    2564 scope.go:117] "RemoveContainer" containerID="4a840a182776ab6f3e0a80b1df90f0bb5c13d2e230b9d15d18477d71d19aad54"
Sep 17 22:16:09.094229 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:09.094197    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 22:16:09.094298 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:09.094275    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 22:16:09.094339 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:09.094309    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 22:16:10.103492 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:10.103444    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn" event={"ID":"746564fc-cdcf-47e9-8f05-d6530fb5769e","Type":"ContainerStarted","Data":"baf1043eabf66a0eac6e6f89e83c994c678dc25fb91d09c59221a79d42a00741"}
Sep 17 22:16:10.277911 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:10.277877    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:16:10.278089 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:10.278028    2564 patch_prober.go:28] interesting pod/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Sep 17 22:16:10.278089 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:10.278076    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn" podUID="746564fc-cdcf-47e9-8f05-d6530fb5769e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Sep 17 22:16:11.277979 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:11.277942    2564 patch_prober.go:28] interesting pod/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Sep 17 22:16:11.278393 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:11.278001    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn" podUID="746564fc-cdcf-47e9-8f05-d6530fb5769e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Sep 17 22:16:12.277259 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:12.277223    2564 patch_prober.go:28] interesting pod/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Sep 17 22:16:12.277460 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:12.277294    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn" podUID="746564fc-cdcf-47e9-8f05-d6530fb5769e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Sep 17 22:16:13.277548 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:13.277503    2564 patch_prober.go:28] interesting pod/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Sep 17 22:16:13.278020 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:13.277581    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn" podUID="746564fc-cdcf-47e9-8f05-d6530fb5769e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Sep 17 22:16:14.277308 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:14.277270    2564 patch_prober.go:28] interesting pod/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Sep 17 22:16:14.277481 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:14.277330    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn" podUID="746564fc-cdcf-47e9-8f05-d6530fb5769e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Sep 17 22:16:15.277568 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:15.277533    2564 patch_prober.go:28] interesting pod/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Sep 17 22:16:15.277952 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:15.277598    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn" podUID="746564fc-cdcf-47e9-8f05-d6530fb5769e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Sep 17 22:16:16.281272 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:16.281245    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:16:16.281712 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:16.281691    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:16:16.282261 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:16.282242    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:16:24.824379 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:24.824350    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdcnw_6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2/ovn-acl-logging/0.log"
Sep 17 22:16:24.825600 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:24.825581    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdcnw_6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2/ovn-acl-logging/0.log"
Sep 17 22:16:25.255670 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:25.255636    2564 scope.go:117] "RemoveContainer" containerID="37f1286042fd98b785cc57f15e31d1023bbd9b42fbdb6cb0fa826af6d84eb0a2"
Sep 17 22:16:25.265229 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:25.265207    2564 scope.go:117] "RemoveContainer" containerID="478f2443fcaf87194804fd06a28b0adb9501fd02bf8e7aeff1de44dbb3bad459"
Sep 17 22:16:32.103318 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.103283    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"]
Sep 17 22:16:32.106685 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.106668    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:16:32.110443 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.110421    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a71836\"/\"openshift-service-ca.crt\""
Sep 17 22:16:32.110707 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.110682    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a71836\"/\"kube-root-ca.crt\""
Sep 17 22:16:32.111672 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.111654    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a71836\"/\"ipp-route-a71836-kserve-self-signed-certs\""
Sep 17 22:16:32.111782 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.111747    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a71836\"/\"default-dockercfg-d8pbq\""
Sep 17 22:16:32.127953 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.127928    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"]
Sep 17 22:16:32.222684 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.222650    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-tmp-dir\") pod \"ipp-route-a71836-kserve-566c58fd4c-hd2bx\" (UID: \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\") " pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:16:32.222684 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.222690    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-dshm\") pod \"ipp-route-a71836-kserve-566c58fd4c-hd2bx\" (UID: \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\") " pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:16:32.222971 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.222716    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-tls-certs\") pod \"ipp-route-a71836-kserve-566c58fd4c-hd2bx\" (UID: \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\") " pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:16:32.222971 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.222773    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4ttf\" (UniqueName: \"kubernetes.io/projected/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-kube-api-access-x4ttf\") pod \"ipp-route-a71836-kserve-566c58fd4c-hd2bx\" (UID: \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\") " pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:16:32.222971 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.222814    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-model-cache\") pod \"ipp-route-a71836-kserve-566c58fd4c-hd2bx\" (UID: \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\") " pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:16:32.222971 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.222891    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-home\") pod \"ipp-route-a71836-kserve-566c58fd4c-hd2bx\" (UID: \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\") " pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:16:32.323810 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.323771    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-model-cache\") pod \"ipp-route-a71836-kserve-566c58fd4c-hd2bx\" (UID: \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\") " pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:16:32.324009 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.323832    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-home\") pod \"ipp-route-a71836-kserve-566c58fd4c-hd2bx\" (UID: \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\") " pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:16:32.324009 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.323873    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-tmp-dir\") pod \"ipp-route-a71836-kserve-566c58fd4c-hd2bx\" (UID: \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\") " pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:16:32.324009 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.323892    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-dshm\") pod \"ipp-route-a71836-kserve-566c58fd4c-hd2bx\" (UID: \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\") " pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:16:32.324009 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.323917    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-tls-certs\") pod \"ipp-route-a71836-kserve-566c58fd4c-hd2bx\" (UID: \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\") " pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:16:32.324009 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.323939    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4ttf\" (UniqueName: \"kubernetes.io/projected/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-kube-api-access-x4ttf\") pod \"ipp-route-a71836-kserve-566c58fd4c-hd2bx\" (UID: \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\") " pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:16:32.324324 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.324288    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-model-cache\") pod \"ipp-route-a71836-kserve-566c58fd4c-hd2bx\" (UID: \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\") " pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:16:32.324324 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.324301    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-tmp-dir\") pod \"ipp-route-a71836-kserve-566c58fd4c-hd2bx\" (UID: \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\") " pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:16:32.324440 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.324383    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-home\") pod \"ipp-route-a71836-kserve-566c58fd4c-hd2bx\" (UID: \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\") " pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:16:32.326347 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.326329    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-dshm\") pod \"ipp-route-a71836-kserve-566c58fd4c-hd2bx\" (UID: \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\") " pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:16:32.326623 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.326603    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-tls-certs\") pod \"ipp-route-a71836-kserve-566c58fd4c-hd2bx\" (UID: \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\") " pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:16:32.351708 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.351674    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4ttf\" (UniqueName: \"kubernetes.io/projected/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-kube-api-access-x4ttf\") pod \"ipp-route-a71836-kserve-566c58fd4c-hd2bx\" (UID: \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\") " pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:16:32.415379 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.415347    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:16:32.701468 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:32.701439    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"]
Sep 17 22:16:32.702909 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:16:32.702883    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb7e9c4d_eef2_4cd2_8b8e_61c21d13b8b3.slice/crio-3577f4fa8b691330f3a90cff72a78ac94402138f71993e5c12f651b47afe3057 WatchSource:0}: Error finding container 3577f4fa8b691330f3a90cff72a78ac94402138f71993e5c12f651b47afe3057: Status 404 returned error can't find the container with id 3577f4fa8b691330f3a90cff72a78ac94402138f71993e5c12f651b47afe3057
Sep 17 22:16:33.228980 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:33.228914    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx" event={"ID":"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3","Type":"ContainerStarted","Data":"7de929906aa1b106c647b2e9bf25bdb6a81226125940fb5ca9b20ab9aa2cbd52"}
Sep 17 22:16:33.228980 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:33.228955    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx" event={"ID":"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3","Type":"ContainerStarted","Data":"3577f4fa8b691330f3a90cff72a78ac94402138f71993e5c12f651b47afe3057"}
Sep 17 22:16:33.252886 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:33.252830    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx" podStartSLOduration=1.252816359 podStartE2EDuration="1.252816359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:16:33.24992043 +0000 UTC m=+2109.068246742" watchObservedRunningTime="2026-09-17 22:16:33.252816359 +0000 UTC m=+2109.071142671"
Sep 17 22:16:42.416318 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:42.416279    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:16:42.416793 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:42.416366    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:16:42.429301 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:42.429278    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:16:43.292058 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:16:43.292032    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:17:12.774352 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:12.774316    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr"]
Sep 17 22:17:12.777872 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:12.777855    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr"
Sep 17 22:17:12.782979 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:12.782958    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-2e0396\""
Sep 17 22:17:12.783212 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:12.783195    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-2e0396-dockercfg-l69b9\""
Sep 17 22:17:12.788665 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:12.788638    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr"]
Sep 17 22:17:12.894062 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:12.894022    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff8gn\" (UniqueName: \"kubernetes.io/projected/93ff7869-e699-4fa1-bc08-bfe7e33ac6f9-kube-api-access-ff8gn\") pod \"payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr\" (UID: \"93ff7869-e699-4fa1-bc08-bfe7e33ac6f9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr"
Sep 17 22:17:12.894493 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:12.894465    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ff7869-e699-4fa1-bc08-bfe7e33ac6f9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr\" (UID: \"93ff7869-e699-4fa1-bc08-bfe7e33ac6f9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr"
Sep 17 22:17:12.995385 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:12.995349    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ff7869-e699-4fa1-bc08-bfe7e33ac6f9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr\" (UID: \"93ff7869-e699-4fa1-bc08-bfe7e33ac6f9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr"
Sep 17 22:17:12.995569 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:12.995409    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff8gn\" (UniqueName: \"kubernetes.io/projected/93ff7869-e699-4fa1-bc08-bfe7e33ac6f9-kube-api-access-ff8gn\") pod \"payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr\" (UID: \"93ff7869-e699-4fa1-bc08-bfe7e33ac6f9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr"
Sep 17 22:17:12.996063 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:12.996042    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ff7869-e699-4fa1-bc08-bfe7e33ac6f9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr\" (UID: \"93ff7869-e699-4fa1-bc08-bfe7e33ac6f9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr"
Sep 17 22:17:13.010341 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:13.010310    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff8gn\" (UniqueName: \"kubernetes.io/projected/93ff7869-e699-4fa1-bc08-bfe7e33ac6f9-kube-api-access-ff8gn\") pod \"payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr\" (UID: \"93ff7869-e699-4fa1-bc08-bfe7e33ac6f9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr"
Sep 17 22:17:13.090620 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:13.090548    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr"
Sep 17 22:17:13.371424 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:13.371345    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr"]
Sep 17 22:17:13.375208 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:17:13.375155    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ff7869_e699_4fa1_bc08_bfe7e33ac6f9.slice/crio-9e2026469d0dc8bcf345bb6768e15f899d5efcfceaf476523593ab840eb32b69 WatchSource:0}: Error finding container 9e2026469d0dc8bcf345bb6768e15f899d5efcfceaf476523593ab840eb32b69: Status 404 returned error can't find the container with id 9e2026469d0dc8bcf345bb6768e15f899d5efcfceaf476523593ab840eb32b69
Sep 17 22:17:13.437564 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:13.437533    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr" event={"ID":"93ff7869-e699-4fa1-bc08-bfe7e33ac6f9","Type":"ContainerStarted","Data":"9e2026469d0dc8bcf345bb6768e15f899d5efcfceaf476523593ab840eb32b69"}
Sep 17 22:17:14.446923 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:14.446889    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr" event={"ID":"93ff7869-e699-4fa1-bc08-bfe7e33ac6f9","Type":"ContainerStarted","Data":"e8cf65a040a0324327ca95506fa5773de4afb2604178334b92ecabbe2447af22"}
Sep 17 22:17:14.447414 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:14.447001    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr"
Sep 17 22:17:14.475463 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:14.475415    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr" podStartSLOduration=2.475399945 podStartE2EDuration="2.475399945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:17:14.471986374 +0000 UTC m=+2150.290312692" watchObservedRunningTime="2026-09-17 22:17:14.475399945 +0000 UTC m=+2150.293726257"
Sep 17 22:17:25.292426 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:25.292400    2564 scope.go:117] "RemoveContainer" containerID="cb347de8a1699672e3264a604e9e982dea6d586d1b48ef33b7903b49e6ed3a79"
Sep 17 22:17:25.455095 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:25.455066    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr"
Sep 17 22:17:38.403500 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:38.403467    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e0396-4kcx7"]
Sep 17 22:17:38.406970 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:38.406952    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e0396-4kcx7"
Sep 17 22:17:38.413390 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:17:38.413359    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-ipp-cleanup-2e0396-4kcx7\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="ac6f138f-263d-43c5-80e5-bf78a4f76b24" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e0396-4kcx7"
Sep 17 22:17:38.413544 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:38.413412    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmdbp\""
Sep 17 22:17:38.413544 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:17:38.413443    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 17 22:17:38.445705 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:38.445682    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e0396-4kcx7"]
Sep 17 22:17:38.448419 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:38.448395    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac6f138f-263d-43c5-80e5-bf78a4f76b24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-2e0396-4kcx7\" (UID: \"ac6f138f-263d-43c5-80e5-bf78a4f76b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e0396-4kcx7"
Sep 17 22:17:38.549637 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:38.549591    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac6f138f-263d-43c5-80e5-bf78a4f76b24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-2e0396-4kcx7\" (UID: \"ac6f138f-263d-43c5-80e5-bf78a4f76b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e0396-4kcx7"
Sep 17 22:17:39.418378 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:39.418344    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 22:17:39.420883 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:39.420863    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac6f138f-263d-43c5-80e5-bf78a4f76b24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-2e0396-4kcx7\" (UID: \"ac6f138f-263d-43c5-80e5-bf78a4f76b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e0396-4kcx7"
Sep 17 22:17:39.616543 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:39.616509    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e0396-4kcx7"
Sep 17 22:17:39.907069 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:39.907040    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e0396-4kcx7"]
Sep 17 22:17:39.910612 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:17:39.910586    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6f138f_263d_43c5_80e5_bf78a4f76b24.slice/crio-4e1b65fe0f2799bad29e6c31f6e4d6f6028914073e55f0aa4f385b025fde142d WatchSource:0}: Error finding container 4e1b65fe0f2799bad29e6c31f6e4d6f6028914073e55f0aa4f385b025fde142d: Status 404 returned error can't find the container with id 4e1b65fe0f2799bad29e6c31f6e4d6f6028914073e55f0aa4f385b025fde142d
Sep 17 22:17:40.582846 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:40.582809    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e0396-4kcx7" event={"ID":"ac6f138f-263d-43c5-80e5-bf78a4f76b24","Type":"ContainerStarted","Data":"c6937f72c81aacd31522126393ec5792bef5157247b80f65be0455b56f79fb30"}
Sep 17 22:17:40.583252 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:40.582866    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e0396-4kcx7" event={"ID":"ac6f138f-263d-43c5-80e5-bf78a4f76b24","Type":"ContainerStarted","Data":"4e1b65fe0f2799bad29e6c31f6e4d6f6028914073e55f0aa4f385b025fde142d"}
Sep 17 22:17:40.608319 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:40.608272    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e0396-4kcx7" podStartSLOduration=2.608258354 podStartE2EDuration="2.608258354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:17:40.605214589 +0000 UTC m=+2176.423540898" watchObservedRunningTime="2026-09-17 22:17:40.608258354 +0000 UTC m=+2176.426584665"
Sep 17 22:17:41.589952 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:41.589919    2564 generic.go:356] "Generic (PLEG): container finished" podID="ac6f138f-263d-43c5-80e5-bf78a4f76b24" containerID="c6937f72c81aacd31522126393ec5792bef5157247b80f65be0455b56f79fb30" exitCode=0
Sep 17 22:17:41.590628 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:41.590099    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e0396-4kcx7" event={"ID":"ac6f138f-263d-43c5-80e5-bf78a4f76b24","Type":"ContainerDied","Data":"c6937f72c81aacd31522126393ec5792bef5157247b80f65be0455b56f79fb30"}
Sep 17 22:17:42.735267 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:42.735243    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e0396-4kcx7"
Sep 17 22:17:42.789697 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:42.789669    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac6f138f-263d-43c5-80e5-bf78a4f76b24-maas-api-service-ca\") pod \"ac6f138f-263d-43c5-80e5-bf78a4f76b24\" (UID: \"ac6f138f-263d-43c5-80e5-bf78a4f76b24\") "
Sep 17 22:17:42.790059 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:42.790028    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6f138f-263d-43c5-80e5-bf78a4f76b24-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ac6f138f-263d-43c5-80e5-bf78a4f76b24" (UID: "ac6f138f-263d-43c5-80e5-bf78a4f76b24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:17:42.891409 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:42.891322    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac6f138f-263d-43c5-80e5-bf78a4f76b24-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:17:43.602530 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:43.602492    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e0396-4kcx7" event={"ID":"ac6f138f-263d-43c5-80e5-bf78a4f76b24","Type":"ContainerDied","Data":"4e1b65fe0f2799bad29e6c31f6e4d6f6028914073e55f0aa4f385b025fde142d"}
Sep 17 22:17:43.602708 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:43.602541    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e1b65fe0f2799bad29e6c31f6e4d6f6028914073e55f0aa4f385b025fde142d"
Sep 17 22:17:43.602708 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:43.602503    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e0396-4kcx7"
Sep 17 22:17:49.792858 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:49.792824    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr"]
Sep 17 22:17:49.793452 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:49.793133    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr" podUID="93ff7869-e699-4fa1-bc08-bfe7e33ac6f9" containerName="payload-processing" containerID="cri-o://e8cf65a040a0324327ca95506fa5773de4afb2604178334b92ecabbe2447af22" gracePeriod=30
Sep 17 22:17:50.025255 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:50.025225    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e0396-4kcx7"]
Sep 17 22:17:50.028137 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:50.028111    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e0396-4kcx7"]
Sep 17 22:17:50.078669 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:50.078648    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr"
Sep 17 22:17:50.157127 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:50.157094    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ff8gn\" (UniqueName: \"kubernetes.io/projected/93ff7869-e699-4fa1-bc08-bfe7e33ac6f9-kube-api-access-ff8gn\") pod \"93ff7869-e699-4fa1-bc08-bfe7e33ac6f9\" (UID: \"93ff7869-e699-4fa1-bc08-bfe7e33ac6f9\") "
Sep 17 22:17:50.157325 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:50.157245    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ff7869-e699-4fa1-bc08-bfe7e33ac6f9-plugins-config-volume\") pod \"93ff7869-e699-4fa1-bc08-bfe7e33ac6f9\" (UID: \"93ff7869-e699-4fa1-bc08-bfe7e33ac6f9\") "
Sep 17 22:17:50.157584 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:50.157562    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93ff7869-e699-4fa1-bc08-bfe7e33ac6f9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "93ff7869-e699-4fa1-bc08-bfe7e33ac6f9" (UID: "93ff7869-e699-4fa1-bc08-bfe7e33ac6f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:17:50.159713 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:50.159694    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ff7869-e699-4fa1-bc08-bfe7e33ac6f9-kube-api-access-ff8gn" (OuterVolumeSpecName: "kube-api-access-ff8gn") pod "93ff7869-e699-4fa1-bc08-bfe7e33ac6f9" (UID: "93ff7869-e699-4fa1-bc08-bfe7e33ac6f9"). InnerVolumeSpecName "kube-api-access-ff8gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:17:50.258333 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:50.258297    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff8gn\" (UniqueName: \"kubernetes.io/projected/93ff7869-e699-4fa1-bc08-bfe7e33ac6f9-kube-api-access-ff8gn\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:17:50.258333 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:50.258327    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ff7869-e699-4fa1-bc08-bfe7e33ac6f9-plugins-config-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:17:50.640116 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:50.640077    2564 generic.go:356] "Generic (PLEG): container finished" podID="93ff7869-e699-4fa1-bc08-bfe7e33ac6f9" containerID="e8cf65a040a0324327ca95506fa5773de4afb2604178334b92ecabbe2447af22" exitCode=0
Sep 17 22:17:50.640326 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:50.640123    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr"
Sep 17 22:17:50.640326 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:50.640162    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr" event={"ID":"93ff7869-e699-4fa1-bc08-bfe7e33ac6f9","Type":"ContainerDied","Data":"e8cf65a040a0324327ca95506fa5773de4afb2604178334b92ecabbe2447af22"}
Sep 17 22:17:50.640326 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:50.640237    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr" event={"ID":"93ff7869-e699-4fa1-bc08-bfe7e33ac6f9","Type":"ContainerDied","Data":"9e2026469d0dc8bcf345bb6768e15f899d5efcfceaf476523593ab840eb32b69"}
Sep 17 22:17:50.640326 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:50.640261    2564 scope.go:117] "RemoveContainer" containerID="e8cf65a040a0324327ca95506fa5773de4afb2604178334b92ecabbe2447af22"
Sep 17 22:17:50.649584 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:50.649565    2564 scope.go:117] "RemoveContainer" containerID="e8cf65a040a0324327ca95506fa5773de4afb2604178334b92ecabbe2447af22"
Sep 17 22:17:50.649865 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:17:50.649829    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8cf65a040a0324327ca95506fa5773de4afb2604178334b92ecabbe2447af22\": container with ID starting with e8cf65a040a0324327ca95506fa5773de4afb2604178334b92ecabbe2447af22 not found: ID does not exist" containerID="e8cf65a040a0324327ca95506fa5773de4afb2604178334b92ecabbe2447af22"
Sep 17 22:17:50.649937 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:50.649871    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8cf65a040a0324327ca95506fa5773de4afb2604178334b92ecabbe2447af22"} err="failed to get container status \"e8cf65a040a0324327ca95506fa5773de4afb2604178334b92ecabbe2447af22\": rpc error: code = NotFound desc = could not find container \"e8cf65a040a0324327ca95506fa5773de4afb2604178334b92ecabbe2447af22\": container with ID starting with e8cf65a040a0324327ca95506fa5773de4afb2604178334b92ecabbe2447af22 not found: ID does not exist"
Sep 17 22:17:50.668999 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:50.668965    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr"]
Sep 17 22:17:50.676534 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:50.676506    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2e0396-588c844685-rlxxr"]
Sep 17 22:17:50.753681 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:50.753639    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93ff7869-e699-4fa1-bc08-bfe7e33ac6f9" path="/var/lib/kubelet/pods/93ff7869-e699-4fa1-bc08-bfe7e33ac6f9/volumes"
Sep 17 22:17:50.754079 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:17:50.754057    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac6f138f-263d-43c5-80e5-bf78a4f76b24" path="/var/lib/kubelet/pods/ac6f138f-263d-43c5-80e5-bf78a4f76b24/volumes"
Sep 17 22:18:01.233893 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:01.233858    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a71836-2s469"]
Sep 17 22:18:01.234297 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:01.234267    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93ff7869-e699-4fa1-bc08-bfe7e33ac6f9" containerName="payload-processing"
Sep 17 22:18:01.234297 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:01.234279    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="93ff7869-e699-4fa1-bc08-bfe7e33ac6f9" containerName="payload-processing"
Sep 17 22:18:01.234297 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:01.234296    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac6f138f-263d-43c5-80e5-bf78a4f76b24" containerName="revoke-keys"
Sep 17 22:18:01.234403 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:01.234302    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac6f138f-263d-43c5-80e5-bf78a4f76b24" containerName="revoke-keys"
Sep 17 22:18:01.234403 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:01.234361    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="93ff7869-e699-4fa1-bc08-bfe7e33ac6f9" containerName="payload-processing"
Sep 17 22:18:01.234403 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:01.234368    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac6f138f-263d-43c5-80e5-bf78a4f76b24" containerName="revoke-keys"
Sep 17 22:18:01.237402 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:01.237385    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a71836-2s469"
Sep 17 22:18:01.241508 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:01.241487    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmdbp\""
Sep 17 22:18:01.241817 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:01.241803    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 22:18:01.257105 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:01.257080    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a71836-2s469"]
Sep 17 22:18:01.359027 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:01.358986    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/899f8e01-5fbd-40f3-8083-966b8b9f8970-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a71836-2s469\" (UID: \"899f8e01-5fbd-40f3-8083-966b8b9f8970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a71836-2s469"
Sep 17 22:18:01.459549 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:01.459511    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/899f8e01-5fbd-40f3-8083-966b8b9f8970-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a71836-2s469\" (UID: \"899f8e01-5fbd-40f3-8083-966b8b9f8970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a71836-2s469"
Sep 17 22:18:01.460124 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:01.460101    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/899f8e01-5fbd-40f3-8083-966b8b9f8970-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a71836-2s469\" (UID: \"899f8e01-5fbd-40f3-8083-966b8b9f8970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a71836-2s469"
Sep 17 22:18:01.547276 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:01.547187    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a71836-2s469"
Sep 17 22:18:01.832680 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:01.832586    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a71836-2s469"]
Sep 17 22:18:01.835694 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:18:01.835661    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899f8e01_5fbd_40f3_8083_966b8b9f8970.slice/crio-d37d5b297f9ae5900ce580137474b56c5988ebaeaa5364cff5255a02d35c89bf WatchSource:0}: Error finding container d37d5b297f9ae5900ce580137474b56c5988ebaeaa5364cff5255a02d35c89bf: Status 404 returned error can't find the container with id d37d5b297f9ae5900ce580137474b56c5988ebaeaa5364cff5255a02d35c89bf
Sep 17 22:18:02.711432 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:02.711400    2564 generic.go:356] "Generic (PLEG): container finished" podID="899f8e01-5fbd-40f3-8083-966b8b9f8970" containerID="0e00440ee136cb55bba5914e9aabde48cce0862180a537071ae714e0f51b7341" exitCode=0
Sep 17 22:18:02.711825 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:02.711460    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a71836-2s469" event={"ID":"899f8e01-5fbd-40f3-8083-966b8b9f8970","Type":"ContainerDied","Data":"0e00440ee136cb55bba5914e9aabde48cce0862180a537071ae714e0f51b7341"}
Sep 17 22:18:02.711825 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:02.711496    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a71836-2s469" event={"ID":"899f8e01-5fbd-40f3-8083-966b8b9f8970","Type":"ContainerStarted","Data":"d37d5b297f9ae5900ce580137474b56c5988ebaeaa5364cff5255a02d35c89bf"}
Sep 17 22:18:03.853729 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:03.853706    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a71836-2s469"
Sep 17 22:18:03.986150 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:03.986072    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/899f8e01-5fbd-40f3-8083-966b8b9f8970-maas-api-service-ca\") pod \"899f8e01-5fbd-40f3-8083-966b8b9f8970\" (UID: \"899f8e01-5fbd-40f3-8083-966b8b9f8970\") "
Sep 17 22:18:03.986504 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:03.986481    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/899f8e01-5fbd-40f3-8083-966b8b9f8970-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "899f8e01-5fbd-40f3-8083-966b8b9f8970" (UID: "899f8e01-5fbd-40f3-8083-966b8b9f8970"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:18:04.087149 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:04.087112    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/899f8e01-5fbd-40f3-8083-966b8b9f8970-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:18:04.722941 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:04.722911    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a71836-2s469"
Sep 17 22:18:04.723123 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:04.722910    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a71836-2s469" event={"ID":"899f8e01-5fbd-40f3-8083-966b8b9f8970","Type":"ContainerDied","Data":"d37d5b297f9ae5900ce580137474b56c5988ebaeaa5364cff5255a02d35c89bf"}
Sep 17 22:18:04.723123 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:04.723036    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d37d5b297f9ae5900ce580137474b56c5988ebaeaa5364cff5255a02d35c89bf"
Sep 17 22:18:17.141355 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:17.141308    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a71836-2s469"]
Sep 17 22:18:17.148050 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:17.148023    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a71836-2s469"]
Sep 17 22:18:18.754286 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:18.754252    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="899f8e01-5fbd-40f3-8083-966b8b9f8970" path="/var/lib/kubelet/pods/899f8e01-5fbd-40f3-8083-966b8b9f8970/volumes"
Sep 17 22:18:22.644319 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:22.644285    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"]
Sep 17 22:18:37.663428 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:37.663379    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx" podUID="db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3" containerName="main" containerID="cri-o://7de929906aa1b106c647b2e9bf25bdb6a81226125940fb5ca9b20ab9aa2cbd52" gracePeriod=45
Sep 17 22:18:37.895879 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:37.895847    2564 generic.go:356] "Generic (PLEG): container finished" podID="db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3" containerID="7de929906aa1b106c647b2e9bf25bdb6a81226125940fb5ca9b20ab9aa2cbd52" exitCode=0
Sep 17 22:18:37.896013 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:37.895918    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx" event={"ID":"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3","Type":"ContainerDied","Data":"7de929906aa1b106c647b2e9bf25bdb6a81226125940fb5ca9b20ab9aa2cbd52"}
Sep 17 22:18:37.914586 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:37.914531    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:18:38.002963 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:38.002931    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-tls-certs\") pod \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\" (UID: \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\") "
Sep 17 22:18:38.003149 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:38.002975    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-model-cache\") pod \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\" (UID: \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\") "
Sep 17 22:18:38.003149 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:38.003003    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-tmp-dir\") pod \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\" (UID: \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\") "
Sep 17 22:18:38.003149 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:38.003080    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4ttf\" (UniqueName: \"kubernetes.io/projected/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-kube-api-access-x4ttf\") pod \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\" (UID: \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\") "
Sep 17 22:18:38.003149 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:38.003106    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-dshm\") pod \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\" (UID: \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\") "
Sep 17 22:18:38.003149 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:38.003122    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-home\") pod \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\" (UID: \"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3\") "
Sep 17 22:18:38.003454 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:38.003291    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3" (UID: "db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:18:38.003454 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:38.003299    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3" (UID: "db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:18:38.003454 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:38.003444    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-model-cache\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:18:38.003602 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:38.003459    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-tmp-dir\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:18:38.003602 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:38.003475    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-home" (OuterVolumeSpecName: "home") pod "db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3" (UID: "db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:18:38.005330 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:38.005308    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-dshm" (OuterVolumeSpecName: "dshm") pod "db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3" (UID: "db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:18:38.005423 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:38.005394    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-kube-api-access-x4ttf" (OuterVolumeSpecName: "kube-api-access-x4ttf") pod "db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3" (UID: "db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3"). InnerVolumeSpecName "kube-api-access-x4ttf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:18:38.005489 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:38.005458    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3" (UID: "db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:18:38.103911 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:38.103878    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-tls-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:18:38.103911 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:38.103906    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4ttf\" (UniqueName: \"kubernetes.io/projected/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-kube-api-access-x4ttf\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:18:38.103911 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:38.103915    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-dshm\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:18:38.104142 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:38.103923    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3-home\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:18:38.901944 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:38.901908    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx" event={"ID":"db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3","Type":"ContainerDied","Data":"3577f4fa8b691330f3a90cff72a78ac94402138f71993e5c12f651b47afe3057"}
Sep 17 22:18:38.901944 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:38.901934    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"
Sep 17 22:18:38.902571 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:38.901969    2564 scope.go:117] "RemoveContainer" containerID="7de929906aa1b106c647b2e9bf25bdb6a81226125940fb5ca9b20ab9aa2cbd52"
Sep 17 22:18:38.923365 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:38.923339    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"]
Sep 17 22:18:38.927821 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:38.927801    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-a71836/ipp-route-a71836-kserve-566c58fd4c-hd2bx"]
Sep 17 22:18:40.753822 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:18:40.753785    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3" path="/var/lib/kubelet/pods/db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3/volumes"
Sep 17 22:19:20.339819 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:20.339783    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"]
Sep 17 22:19:20.340337 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:20.340011    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd" podUID="a7d2101c-fae4-4ddc-bbb2-0dd4543905f9" containerName="istio-proxy" containerID="cri-o://c69118b5ca47d70cf1a68b68874843e7297a58cdd46073b08ebcd89166d0c4e5" gracePeriod=30
Sep 17 22:19:20.728701 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:20.728666    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-813cee-fqhch"]
Sep 17 22:19:20.729061 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:20.729049    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3" containerName="main"
Sep 17 22:19:20.729109 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:20.729064    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3" containerName="main"
Sep 17 22:19:20.729109 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:20.729073    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="899f8e01-5fbd-40f3-8083-966b8b9f8970" containerName="revoke-keys"
Sep 17 22:19:20.729109 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:20.729078    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="899f8e01-5fbd-40f3-8083-966b8b9f8970" containerName="revoke-keys"
Sep 17 22:19:20.729228 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:20.729144    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="899f8e01-5fbd-40f3-8083-966b8b9f8970" containerName="revoke-keys"
Sep 17 22:19:20.729228 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:20.729154    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="db7e9c4d-eef2-4cd2-8b8e-61c21d13b8b3" containerName="main"
Sep 17 22:19:20.732270 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:20.732253    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-813cee-fqhch"
Sep 17 22:19:20.735107 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:20.735088    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 22:19:20.735221 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:20.735202    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmdbp\""
Sep 17 22:19:20.739942 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:20.739919    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-813cee-fqhch"]
Sep 17 22:19:20.784383 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:20.784356    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab992b1-c561-45a5-9766-bb7fbd37fe37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-813cee-fqhch\" (UID: \"0ab992b1-c561-45a5-9766-bb7fbd37fe37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-813cee-fqhch"
Sep 17 22:19:20.885820 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:20.885788    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab992b1-c561-45a5-9766-bb7fbd37fe37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-813cee-fqhch\" (UID: \"0ab992b1-c561-45a5-9766-bb7fbd37fe37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-813cee-fqhch"
Sep 17 22:19:20.886400 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:20.886381    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab992b1-c561-45a5-9766-bb7fbd37fe37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-813cee-fqhch\" (UID: \"0ab992b1-c561-45a5-9766-bb7fbd37fe37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-813cee-fqhch"
Sep 17 22:19:21.044203 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:21.044102    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-813cee-fqhch"
Sep 17 22:19:21.314762 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:21.314734    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-813cee-fqhch"]
Sep 17 22:19:21.315996 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:19:21.315974    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab992b1_c561_45a5_9766_bb7fbd37fe37.slice/crio-73a641a26c464bb15b021bfc23fa6ba1743e63e3f8cafd44b39f1f86aab08a60 WatchSource:0}: Error finding container 73a641a26c464bb15b021bfc23fa6ba1743e63e3f8cafd44b39f1f86aab08a60: Status 404 returned error can't find the container with id 73a641a26c464bb15b021bfc23fa6ba1743e63e3f8cafd44b39f1f86aab08a60
Sep 17 22:19:22.126985 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:22.126947    2564 generic.go:356] "Generic (PLEG): container finished" podID="0ab992b1-c561-45a5-9766-bb7fbd37fe37" containerID="1beb34f6cabbb24e2a841c1d29e92225d7aa5adb70e895691ccc0ae05f6bd9ec" exitCode=0
Sep 17 22:19:22.127448 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:22.126992    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-813cee-fqhch" event={"ID":"0ab992b1-c561-45a5-9766-bb7fbd37fe37","Type":"ContainerDied","Data":"1beb34f6cabbb24e2a841c1d29e92225d7aa5adb70e895691ccc0ae05f6bd9ec"}
Sep 17 22:19:22.127448 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:22.127073    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-813cee-fqhch" event={"ID":"0ab992b1-c561-45a5-9766-bb7fbd37fe37","Type":"ContainerStarted","Data":"73a641a26c464bb15b021bfc23fa6ba1743e63e3f8cafd44b39f1f86aab08a60"}
Sep 17 22:19:23.271913 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:23.271889    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-813cee-fqhch"
Sep 17 22:19:23.312333 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:23.312294    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab992b1-c561-45a5-9766-bb7fbd37fe37-maas-api-service-ca\") pod \"0ab992b1-c561-45a5-9766-bb7fbd37fe37\" (UID: \"0ab992b1-c561-45a5-9766-bb7fbd37fe37\") "
Sep 17 22:19:23.312647 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:23.312622    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab992b1-c561-45a5-9766-bb7fbd37fe37-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0ab992b1-c561-45a5-9766-bb7fbd37fe37" (UID: "0ab992b1-c561-45a5-9766-bb7fbd37fe37"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:19:23.413806 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:23.413771    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab992b1-c561-45a5-9766-bb7fbd37fe37-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:19:24.138127 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:24.138091    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-813cee-fqhch" event={"ID":"0ab992b1-c561-45a5-9766-bb7fbd37fe37","Type":"ContainerDied","Data":"73a641a26c464bb15b021bfc23fa6ba1743e63e3f8cafd44b39f1f86aab08a60"}
Sep 17 22:19:24.138363 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:24.138149    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73a641a26c464bb15b021bfc23fa6ba1743e63e3f8cafd44b39f1f86aab08a60"
Sep 17 22:19:24.138363 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:24.138104    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-813cee-fqhch"
Sep 17 22:19:25.586450 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.586425    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:19:25.634433 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.634346    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istiod-ca-cert\") pod \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") "
Sep 17 22:19:25.634433 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.634384    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istio-data\") pod \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") "
Sep 17 22:19:25.634433 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.634404    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istio-token\") pod \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") "
Sep 17 22:19:25.634433 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.634432    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istio-podinfo\") pod \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") "
Sep 17 22:19:25.634757 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.634491    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptk4c\" (UniqueName: \"kubernetes.io/projected/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-kube-api-access-ptk4c\") pod \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") "
Sep 17 22:19:25.634757 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.634522    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-credential-socket\") pod \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") "
Sep 17 22:19:25.634757 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.634565    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-workload-socket\") pod \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") "
Sep 17 22:19:25.634757 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.634629    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-workload-certs\") pod \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") "
Sep 17 22:19:25.634757 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.634666    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istio-envoy\") pod \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\" (UID: \"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9\") "
Sep 17 22:19:25.634757 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.634716    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istio-data" (OuterVolumeSpecName: "istio-data") pod "a7d2101c-fae4-4ddc-bbb2-0dd4543905f9" (UID: "a7d2101c-fae4-4ddc-bbb2-0dd4543905f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:19:25.635087 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.634879    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a7d2101c-fae4-4ddc-bbb2-0dd4543905f9" (UID: "a7d2101c-fae4-4ddc-bbb2-0dd4543905f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:19:25.635087 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.634890    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a7d2101c-fae4-4ddc-bbb2-0dd4543905f9" (UID: "a7d2101c-fae4-4ddc-bbb2-0dd4543905f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:19:25.635087 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.634901    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a7d2101c-fae4-4ddc-bbb2-0dd4543905f9" (UID: "a7d2101c-fae4-4ddc-bbb2-0dd4543905f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:19:25.635087 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.634967    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a7d2101c-fae4-4ddc-bbb2-0dd4543905f9" (UID: "a7d2101c-fae4-4ddc-bbb2-0dd4543905f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:19:25.635351 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.635248    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istiod-ca-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:19:25.635351 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.635265    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istio-data\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:19:25.635351 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.635278    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-credential-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:19:25.635351 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.635291    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-workload-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:19:25.635351 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.635299    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-workload-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:19:25.637574 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.637544    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a7d2101c-fae4-4ddc-bbb2-0dd4543905f9" (UID: "a7d2101c-fae4-4ddc-bbb2-0dd4543905f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:19:25.637574 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.637563    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a7d2101c-fae4-4ddc-bbb2-0dd4543905f9" (UID: "a7d2101c-fae4-4ddc-bbb2-0dd4543905f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 22:19:25.637721 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.637575    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istio-token" (OuterVolumeSpecName: "istio-token") pod "a7d2101c-fae4-4ddc-bbb2-0dd4543905f9" (UID: "a7d2101c-fae4-4ddc-bbb2-0dd4543905f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:19:25.637721 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.637581    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-kube-api-access-ptk4c" (OuterVolumeSpecName: "kube-api-access-ptk4c") pod "a7d2101c-fae4-4ddc-bbb2-0dd4543905f9" (UID: "a7d2101c-fae4-4ddc-bbb2-0dd4543905f9"). InnerVolumeSpecName "kube-api-access-ptk4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:19:25.735910 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.735866    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istio-envoy\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:19:25.735910 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.735900    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istio-token\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:19:25.735910 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.735913    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-istio-podinfo\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:19:25.736156 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:25.735926    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptk4c\" (UniqueName: \"kubernetes.io/projected/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9-kube-api-access-ptk4c\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:19:26.150830 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:26.150791    2564 generic.go:356] "Generic (PLEG): container finished" podID="a7d2101c-fae4-4ddc-bbb2-0dd4543905f9" containerID="c69118b5ca47d70cf1a68b68874843e7297a58cdd46073b08ebcd89166d0c4e5" exitCode=0
Sep 17 22:19:26.151027 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:26.150844    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"
Sep 17 22:19:26.151027 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:26.150841    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd" event={"ID":"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9","Type":"ContainerDied","Data":"c69118b5ca47d70cf1a68b68874843e7297a58cdd46073b08ebcd89166d0c4e5"}
Sep 17 22:19:26.151027 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:26.150907    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd" event={"ID":"a7d2101c-fae4-4ddc-bbb2-0dd4543905f9","Type":"ContainerDied","Data":"5bf9e8ca458b834a945b82d9971e8a1d0d9034443e0300ec3cddfa4fa4eb7ee1"}
Sep 17 22:19:26.151027 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:26.150949    2564 scope.go:117] "RemoveContainer" containerID="c69118b5ca47d70cf1a68b68874843e7297a58cdd46073b08ebcd89166d0c4e5"
Sep 17 22:19:26.160942 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:26.160924    2564 scope.go:117] "RemoveContainer" containerID="742ebde603f4f05e09d34816b833eca1e7ce237a45bbce2b306ab3c5baa5d481"
Sep 17 22:19:26.169394 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:26.169376    2564 scope.go:117] "RemoveContainer" containerID="c69118b5ca47d70cf1a68b68874843e7297a58cdd46073b08ebcd89166d0c4e5"
Sep 17 22:19:26.169672 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:19:26.169652    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c69118b5ca47d70cf1a68b68874843e7297a58cdd46073b08ebcd89166d0c4e5\": container with ID starting with c69118b5ca47d70cf1a68b68874843e7297a58cdd46073b08ebcd89166d0c4e5 not found: ID does not exist" containerID="c69118b5ca47d70cf1a68b68874843e7297a58cdd46073b08ebcd89166d0c4e5"
Sep 17 22:19:26.169751 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:26.169685    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c69118b5ca47d70cf1a68b68874843e7297a58cdd46073b08ebcd89166d0c4e5"} err="failed to get container status \"c69118b5ca47d70cf1a68b68874843e7297a58cdd46073b08ebcd89166d0c4e5\": rpc error: code = NotFound desc = could not find container \"c69118b5ca47d70cf1a68b68874843e7297a58cdd46073b08ebcd89166d0c4e5\": container with ID starting with c69118b5ca47d70cf1a68b68874843e7297a58cdd46073b08ebcd89166d0c4e5 not found: ID does not exist"
Sep 17 22:19:26.169751 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:26.169712    2564 scope.go:117] "RemoveContainer" containerID="742ebde603f4f05e09d34816b833eca1e7ce237a45bbce2b306ab3c5baa5d481"
Sep 17 22:19:26.169959 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:19:26.169942    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"742ebde603f4f05e09d34816b833eca1e7ce237a45bbce2b306ab3c5baa5d481\": container with ID starting with 742ebde603f4f05e09d34816b833eca1e7ce237a45bbce2b306ab3c5baa5d481 not found: ID does not exist" containerID="742ebde603f4f05e09d34816b833eca1e7ce237a45bbce2b306ab3c5baa5d481"
Sep 17 22:19:26.169999 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:26.169966    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742ebde603f4f05e09d34816b833eca1e7ce237a45bbce2b306ab3c5baa5d481"} err="failed to get container status \"742ebde603f4f05e09d34816b833eca1e7ce237a45bbce2b306ab3c5baa5d481\": rpc error: code = NotFound desc = could not find container \"742ebde603f4f05e09d34816b833eca1e7ce237a45bbce2b306ab3c5baa5d481\": container with ID starting with 742ebde603f4f05e09d34816b833eca1e7ce237a45bbce2b306ab3c5baa5d481 not found: ID does not exist"
Sep 17 22:19:27.003395 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:27.003358    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"]
Sep 17 22:19:27.007216 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:27.007165    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a71836-openshift-default-d6cd546f4-982fd"]
Sep 17 22:19:28.753891 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:28.753858    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7d2101c-fae4-4ddc-bbb2-0dd4543905f9" path="/var/lib/kubelet/pods/a7d2101c-fae4-4ddc-bbb2-0dd4543905f9/volumes"
Sep 17 22:19:33.124456 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:33.124424    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l"]
Sep 17 22:19:33.124991 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:33.124646    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l" podUID="a55f475f-04aa-4317-bddc-6b7f0b887ca4" containerName="payload-pre-processing" containerID="cri-o://00501276aa6b2dec41eabd5393b3702e6682963528e964aad780c300aaf527ca" gracePeriod=30
Sep 17 22:19:33.340233 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:33.340199    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-813cee-fqhch"]
Sep 17 22:19:33.342978 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:33.342954    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-813cee-fqhch"]
Sep 17 22:19:33.382772 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:33.382721    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l"
Sep 17 22:19:33.508874 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:33.508841    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55f475f-04aa-4317-bddc-6b7f0b887ca4-plugins-config-volume\") pod \"a55f475f-04aa-4317-bddc-6b7f0b887ca4\" (UID: \"a55f475f-04aa-4317-bddc-6b7f0b887ca4\") "
Sep 17 22:19:33.509141 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:33.508955    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nz688\" (UniqueName: \"kubernetes.io/projected/a55f475f-04aa-4317-bddc-6b7f0b887ca4-kube-api-access-nz688\") pod \"a55f475f-04aa-4317-bddc-6b7f0b887ca4\" (UID: \"a55f475f-04aa-4317-bddc-6b7f0b887ca4\") "
Sep 17 22:19:33.509270 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:33.509247    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a55f475f-04aa-4317-bddc-6b7f0b887ca4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a55f475f-04aa-4317-bddc-6b7f0b887ca4" (UID: "a55f475f-04aa-4317-bddc-6b7f0b887ca4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:19:33.511140 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:33.511116    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a55f475f-04aa-4317-bddc-6b7f0b887ca4-kube-api-access-nz688" (OuterVolumeSpecName: "kube-api-access-nz688") pod "a55f475f-04aa-4317-bddc-6b7f0b887ca4" (UID: "a55f475f-04aa-4317-bddc-6b7f0b887ca4"). InnerVolumeSpecName "kube-api-access-nz688". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:19:33.610134 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:33.610096    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz688\" (UniqueName: \"kubernetes.io/projected/a55f475f-04aa-4317-bddc-6b7f0b887ca4-kube-api-access-nz688\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:19:33.610134 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:33.610128    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55f475f-04aa-4317-bddc-6b7f0b887ca4-plugins-config-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:19:34.197897 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:34.197857    2564 generic.go:356] "Generic (PLEG): container finished" podID="a55f475f-04aa-4317-bddc-6b7f0b887ca4" containerID="00501276aa6b2dec41eabd5393b3702e6682963528e964aad780c300aaf527ca" exitCode=0
Sep 17 22:19:34.198407 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:34.197901    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l"
Sep 17 22:19:34.198407 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:34.197941    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l" event={"ID":"a55f475f-04aa-4317-bddc-6b7f0b887ca4","Type":"ContainerDied","Data":"00501276aa6b2dec41eabd5393b3702e6682963528e964aad780c300aaf527ca"}
Sep 17 22:19:34.198407 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:34.197997    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l" event={"ID":"a55f475f-04aa-4317-bddc-6b7f0b887ca4","Type":"ContainerDied","Data":"ed058c050d367527d85a6b8f110c712266a54b7768255700532ce5eae6d53cd2"}
Sep 17 22:19:34.198407 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:34.198022    2564 scope.go:117] "RemoveContainer" containerID="00501276aa6b2dec41eabd5393b3702e6682963528e964aad780c300aaf527ca"
Sep 17 22:19:34.207695 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:34.207681    2564 scope.go:117] "RemoveContainer" containerID="00501276aa6b2dec41eabd5393b3702e6682963528e964aad780c300aaf527ca"
Sep 17 22:19:34.207975 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:19:34.207956    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00501276aa6b2dec41eabd5393b3702e6682963528e964aad780c300aaf527ca\": container with ID starting with 00501276aa6b2dec41eabd5393b3702e6682963528e964aad780c300aaf527ca not found: ID does not exist" containerID="00501276aa6b2dec41eabd5393b3702e6682963528e964aad780c300aaf527ca"
Sep 17 22:19:34.208026 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:34.207984    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00501276aa6b2dec41eabd5393b3702e6682963528e964aad780c300aaf527ca"} err="failed to get container status \"00501276aa6b2dec41eabd5393b3702e6682963528e964aad780c300aaf527ca\": rpc error: code = NotFound desc = could not find container \"00501276aa6b2dec41eabd5393b3702e6682963528e964aad780c300aaf527ca\": container with ID starting with 00501276aa6b2dec41eabd5393b3702e6682963528e964aad780c300aaf527ca not found: ID does not exist"
Sep 17 22:19:34.225038 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:34.225012    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l"]
Sep 17 22:19:34.228141 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:34.228120    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-813cee-d56db7ffd-cjc6l"]
Sep 17 22:19:34.760161 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:34.760123    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ab992b1-c561-45a5-9766-bb7fbd37fe37" path="/var/lib/kubelet/pods/0ab992b1-c561-45a5-9766-bb7fbd37fe37/volumes"
Sep 17 22:19:34.760779 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:34.760749    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a55f475f-04aa-4317-bddc-6b7f0b887ca4" path="/var/lib/kubelet/pods/a55f475f-04aa-4317-bddc-6b7f0b887ca4/volumes"
Sep 17 22:19:39.934439 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:39.934401    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"]
Sep 17 22:19:39.934839 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:39.934667    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn" podUID="746564fc-cdcf-47e9-8f05-d6530fb5769e" containerName="istio-proxy" containerID="cri-o://baf1043eabf66a0eac6e6f89e83c994c678dc25fb91d09c59221a79d42a00741" gracePeriod=30
Sep 17 22:19:40.308121 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:40.308039    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f8dad-sgtjc"]
Sep 17 22:19:40.308427 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:40.308415    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ab992b1-c561-45a5-9766-bb7fbd37fe37" containerName="revoke-keys"
Sep 17 22:19:40.308478 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:40.308428    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ab992b1-c561-45a5-9766-bb7fbd37fe37" containerName="revoke-keys"
Sep 17 22:19:40.308478 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:40.308439    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a55f475f-04aa-4317-bddc-6b7f0b887ca4" containerName="payload-pre-processing"
Sep 17 22:19:40.308478 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:40.308446    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a55f475f-04aa-4317-bddc-6b7f0b887ca4" containerName="payload-pre-processing"
Sep 17 22:19:40.308478 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:40.308455    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7d2101c-fae4-4ddc-bbb2-0dd4543905f9" containerName="istio-proxy"
Sep 17 22:19:40.308478 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:40.308461    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7d2101c-fae4-4ddc-bbb2-0dd4543905f9" containerName="istio-proxy"
Sep 17 22:19:40.308478 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:40.308468    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7d2101c-fae4-4ddc-bbb2-0dd4543905f9" containerName="istio-proxy"
Sep 17 22:19:40.308478 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:40.308473    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7d2101c-fae4-4ddc-bbb2-0dd4543905f9" containerName="istio-proxy"
Sep 17 22:19:40.308701 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:40.308535    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7d2101c-fae4-4ddc-bbb2-0dd4543905f9" containerName="istio-proxy"
Sep 17 22:19:40.308701 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:40.308545    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a55f475f-04aa-4317-bddc-6b7f0b887ca4" containerName="payload-pre-processing"
Sep 17 22:19:40.308701 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:40.308552    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ab992b1-c561-45a5-9766-bb7fbd37fe37" containerName="revoke-keys"
Sep 17 22:19:40.312860 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:40.312843    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f8dad-sgtjc"
Sep 17 22:19:40.315514 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:40.315495    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmdbp\""
Sep 17 22:19:40.315702 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:40.315492    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 22:19:40.320593 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:40.320567    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f8dad-sgtjc"]
Sep 17 22:19:40.477361 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:40.477320    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8416850-ebc5-488b-9431-04207af1dd0c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8f8dad-sgtjc\" (UID: \"b8416850-ebc5-488b-9431-04207af1dd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f8dad-sgtjc"
Sep 17 22:19:40.578046 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:40.577960    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8416850-ebc5-488b-9431-04207af1dd0c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8f8dad-sgtjc\" (UID: \"b8416850-ebc5-488b-9431-04207af1dd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f8dad-sgtjc"
Sep 17 22:19:40.578600 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:40.578580    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8416850-ebc5-488b-9431-04207af1dd0c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8f8dad-sgtjc\" (UID: \"b8416850-ebc5-488b-9431-04207af1dd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f8dad-sgtjc"
Sep 17 22:19:40.628703 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:40.628667    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f8dad-sgtjc"
Sep 17 22:19:40.900938 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:40.900911    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f8dad-sgtjc"]
Sep 17 22:19:40.902905 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:19:40.902875    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8416850_ebc5_488b_9431_04207af1dd0c.slice/crio-0e2a65373574889ca380b6f893e17f49fce2880227445b99b03f0e5ceadeae51 WatchSource:0}: Error finding container 0e2a65373574889ca380b6f893e17f49fce2880227445b99b03f0e5ceadeae51: Status 404 returned error can't find the container with id 0e2a65373574889ca380b6f893e17f49fce2880227445b99b03f0e5ceadeae51
Sep 17 22:19:41.238822 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:41.238788    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f8dad-sgtjc" event={"ID":"b8416850-ebc5-488b-9431-04207af1dd0c","Type":"ContainerStarted","Data":"728578aa97acf5bf286744eb5a1b8df5e05a63f118bb6aa46bb903372bcb031b"}
Sep 17 22:19:41.239220 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:41.238850    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f8dad-sgtjc" event={"ID":"b8416850-ebc5-488b-9431-04207af1dd0c","Type":"ContainerStarted","Data":"0e2a65373574889ca380b6f893e17f49fce2880227445b99b03f0e5ceadeae51"}
Sep 17 22:19:41.260106 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:41.260059    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f8dad-sgtjc" podStartSLOduration=1.2600433739999999 podStartE2EDuration="1.260043374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:19:41.256469716 +0000 UTC m=+2297.074796028" watchObservedRunningTime="2026-09-17 22:19:41.260043374 +0000 UTC m=+2297.078369735"
Sep 17 22:19:42.245584 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:42.245548    2564 generic.go:356] "Generic (PLEG): container finished" podID="b8416850-ebc5-488b-9431-04207af1dd0c" containerID="728578aa97acf5bf286744eb5a1b8df5e05a63f118bb6aa46bb903372bcb031b" exitCode=0
Sep 17 22:19:42.245959 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:42.245626    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f8dad-sgtjc" event={"ID":"b8416850-ebc5-488b-9431-04207af1dd0c","Type":"ContainerDied","Data":"728578aa97acf5bf286744eb5a1b8df5e05a63f118bb6aa46bb903372bcb031b"}
Sep 17 22:19:43.395687 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:43.395660    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f8dad-sgtjc"
Sep 17 22:19:43.506956 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:43.506867    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8416850-ebc5-488b-9431-04207af1dd0c-maas-api-service-ca\") pod \"b8416850-ebc5-488b-9431-04207af1dd0c\" (UID: \"b8416850-ebc5-488b-9431-04207af1dd0c\") "
Sep 17 22:19:43.507260 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:43.507239    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8416850-ebc5-488b-9431-04207af1dd0c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b8416850-ebc5-488b-9431-04207af1dd0c" (UID: "b8416850-ebc5-488b-9431-04207af1dd0c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:19:43.607812 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:43.607773    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8416850-ebc5-488b-9431-04207af1dd0c-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:19:44.256789 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:44.256758    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f8dad-sgtjc"
Sep 17 22:19:44.256789 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:44.256758    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f8dad-sgtjc" event={"ID":"b8416850-ebc5-488b-9431-04207af1dd0c","Type":"ContainerDied","Data":"0e2a65373574889ca380b6f893e17f49fce2880227445b99b03f0e5ceadeae51"}
Sep 17 22:19:44.257019 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:44.256820    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e2a65373574889ca380b6f893e17f49fce2880227445b99b03f0e5ceadeae51"
Sep 17 22:19:45.186267 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.186235    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:19:45.264081 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.263993    2564 generic.go:356] "Generic (PLEG): container finished" podID="746564fc-cdcf-47e9-8f05-d6530fb5769e" containerID="baf1043eabf66a0eac6e6f89e83c994c678dc25fb91d09c59221a79d42a00741" exitCode=0
Sep 17 22:19:45.264081 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.264040    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn" event={"ID":"746564fc-cdcf-47e9-8f05-d6530fb5769e","Type":"ContainerDied","Data":"baf1043eabf66a0eac6e6f89e83c994c678dc25fb91d09c59221a79d42a00741"}
Sep 17 22:19:45.264081 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.264043    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"
Sep 17 22:19:45.264375 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.264084    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn" event={"ID":"746564fc-cdcf-47e9-8f05-d6530fb5769e","Type":"ContainerDied","Data":"a59bd45070ac764fe69e088431eef4b5884504d776cf7d29f80472cb40fdbfdd"}
Sep 17 22:19:45.264375 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.264109    2564 scope.go:117] "RemoveContainer" containerID="baf1043eabf66a0eac6e6f89e83c994c678dc25fb91d09c59221a79d42a00741"
Sep 17 22:19:45.272963 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.272943    2564 scope.go:117] "RemoveContainer" containerID="4a840a182776ab6f3e0a80b1df90f0bb5c13d2e230b9d15d18477d71d19aad54"
Sep 17 22:19:45.281015 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.280995    2564 scope.go:117] "RemoveContainer" containerID="baf1043eabf66a0eac6e6f89e83c994c678dc25fb91d09c59221a79d42a00741"
Sep 17 22:19:45.281317 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:19:45.281296    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baf1043eabf66a0eac6e6f89e83c994c678dc25fb91d09c59221a79d42a00741\": container with ID starting with baf1043eabf66a0eac6e6f89e83c994c678dc25fb91d09c59221a79d42a00741 not found: ID does not exist" containerID="baf1043eabf66a0eac6e6f89e83c994c678dc25fb91d09c59221a79d42a00741"
Sep 17 22:19:45.281376 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.281326    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baf1043eabf66a0eac6e6f89e83c994c678dc25fb91d09c59221a79d42a00741"} err="failed to get container status \"baf1043eabf66a0eac6e6f89e83c994c678dc25fb91d09c59221a79d42a00741\": rpc error: code = NotFound desc = could not find container \"baf1043eabf66a0eac6e6f89e83c994c678dc25fb91d09c59221a79d42a00741\": container with ID starting with baf1043eabf66a0eac6e6f89e83c994c678dc25fb91d09c59221a79d42a00741 not found: ID does not exist"
Sep 17 22:19:45.281376 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.281345    2564 scope.go:117] "RemoveContainer" containerID="4a840a182776ab6f3e0a80b1df90f0bb5c13d2e230b9d15d18477d71d19aad54"
Sep 17 22:19:45.281619 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:19:45.281598    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a840a182776ab6f3e0a80b1df90f0bb5c13d2e230b9d15d18477d71d19aad54\": container with ID starting with 4a840a182776ab6f3e0a80b1df90f0bb5c13d2e230b9d15d18477d71d19aad54 not found: ID does not exist" containerID="4a840a182776ab6f3e0a80b1df90f0bb5c13d2e230b9d15d18477d71d19aad54"
Sep 17 22:19:45.281669 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.281625    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a840a182776ab6f3e0a80b1df90f0bb5c13d2e230b9d15d18477d71d19aad54"} err="failed to get container status \"4a840a182776ab6f3e0a80b1df90f0bb5c13d2e230b9d15d18477d71d19aad54\": rpc error: code = NotFound desc = could not find container \"4a840a182776ab6f3e0a80b1df90f0bb5c13d2e230b9d15d18477d71d19aad54\": container with ID starting with 4a840a182776ab6f3e0a80b1df90f0bb5c13d2e230b9d15d18477d71d19aad54 not found: ID does not exist"
Sep 17 22:19:45.324438 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.324406    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-workload-certs\") pod \"746564fc-cdcf-47e9-8f05-d6530fb5769e\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") "
Sep 17 22:19:45.324604 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.324476    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746564fc-cdcf-47e9-8f05-d6530fb5769e-istiod-ca-cert\") pod \"746564fc-cdcf-47e9-8f05-d6530fb5769e\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") "
Sep 17 22:19:45.324604 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.324507    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-istio-data\") pod \"746564fc-cdcf-47e9-8f05-d6530fb5769e\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") "
Sep 17 22:19:45.324604 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.324584    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-credential-socket\") pod \"746564fc-cdcf-47e9-8f05-d6530fb5769e\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") "
Sep 17 22:19:45.324747 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.324609    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-istio-envoy\") pod \"746564fc-cdcf-47e9-8f05-d6530fb5769e\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") "
Sep 17 22:19:45.324747 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.324638    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746564fc-cdcf-47e9-8f05-d6530fb5769e-istio-podinfo\") pod \"746564fc-cdcf-47e9-8f05-d6530fb5769e\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") "
Sep 17 22:19:45.324747 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.324669    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78p5b\" (UniqueName: \"kubernetes.io/projected/746564fc-cdcf-47e9-8f05-d6530fb5769e-kube-api-access-78p5b\") pod \"746564fc-cdcf-47e9-8f05-d6530fb5769e\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") "
Sep 17 22:19:45.324747 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.324698    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-workload-socket\") pod \"746564fc-cdcf-47e9-8f05-d6530fb5769e\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") "
Sep 17 22:19:45.324747 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.324730    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "746564fc-cdcf-47e9-8f05-d6530fb5769e" (UID: "746564fc-cdcf-47e9-8f05-d6530fb5769e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:19:45.324747 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.324743    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746564fc-cdcf-47e9-8f05-d6530fb5769e-istio-token\") pod \"746564fc-cdcf-47e9-8f05-d6530fb5769e\" (UID: \"746564fc-cdcf-47e9-8f05-d6530fb5769e\") "
Sep 17 22:19:45.325066 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.324805    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-istio-data" (OuterVolumeSpecName: "istio-data") pod "746564fc-cdcf-47e9-8f05-d6530fb5769e" (UID: "746564fc-cdcf-47e9-8f05-d6530fb5769e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:19:45.325066 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.324855    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "746564fc-cdcf-47e9-8f05-d6530fb5769e" (UID: "746564fc-cdcf-47e9-8f05-d6530fb5769e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:19:45.325066 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.324937    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/746564fc-cdcf-47e9-8f05-d6530fb5769e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "746564fc-cdcf-47e9-8f05-d6530fb5769e" (UID: "746564fc-cdcf-47e9-8f05-d6530fb5769e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:19:45.325233 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.325069    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "746564fc-cdcf-47e9-8f05-d6530fb5769e" (UID: "746564fc-cdcf-47e9-8f05-d6530fb5769e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:19:45.325290 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.325236    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746564fc-cdcf-47e9-8f05-d6530fb5769e-istiod-ca-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:19:45.325290 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.325255    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-istio-data\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:19:45.325290 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.325269    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-credential-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:19:45.325290 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.325282    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-workload-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:19:45.325458 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.325295    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-workload-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:19:45.327012 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.326987    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746564fc-cdcf-47e9-8f05-d6530fb5769e-kube-api-access-78p5b" (OuterVolumeSpecName: "kube-api-access-78p5b") pod "746564fc-cdcf-47e9-8f05-d6530fb5769e" (UID: "746564fc-cdcf-47e9-8f05-d6530fb5769e"). InnerVolumeSpecName "kube-api-access-78p5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:19:45.327137 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.327023    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "746564fc-cdcf-47e9-8f05-d6530fb5769e" (UID: "746564fc-cdcf-47e9-8f05-d6530fb5769e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:19:45.327386 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.327364    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/746564fc-cdcf-47e9-8f05-d6530fb5769e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "746564fc-cdcf-47e9-8f05-d6530fb5769e" (UID: "746564fc-cdcf-47e9-8f05-d6530fb5769e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 22:19:45.327451 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.327383    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746564fc-cdcf-47e9-8f05-d6530fb5769e-istio-token" (OuterVolumeSpecName: "istio-token") pod "746564fc-cdcf-47e9-8f05-d6530fb5769e" (UID: "746564fc-cdcf-47e9-8f05-d6530fb5769e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:19:45.426391 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.426351    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746564fc-cdcf-47e9-8f05-d6530fb5769e-istio-podinfo\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:19:45.426391 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.426381    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78p5b\" (UniqueName: \"kubernetes.io/projected/746564fc-cdcf-47e9-8f05-d6530fb5769e-kube-api-access-78p5b\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:19:45.426391 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.426391    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746564fc-cdcf-47e9-8f05-d6530fb5769e-istio-token\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:19:45.426391 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.426400    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746564fc-cdcf-47e9-8f05-d6530fb5769e-istio-envoy\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:19:45.589568 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.589535    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"]
Sep 17 22:19:45.593445 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:45.593417    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-813cee-openshift-default-777c895b74-rjjwn"]
Sep 17 22:19:46.753815 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:46.753784    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="746564fc-cdcf-47e9-8f05-d6530fb5769e" path="/var/lib/kubelet/pods/746564fc-cdcf-47e9-8f05-d6530fb5769e/volumes"
Sep 17 22:19:51.059852 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:51.059807    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8f8dad-29828055-v8dxk"]
Sep 17 22:19:51.066934 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:51.066908    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8f8dad-29828055-v8dxk"]
Sep 17 22:19:51.127287 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:51.127257    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc"]
Sep 17 22:19:51.127492 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:51.127459    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc" podUID="8172d37e-4071-4548-9ac1-c6378f7e6398" containerName="payload-pre-processing" containerID="cri-o://ed4e4bf6ae62653b2a8d569341b098626325f09b4e7ce1aa077b96fedc8055ee" gracePeriod=30
Sep 17 22:19:51.301335 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:51.301303    2564 generic.go:356] "Generic (PLEG): container finished" podID="8172d37e-4071-4548-9ac1-c6378f7e6398" containerID="ed4e4bf6ae62653b2a8d569341b098626325f09b4e7ce1aa077b96fedc8055ee" exitCode=0
Sep 17 22:19:51.301486 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:51.301375    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc" event={"ID":"8172d37e-4071-4548-9ac1-c6378f7e6398","Type":"ContainerDied","Data":"ed4e4bf6ae62653b2a8d569341b098626325f09b4e7ce1aa077b96fedc8055ee"}
Sep 17 22:19:51.397698 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:51.397668    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f8dad-sgtjc"]
Sep 17 22:19:51.398887 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:51.398857    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc"
Sep 17 22:19:51.400681 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:51.400659    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f8dad-sgtjc"]
Sep 17 22:19:51.481466 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:51.481433    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8172d37e-4071-4548-9ac1-c6378f7e6398-plugins-config-volume\") pod \"8172d37e-4071-4548-9ac1-c6378f7e6398\" (UID: \"8172d37e-4071-4548-9ac1-c6378f7e6398\") "
Sep 17 22:19:51.481466 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:51.481475    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ml4c\" (UniqueName: \"kubernetes.io/projected/8172d37e-4071-4548-9ac1-c6378f7e6398-kube-api-access-8ml4c\") pod \"8172d37e-4071-4548-9ac1-c6378f7e6398\" (UID: \"8172d37e-4071-4548-9ac1-c6378f7e6398\") "
Sep 17 22:19:51.481868 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:51.481847    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8172d37e-4071-4548-9ac1-c6378f7e6398-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8172d37e-4071-4548-9ac1-c6378f7e6398" (UID: "8172d37e-4071-4548-9ac1-c6378f7e6398"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:19:51.484003 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:51.483957    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8172d37e-4071-4548-9ac1-c6378f7e6398-kube-api-access-8ml4c" (OuterVolumeSpecName: "kube-api-access-8ml4c") pod "8172d37e-4071-4548-9ac1-c6378f7e6398" (UID: "8172d37e-4071-4548-9ac1-c6378f7e6398"). InnerVolumeSpecName "kube-api-access-8ml4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:19:51.582402 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:51.582366    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8172d37e-4071-4548-9ac1-c6378f7e6398-plugins-config-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:19:51.582402 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:51.582396    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ml4c\" (UniqueName: \"kubernetes.io/projected/8172d37e-4071-4548-9ac1-c6378f7e6398-kube-api-access-8ml4c\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:19:52.307903 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:52.307873    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc"
Sep 17 22:19:52.307903 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:52.307881    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc" event={"ID":"8172d37e-4071-4548-9ac1-c6378f7e6398","Type":"ContainerDied","Data":"124f1d43decdfa548b366ee3d4c1b6688540580332746fddf710126c23502b1f"}
Sep 17 22:19:52.308420 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:52.307943    2564 scope.go:117] "RemoveContainer" containerID="ed4e4bf6ae62653b2a8d569341b098626325f09b4e7ce1aa077b96fedc8055ee"
Sep 17 22:19:52.357926 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:52.357894    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc"]
Sep 17 22:19:52.397107 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:52.397076    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8f8dad-85dbf497bb-tvvcc"]
Sep 17 22:19:52.753822 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:52.753790    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8172d37e-4071-4548-9ac1-c6378f7e6398" path="/var/lib/kubelet/pods/8172d37e-4071-4548-9ac1-c6378f7e6398/volumes"
Sep 17 22:19:52.754117 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:52.754105    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8416850-ebc5-488b-9431-04207af1dd0c" path="/var/lib/kubelet/pods/b8416850-ebc5-488b-9431-04207af1dd0c/volumes"
Sep 17 22:19:52.754378 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:52.754365    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e987c5ac-a0f3-4fbc-b4e3-a7eb1ea01899" path="/var/lib/kubelet/pods/e987c5ac-a0f3-4fbc-b4e3-a7eb1ea01899/volumes"
Sep 17 22:19:58.229801 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:58.229764    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"]
Sep 17 22:19:58.230310 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:58.230037    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs" podUID="c4e2600c-dda1-43fb-8b0f-a9128366c25d" containerName="istio-proxy" containerID="cri-o://34e223c021db7ef2839cde6c3968835006c7118e85e220c3d2b8e37965087f8e" gracePeriod=30
Sep 17 22:19:58.630128 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:58.630043    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-35b181-mnjnz"]
Sep 17 22:19:58.630538 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:58.630519    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8172d37e-4071-4548-9ac1-c6378f7e6398" containerName="payload-pre-processing"
Sep 17 22:19:58.630631 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:58.630540    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8172d37e-4071-4548-9ac1-c6378f7e6398" containerName="payload-pre-processing"
Sep 17 22:19:58.630631 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:58.630551    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="746564fc-cdcf-47e9-8f05-d6530fb5769e" containerName="istio-proxy"
Sep 17 22:19:58.630631 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:58.630560    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="746564fc-cdcf-47e9-8f05-d6530fb5769e" containerName="istio-proxy"
Sep 17 22:19:58.630631 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:58.630585    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8416850-ebc5-488b-9431-04207af1dd0c" containerName="revoke-keys"
Sep 17 22:19:58.630631 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:58.630594    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8416850-ebc5-488b-9431-04207af1dd0c" containerName="revoke-keys"
Sep 17 22:19:58.630869 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:58.630674    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8172d37e-4071-4548-9ac1-c6378f7e6398" containerName="payload-pre-processing"
Sep 17 22:19:58.630869 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:58.630686    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7d2101c-fae4-4ddc-bbb2-0dd4543905f9" containerName="istio-proxy"
Sep 17 22:19:58.630869 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:58.630696    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="746564fc-cdcf-47e9-8f05-d6530fb5769e" containerName="istio-proxy"
Sep 17 22:19:58.630869 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:58.630706    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="746564fc-cdcf-47e9-8f05-d6530fb5769e" containerName="istio-proxy"
Sep 17 22:19:58.630869 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:58.630714    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8416850-ebc5-488b-9431-04207af1dd0c" containerName="revoke-keys"
Sep 17 22:19:58.635077 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:58.635056    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-35b181-mnjnz"
Sep 17 22:19:58.637842 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:58.637817    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 22:19:58.637951 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:58.637882    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmdbp\""
Sep 17 22:19:58.642546 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:58.642272    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-35b181-mnjnz"]
Sep 17 22:19:58.736861 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:58.736829    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3387722a-c655-4ce7-b658-e5030ea01cb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-35b181-mnjnz\" (UID: \"3387722a-c655-4ce7-b658-e5030ea01cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-35b181-mnjnz"
Sep 17 22:19:58.838153 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:58.837981    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3387722a-c655-4ce7-b658-e5030ea01cb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-35b181-mnjnz\" (UID: \"3387722a-c655-4ce7-b658-e5030ea01cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-35b181-mnjnz"
Sep 17 22:19:58.838699 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:58.838676    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3387722a-c655-4ce7-b658-e5030ea01cb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-35b181-mnjnz\" (UID: \"3387722a-c655-4ce7-b658-e5030ea01cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-35b181-mnjnz"
Sep 17 22:19:58.944880 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:58.944845    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 22:19:58.945027 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:58.944913    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs" podUID="c4e2600c-dda1-43fb-8b0f-a9128366c25d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:19:58.947037 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:58.947011    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-35b181-mnjnz"
Sep 17 22:19:59.231689 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:59.231661    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-35b181-mnjnz"]
Sep 17 22:19:59.233383 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:19:59.233354    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3387722a_c655_4ce7_b658_e5030ea01cb2.slice/crio-e5a14148e00f92f9c1e5d0d80b949582cd0cf3b9b27f861a9e9100cfc2c5717c WatchSource:0}: Error finding container e5a14148e00f92f9c1e5d0d80b949582cd0cf3b9b27f861a9e9100cfc2c5717c: Status 404 returned error can't find the container with id e5a14148e00f92f9c1e5d0d80b949582cd0cf3b9b27f861a9e9100cfc2c5717c
Sep 17 22:19:59.347959 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:19:59.347920    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-35b181-mnjnz" event={"ID":"3387722a-c655-4ce7-b658-e5030ea01cb2","Type":"ContainerStarted","Data":"e5a14148e00f92f9c1e5d0d80b949582cd0cf3b9b27f861a9e9100cfc2c5717c"}
Sep 17 22:20:00.355408 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:00.355312    2564 generic.go:356] "Generic (PLEG): container finished" podID="3387722a-c655-4ce7-b658-e5030ea01cb2" containerID="1841b97303f994198d50683eb8a0ed8783beb660c3baa8828e21f82269a64ede" exitCode=0
Sep 17 22:20:00.355408 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:00.355391    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-35b181-mnjnz" event={"ID":"3387722a-c655-4ce7-b658-e5030ea01cb2","Type":"ContainerDied","Data":"1841b97303f994198d50683eb8a0ed8783beb660c3baa8828e21f82269a64ede"}
Sep 17 22:20:01.497787 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:01.497758    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-35b181-mnjnz"
Sep 17 22:20:01.561459 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:01.561425    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3387722a-c655-4ce7-b658-e5030ea01cb2-maas-api-service-ca\") pod \"3387722a-c655-4ce7-b658-e5030ea01cb2\" (UID: \"3387722a-c655-4ce7-b658-e5030ea01cb2\") "
Sep 17 22:20:01.561794 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:01.561772    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3387722a-c655-4ce7-b658-e5030ea01cb2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3387722a-c655-4ce7-b658-e5030ea01cb2" (UID: "3387722a-c655-4ce7-b658-e5030ea01cb2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:20:01.663147 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:01.663112    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3387722a-c655-4ce7-b658-e5030ea01cb2-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:20:02.366843 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:02.366807    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-35b181-mnjnz" event={"ID":"3387722a-c655-4ce7-b658-e5030ea01cb2","Type":"ContainerDied","Data":"e5a14148e00f92f9c1e5d0d80b949582cd0cf3b9b27f861a9e9100cfc2c5717c"}
Sep 17 22:20:02.366843 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:02.366840    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-35b181-mnjnz"
Sep 17 22:20:02.367078 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:02.366881    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5a14148e00f92f9c1e5d0d80b949582cd0cf3b9b27f861a9e9100cfc2c5717c"
Sep 17 22:20:03.373697 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.373609    2564 generic.go:356] "Generic (PLEG): container finished" podID="c4e2600c-dda1-43fb-8b0f-a9128366c25d" containerID="34e223c021db7ef2839cde6c3968835006c7118e85e220c3d2b8e37965087f8e" exitCode=0
Sep 17 22:20:03.373697 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.373682    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs" event={"ID":"c4e2600c-dda1-43fb-8b0f-a9128366c25d","Type":"ContainerDied","Data":"34e223c021db7ef2839cde6c3968835006c7118e85e220c3d2b8e37965087f8e"}
Sep 17 22:20:03.374229 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.373745    2564 scope.go:117] "RemoveContainer" containerID="33de90316c6f9bf770b6e5f9feb16c5bb2d103b0fb857cbd4dffca7cc14eb5aa"
Sep 17 22:20:03.483262 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.483238    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:20:03.578995 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.578958    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istio-token\") pod \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") "
Sep 17 22:20:03.578995 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.578995    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istio-envoy\") pod \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") "
Sep 17 22:20:03.579251 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.579042    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-workload-socket\") pod \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") "
Sep 17 22:20:03.579251 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.579072    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istiod-ca-cert\") pod \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") "
Sep 17 22:20:03.579251 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.579099    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istio-data\") pod \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") "
Sep 17 22:20:03.579251 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.579133    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istio-podinfo\") pod \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") "
Sep 17 22:20:03.579542 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.579307    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-workload-certs\") pod \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") "
Sep 17 22:20:03.579542 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.579349    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dzxp\" (UniqueName: \"kubernetes.io/projected/c4e2600c-dda1-43fb-8b0f-a9128366c25d-kube-api-access-6dzxp\") pod \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") "
Sep 17 22:20:03.579542 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.579398    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-credential-socket\") pod \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\" (UID: \"c4e2600c-dda1-43fb-8b0f-a9128366c25d\") "
Sep 17 22:20:03.579542 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.579495    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istio-data" (OuterVolumeSpecName: "istio-data") pod "c4e2600c-dda1-43fb-8b0f-a9128366c25d" (UID: "c4e2600c-dda1-43fb-8b0f-a9128366c25d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:20:03.579542 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.579509    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c4e2600c-dda1-43fb-8b0f-a9128366c25d" (UID: "c4e2600c-dda1-43fb-8b0f-a9128366c25d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:20:03.579816 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.579603    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c4e2600c-dda1-43fb-8b0f-a9128366c25d" (UID: "c4e2600c-dda1-43fb-8b0f-a9128366c25d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:20:03.579816 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.579613    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c4e2600c-dda1-43fb-8b0f-a9128366c25d" (UID: "c4e2600c-dda1-43fb-8b0f-a9128366c25d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:20:03.579816 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.579737    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c4e2600c-dda1-43fb-8b0f-a9128366c25d" (UID: "c4e2600c-dda1-43fb-8b0f-a9128366c25d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:20:03.579978 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.579866    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-workload-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:20:03.579978 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.579883    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-credential-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:20:03.579978 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.579898    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-workload-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:20:03.579978 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.579912    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istiod-ca-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:20:03.579978 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.579924    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istio-data\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:20:03.581475 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.581457    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c4e2600c-dda1-43fb-8b0f-a9128366c25d" (UID: "c4e2600c-dda1-43fb-8b0f-a9128366c25d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 22:20:03.581679 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.581664    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e2600c-dda1-43fb-8b0f-a9128366c25d-kube-api-access-6dzxp" (OuterVolumeSpecName: "kube-api-access-6dzxp") pod "c4e2600c-dda1-43fb-8b0f-a9128366c25d" (UID: "c4e2600c-dda1-43fb-8b0f-a9128366c25d"). InnerVolumeSpecName "kube-api-access-6dzxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:20:03.581966 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.581952    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c4e2600c-dda1-43fb-8b0f-a9128366c25d" (UID: "c4e2600c-dda1-43fb-8b0f-a9128366c25d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:20:03.582012 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.581983    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istio-token" (OuterVolumeSpecName: "istio-token") pod "c4e2600c-dda1-43fb-8b0f-a9128366c25d" (UID: "c4e2600c-dda1-43fb-8b0f-a9128366c25d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:20:03.680617 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.680581    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istio-token\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:20:03.680617 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.680612    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istio-envoy\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:20:03.680617 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.680620    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e2600c-dda1-43fb-8b0f-a9128366c25d-istio-podinfo\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:20:03.680617 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:03.680629    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dzxp\" (UniqueName: \"kubernetes.io/projected/c4e2600c-dda1-43fb-8b0f-a9128366c25d-kube-api-access-6dzxp\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:20:04.379538 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:04.379506    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"
Sep 17 22:20:04.380002 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:04.379542    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs" event={"ID":"c4e2600c-dda1-43fb-8b0f-a9128366c25d","Type":"ContainerDied","Data":"33d3b9550b05d0327660fa3eb1610db063b32a1605895c105b7b6481b3efa496"}
Sep 17 22:20:04.380002 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:04.379602    2564 scope.go:117] "RemoveContainer" containerID="34e223c021db7ef2839cde6c3968835006c7118e85e220c3d2b8e37965087f8e"
Sep 17 22:20:04.401185 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:04.401144    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"]
Sep 17 22:20:04.404954 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:04.404931    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8f8dad-openshift-default-55b6f478f8-r5hzs"]
Sep 17 22:20:04.753792 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:04.753761    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4e2600c-dda1-43fb-8b0f-a9128366c25d" path="/var/lib/kubelet/pods/c4e2600c-dda1-43fb-8b0f-a9128366c25d/volumes"
Sep 17 22:20:09.210776 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:09.210727    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-35b181-29828055-v5dl5"]
Sep 17 22:20:09.212471 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:09.212448    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-35b181-29828055-v5dl5"]
Sep 17 22:20:09.241889 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:09.241856    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h"]
Sep 17 22:20:09.242154 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:09.242108    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h" podUID="dc757bd4-4892-426a-b53b-6b7bab26d3cd" containerName="payload-processing" containerID="cri-o://239cdf3b742e6c9feb6159c92f691cf2034f36ffe6648b4b462c44b59448789d" gracePeriod=30
Sep 17 22:20:09.406880 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:09.406849    2564 generic.go:356] "Generic (PLEG): container finished" podID="dc757bd4-4892-426a-b53b-6b7bab26d3cd" containerID="239cdf3b742e6c9feb6159c92f691cf2034f36ffe6648b4b462c44b59448789d" exitCode=0
Sep 17 22:20:09.407070 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:09.406934    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h" event={"ID":"dc757bd4-4892-426a-b53b-6b7bab26d3cd","Type":"ContainerDied","Data":"239cdf3b742e6c9feb6159c92f691cf2034f36ffe6648b4b462c44b59448789d"}
Sep 17 22:20:09.457842 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:09.457806    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-35b181-mnjnz"]
Sep 17 22:20:09.461983 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:09.461913    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-35b181-mnjnz"]
Sep 17 22:20:09.510650 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:09.510622    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h"
Sep 17 22:20:09.632693 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:09.632663    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc757bd4-4892-426a-b53b-6b7bab26d3cd-plugins-config-volume\") pod \"dc757bd4-4892-426a-b53b-6b7bab26d3cd\" (UID: \"dc757bd4-4892-426a-b53b-6b7bab26d3cd\") "
Sep 17 22:20:09.632866 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:09.632770    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzld4\" (UniqueName: \"kubernetes.io/projected/dc757bd4-4892-426a-b53b-6b7bab26d3cd-kube-api-access-nzld4\") pod \"dc757bd4-4892-426a-b53b-6b7bab26d3cd\" (UID: \"dc757bd4-4892-426a-b53b-6b7bab26d3cd\") "
Sep 17 22:20:09.633080 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:09.633061    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc757bd4-4892-426a-b53b-6b7bab26d3cd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dc757bd4-4892-426a-b53b-6b7bab26d3cd" (UID: "dc757bd4-4892-426a-b53b-6b7bab26d3cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:20:09.634996 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:09.634973    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc757bd4-4892-426a-b53b-6b7bab26d3cd-kube-api-access-nzld4" (OuterVolumeSpecName: "kube-api-access-nzld4") pod "dc757bd4-4892-426a-b53b-6b7bab26d3cd" (UID: "dc757bd4-4892-426a-b53b-6b7bab26d3cd"). InnerVolumeSpecName "kube-api-access-nzld4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:20:09.733536 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:09.733453    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc757bd4-4892-426a-b53b-6b7bab26d3cd-plugins-config-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:20:09.733536 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:09.733477    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzld4\" (UniqueName: \"kubernetes.io/projected/dc757bd4-4892-426a-b53b-6b7bab26d3cd-kube-api-access-nzld4\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:20:10.413650 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:10.413625    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h"
Sep 17 22:20:10.414074 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:10.413621    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h" event={"ID":"dc757bd4-4892-426a-b53b-6b7bab26d3cd","Type":"ContainerDied","Data":"175b848bd600500d587c87f7e3418a4ca10eff51dc9e0d2a80b80fe0ba92072c"}
Sep 17 22:20:10.414074 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:10.413751    2564 scope.go:117] "RemoveContainer" containerID="239cdf3b742e6c9feb6159c92f691cf2034f36ffe6648b4b462c44b59448789d"
Sep 17 22:20:10.439436 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:10.439407    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h"]
Sep 17 22:20:10.444894 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:10.444874    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-35b181-777c5f9b69-jc62h"]
Sep 17 22:20:10.753422 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:10.753342    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3387722a-c655-4ce7-b658-e5030ea01cb2" path="/var/lib/kubelet/pods/3387722a-c655-4ce7-b658-e5030ea01cb2/volumes"
Sep 17 22:20:10.753623 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:10.753610    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93756527-cf7c-4fb6-a6d0-4b8cad8fc89a" path="/var/lib/kubelet/pods/93756527-cf7c-4fb6-a6d0-4b8cad8fc89a/volumes"
Sep 17 22:20:10.753885 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:20:10.753874    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc757bd4-4892-426a-b53b-6b7bab26d3cd" path="/var/lib/kubelet/pods/dc757bd4-4892-426a-b53b-6b7bab26d3cd/volumes"
Sep 17 22:21:24.858869 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:21:24.858837    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdcnw_6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2/ovn-acl-logging/0.log"
Sep 17 22:21:24.861430 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:21:24.861306    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdcnw_6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2/ovn-acl-logging/0.log"
Sep 17 22:21:25.377757 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:21:25.377727    2564 scope.go:117] "RemoveContainer" containerID="2ca8a3a494fc20bd5661bc07164e971656a1a37f3e5a6ec5a0bf9ab25e7d4375"
Sep 17 22:21:25.385934 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:21:25.385918    2564 scope.go:117] "RemoveContainer" containerID="a76c6114261abfbde2eea1cbc5d59e7906cc0396eae905ae2ba1424cbc36972f"
Sep 17 22:23:14.608285 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:23:14.608249    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf"]
Sep 17 22:23:14.608828 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:23:14.608471    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf" podUID="6b923825-e691-4554-aef8-c604b8fa3749" containerName="manager" containerID="cri-o://0ff0a92007b5b90d8169aafb6a281e55a7c5f9ef653ffe1baff4923dded31237" gracePeriod=10
Sep 17 22:23:15.686951 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:23:15.686856    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf" podUID="6b923825-e691-4554-aef8-c604b8fa3749" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.28:8081/readyz\": dial tcp 10.133.0.28:8081: connect: connection refused"
Sep 17 22:23:15.865049 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:23:15.865023    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf"
Sep 17 22:23:15.953916 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:23:15.953837    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b923825-e691-4554-aef8-c604b8fa3749-extensions-socket-volume\") pod \"6b923825-e691-4554-aef8-c604b8fa3749\" (UID: \"6b923825-e691-4554-aef8-c604b8fa3749\") "
Sep 17 22:23:15.953916 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:23:15.953903    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7wlx\" (UniqueName: \"kubernetes.io/projected/6b923825-e691-4554-aef8-c604b8fa3749-kube-api-access-h7wlx\") pod \"6b923825-e691-4554-aef8-c604b8fa3749\" (UID: \"6b923825-e691-4554-aef8-c604b8fa3749\") "
Sep 17 22:23:15.954331 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:23:15.954301    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b923825-e691-4554-aef8-c604b8fa3749-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6b923825-e691-4554-aef8-c604b8fa3749" (UID: "6b923825-e691-4554-aef8-c604b8fa3749"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:23:15.956043 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:23:15.956019    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b923825-e691-4554-aef8-c604b8fa3749-kube-api-access-h7wlx" (OuterVolumeSpecName: "kube-api-access-h7wlx") pod "6b923825-e691-4554-aef8-c604b8fa3749" (UID: "6b923825-e691-4554-aef8-c604b8fa3749"). InnerVolumeSpecName "kube-api-access-h7wlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:23:16.054562 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:23:16.054529    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b923825-e691-4554-aef8-c604b8fa3749-extensions-socket-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:23:16.054562 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:23:16.054555    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7wlx\" (UniqueName: \"kubernetes.io/projected/6b923825-e691-4554-aef8-c604b8fa3749-kube-api-access-h7wlx\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:23:16.315602 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:23:16.315513    2564 generic.go:356] "Generic (PLEG): container finished" podID="6b923825-e691-4554-aef8-c604b8fa3749" containerID="0ff0a92007b5b90d8169aafb6a281e55a7c5f9ef653ffe1baff4923dded31237" exitCode=0
Sep 17 22:23:16.315602 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:23:16.315562    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf"
Sep 17 22:23:16.315787 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:23:16.315598    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf" event={"ID":"6b923825-e691-4554-aef8-c604b8fa3749","Type":"ContainerDied","Data":"0ff0a92007b5b90d8169aafb6a281e55a7c5f9ef653ffe1baff4923dded31237"}
Sep 17 22:23:16.315787 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:23:16.315656    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf" event={"ID":"6b923825-e691-4554-aef8-c604b8fa3749","Type":"ContainerDied","Data":"146beaaf8e13f2dbad98466f0ab6d92e538f802fdb36d59cf42de3e637407b31"}
Sep 17 22:23:16.315787 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:23:16.315679    2564 scope.go:117] "RemoveContainer" containerID="0ff0a92007b5b90d8169aafb6a281e55a7c5f9ef653ffe1baff4923dded31237"
Sep 17 22:23:16.325342 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:23:16.325326    2564 scope.go:117] "RemoveContainer" containerID="03f5a5e0832b6f9e684484d9726470a3da140cc1b40b9dfb043693d60b996305"
Sep 17 22:23:16.332813 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:23:16.332795    2564 scope.go:117] "RemoveContainer" containerID="0ff0a92007b5b90d8169aafb6a281e55a7c5f9ef653ffe1baff4923dded31237"
Sep 17 22:23:16.333079 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:23:16.333061    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff0a92007b5b90d8169aafb6a281e55a7c5f9ef653ffe1baff4923dded31237\": container with ID starting with 0ff0a92007b5b90d8169aafb6a281e55a7c5f9ef653ffe1baff4923dded31237 not found: ID does not exist" containerID="0ff0a92007b5b90d8169aafb6a281e55a7c5f9ef653ffe1baff4923dded31237"
Sep 17 22:23:16.333128 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:23:16.333087    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff0a92007b5b90d8169aafb6a281e55a7c5f9ef653ffe1baff4923dded31237"} err="failed to get container status \"0ff0a92007b5b90d8169aafb6a281e55a7c5f9ef653ffe1baff4923dded31237\": rpc error: code = NotFound desc = could not find container \"0ff0a92007b5b90d8169aafb6a281e55a7c5f9ef653ffe1baff4923dded31237\": container with ID starting with 0ff0a92007b5b90d8169aafb6a281e55a7c5f9ef653ffe1baff4923dded31237 not found: ID does not exist"
Sep 17 22:23:16.333128 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:23:16.333102    2564 scope.go:117] "RemoveContainer" containerID="03f5a5e0832b6f9e684484d9726470a3da140cc1b40b9dfb043693d60b996305"
Sep 17 22:23:16.333377 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:23:16.333360    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03f5a5e0832b6f9e684484d9726470a3da140cc1b40b9dfb043693d60b996305\": container with ID starting with 03f5a5e0832b6f9e684484d9726470a3da140cc1b40b9dfb043693d60b996305 not found: ID does not exist" containerID="03f5a5e0832b6f9e684484d9726470a3da140cc1b40b9dfb043693d60b996305"
Sep 17 22:23:16.333432 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:23:16.333386    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f5a5e0832b6f9e684484d9726470a3da140cc1b40b9dfb043693d60b996305"} err="failed to get container status \"03f5a5e0832b6f9e684484d9726470a3da140cc1b40b9dfb043693d60b996305\": rpc error: code = NotFound desc = could not find container \"03f5a5e0832b6f9e684484d9726470a3da140cc1b40b9dfb043693d60b996305\": container with ID starting with 03f5a5e0832b6f9e684484d9726470a3da140cc1b40b9dfb043693d60b996305 not found: ID does not exist"
Sep 17 22:23:16.348766 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:23:16.348746    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf"]
Sep 17 22:23:16.355850 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:23:16.355830    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dxkvf"]
Sep 17 22:23:16.753927 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:23:16.753895    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b923825-e691-4554-aef8-c604b8fa3749" path="/var/lib/kubelet/pods/6b923825-e691-4554-aef8-c604b8fa3749/volumes"
Sep 17 22:24:15.367512 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.367476    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zgmgg"]
Sep 17 22:24:15.367998 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.367838    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="746564fc-cdcf-47e9-8f05-d6530fb5769e" containerName="istio-proxy"
Sep 17 22:24:15.367998 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.367851    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="746564fc-cdcf-47e9-8f05-d6530fb5769e" containerName="istio-proxy"
Sep 17 22:24:15.367998 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.367862    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc757bd4-4892-426a-b53b-6b7bab26d3cd" containerName="payload-processing"
Sep 17 22:24:15.367998 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.367869    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc757bd4-4892-426a-b53b-6b7bab26d3cd" containerName="payload-processing"
Sep 17 22:24:15.367998 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.367877    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4e2600c-dda1-43fb-8b0f-a9128366c25d" containerName="istio-proxy"
Sep 17 22:24:15.367998 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.367883    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4e2600c-dda1-43fb-8b0f-a9128366c25d" containerName="istio-proxy"
Sep 17 22:24:15.367998 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.367890    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4e2600c-dda1-43fb-8b0f-a9128366c25d" containerName="istio-proxy"
Sep 17 22:24:15.367998 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.367895    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4e2600c-dda1-43fb-8b0f-a9128366c25d" containerName="istio-proxy"
Sep 17 22:24:15.367998 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.367905    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b923825-e691-4554-aef8-c604b8fa3749" containerName="manager"
Sep 17 22:24:15.367998 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.367910    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b923825-e691-4554-aef8-c604b8fa3749" containerName="manager"
Sep 17 22:24:15.367998 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.367916    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3387722a-c655-4ce7-b658-e5030ea01cb2" containerName="revoke-keys"
Sep 17 22:24:15.367998 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.367921    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3387722a-c655-4ce7-b658-e5030ea01cb2" containerName="revoke-keys"
Sep 17 22:24:15.367998 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.367925    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b923825-e691-4554-aef8-c604b8fa3749" containerName="manager"
Sep 17 22:24:15.367998 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.367930    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b923825-e691-4554-aef8-c604b8fa3749" containerName="manager"
Sep 17 22:24:15.367998 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.367979    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4e2600c-dda1-43fb-8b0f-a9128366c25d" containerName="istio-proxy"
Sep 17 22:24:15.367998 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.367987    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b923825-e691-4554-aef8-c604b8fa3749" containerName="manager"
Sep 17 22:24:15.367998 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.367992    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc757bd4-4892-426a-b53b-6b7bab26d3cd" containerName="payload-processing"
Sep 17 22:24:15.367998 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.368000    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3387722a-c655-4ce7-b658-e5030ea01cb2" containerName="revoke-keys"
Sep 17 22:24:15.370768 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.370751    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zgmgg"
Sep 17 22:24:15.376016 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.375982    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-z6995\""
Sep 17 22:24:15.392093 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.392067    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zgmgg"]
Sep 17 22:24:15.451854 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.451815    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/59d57036-cd0c-40b5-afbd-a0ffa12eae1e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zgmgg\" (UID: \"59d57036-cd0c-40b5-afbd-a0ffa12eae1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zgmgg"
Sep 17 22:24:15.451854 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.451856    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjx4r\" (UniqueName: \"kubernetes.io/projected/59d57036-cd0c-40b5-afbd-a0ffa12eae1e-kube-api-access-jjx4r\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zgmgg\" (UID: \"59d57036-cd0c-40b5-afbd-a0ffa12eae1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zgmgg"
Sep 17 22:24:15.552504 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.552465    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/59d57036-cd0c-40b5-afbd-a0ffa12eae1e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zgmgg\" (UID: \"59d57036-cd0c-40b5-afbd-a0ffa12eae1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zgmgg"
Sep 17 22:24:15.552504 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.552510    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjx4r\" (UniqueName: \"kubernetes.io/projected/59d57036-cd0c-40b5-afbd-a0ffa12eae1e-kube-api-access-jjx4r\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zgmgg\" (UID: \"59d57036-cd0c-40b5-afbd-a0ffa12eae1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zgmgg"
Sep 17 22:24:15.552848 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.552828    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/59d57036-cd0c-40b5-afbd-a0ffa12eae1e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zgmgg\" (UID: \"59d57036-cd0c-40b5-afbd-a0ffa12eae1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zgmgg"
Sep 17 22:24:15.562303 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.562271    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjx4r\" (UniqueName: \"kubernetes.io/projected/59d57036-cd0c-40b5-afbd-a0ffa12eae1e-kube-api-access-jjx4r\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zgmgg\" (UID: \"59d57036-cd0c-40b5-afbd-a0ffa12eae1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zgmgg"
Sep 17 22:24:15.682072 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.682040    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zgmgg"
Sep 17 22:24:15.965027 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:15.964999    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zgmgg"]
Sep 17 22:24:15.967011 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:24:15.966987    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d57036_cd0c_40b5_afbd_a0ffa12eae1e.slice/crio-a0b4de85f1238a76dde104ffa16afc3d9102960e3d6deb74ba2d7622984bad9b WatchSource:0}: Error finding container a0b4de85f1238a76dde104ffa16afc3d9102960e3d6deb74ba2d7622984bad9b: Status 404 returned error can't find the container with id a0b4de85f1238a76dde104ffa16afc3d9102960e3d6deb74ba2d7622984bad9b
Sep 17 22:24:16.601654 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:16.601615    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zgmgg" event={"ID":"59d57036-cd0c-40b5-afbd-a0ffa12eae1e","Type":"ContainerStarted","Data":"01627d7b59f1ccfd20aa73f36d7fdc0d309b3f0654f52486fdc8921ec32c9025"}
Sep 17 22:24:16.602060 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:16.601677    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zgmgg" event={"ID":"59d57036-cd0c-40b5-afbd-a0ffa12eae1e","Type":"ContainerStarted","Data":"a0b4de85f1238a76dde104ffa16afc3d9102960e3d6deb74ba2d7622984bad9b"}
Sep 17 22:24:16.602060 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:16.601715    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zgmgg"
Sep 17 22:24:16.626251 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:16.626202    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zgmgg" podStartSLOduration=1.626164744 podStartE2EDuration="1.626164744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:24:16.625529089 +0000 UTC m=+2572.443855400" watchObservedRunningTime="2026-09-17 22:24:16.626164744 +0000 UTC m=+2572.444491056"
Sep 17 22:24:25.425996 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:25.425958    2564 scope.go:117] "RemoveContainer" containerID="0e00440ee136cb55bba5914e9aabde48cce0862180a537071ae714e0f51b7341"
Sep 17 22:24:25.434302 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:25.434276    2564 scope.go:117] "RemoveContainer" containerID="c6937f72c81aacd31522126393ec5792bef5157247b80f65be0455b56f79fb30"
Sep 17 22:24:27.608931 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:24:27.608896    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zgmgg"
Sep 17 22:25:10.643094 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.643003    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"]
Sep 17 22:25:10.643705 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.643620    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4e2600c-dda1-43fb-8b0f-a9128366c25d" containerName="istio-proxy"
Sep 17 22:25:10.643705 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.643640    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b923825-e691-4554-aef8-c604b8fa3749" containerName="manager"
Sep 17 22:25:10.646324 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.646303    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:10.648974 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.648951    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-a0433cba-kserve-self-signed-certs\""
Sep 17 22:25:10.656778 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.656750    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"]
Sep 17 22:25:10.747826 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.747787    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c71bea14-44ee-45ef-81fb-d8fbbcae9099-tls-certs\") pod \"e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw\" (UID: \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\") " pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:10.748038 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.747836    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71bea14-44ee-45ef-81fb-d8fbbcae9099-dshm\") pod \"e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw\" (UID: \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\") " pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:10.748038 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.747875    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71bea14-44ee-45ef-81fb-d8fbbcae9099-model-cache\") pod \"e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw\" (UID: \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\") " pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:10.748038 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.747896    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c71bea14-44ee-45ef-81fb-d8fbbcae9099-home\") pod \"e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw\" (UID: \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\") " pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:10.748038 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.747940    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkm99\" (UniqueName: \"kubernetes.io/projected/c71bea14-44ee-45ef-81fb-d8fbbcae9099-kube-api-access-gkm99\") pod \"e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw\" (UID: \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\") " pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:10.748038 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.747963    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71bea14-44ee-45ef-81fb-d8fbbcae9099-tmp-dir\") pod \"e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw\" (UID: \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\") " pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:10.849270 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.849226    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c71bea14-44ee-45ef-81fb-d8fbbcae9099-tls-certs\") pod \"e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw\" (UID: \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\") " pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:10.849270 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.849268    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71bea14-44ee-45ef-81fb-d8fbbcae9099-dshm\") pod \"e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw\" (UID: \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\") " pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:10.849536 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.849293    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71bea14-44ee-45ef-81fb-d8fbbcae9099-model-cache\") pod \"e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw\" (UID: \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\") " pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:10.849536 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.849321    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c71bea14-44ee-45ef-81fb-d8fbbcae9099-home\") pod \"e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw\" (UID: \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\") " pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:10.849536 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.849367    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkm99\" (UniqueName: \"kubernetes.io/projected/c71bea14-44ee-45ef-81fb-d8fbbcae9099-kube-api-access-gkm99\") pod \"e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw\" (UID: \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\") " pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:10.849536 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.849393    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71bea14-44ee-45ef-81fb-d8fbbcae9099-tmp-dir\") pod \"e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw\" (UID: \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\") " pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:10.849829 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.849806    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c71bea14-44ee-45ef-81fb-d8fbbcae9099-home\") pod \"e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw\" (UID: \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\") " pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:10.849892 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.849814    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71bea14-44ee-45ef-81fb-d8fbbcae9099-model-cache\") pod \"e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw\" (UID: \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\") " pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:10.849892 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.849855    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71bea14-44ee-45ef-81fb-d8fbbcae9099-tmp-dir\") pod \"e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw\" (UID: \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\") " pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:10.851781 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.851754    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71bea14-44ee-45ef-81fb-d8fbbcae9099-dshm\") pod \"e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw\" (UID: \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\") " pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:10.852284 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.852268    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c71bea14-44ee-45ef-81fb-d8fbbcae9099-tls-certs\") pod \"e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw\" (UID: \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\") " pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:10.858406 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.858386    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkm99\" (UniqueName: \"kubernetes.io/projected/c71bea14-44ee-45ef-81fb-d8fbbcae9099-kube-api-access-gkm99\") pod \"e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw\" (UID: \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\") " pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:10.956864 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:10.956758    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:11.227970 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:11.227947    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"]
Sep 17 22:25:11.229994 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:25:11.229968    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc71bea14_44ee_45ef_81fb_d8fbbcae9099.slice/crio-7b44e0bc1b2f545f1a5b2b2e76e408c3e70b704ed54c7d5d80175e2e9a24633b WatchSource:0}: Error finding container 7b44e0bc1b2f545f1a5b2b2e76e408c3e70b704ed54c7d5d80175e2e9a24633b: Status 404 returned error can't find the container with id 7b44e0bc1b2f545f1a5b2b2e76e408c3e70b704ed54c7d5d80175e2e9a24633b
Sep 17 22:25:11.878590 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:11.878555    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw" event={"ID":"c71bea14-44ee-45ef-81fb-d8fbbcae9099","Type":"ContainerStarted","Data":"c7e795390593893d23babfed8faae45950eb7790240e9436d09a96ec576fb476"}
Sep 17 22:25:11.879005 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:11.878600    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw" event={"ID":"c71bea14-44ee-45ef-81fb-d8fbbcae9099","Type":"ContainerStarted","Data":"7b44e0bc1b2f545f1a5b2b2e76e408c3e70b704ed54c7d5d80175e2e9a24633b"}
Sep 17 22:25:11.898308 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:11.898251    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw" podStartSLOduration=1.898234668 podStartE2EDuration="1.898234668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:25:11.896602162 +0000 UTC m=+2627.714928509" watchObservedRunningTime="2026-09-17 22:25:11.898234668 +0000 UTC m=+2627.716560979"
Sep 17 22:25:20.957791 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:20.957749    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:20.958307 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:20.957806    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:20.970300 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:20.970275    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:21.939992 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:21.939957    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:25.455616 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:25.455582    2564 scope.go:117] "RemoveContainer" containerID="1beb34f6cabbb24e2a841c1d29e92225d7aa5adb70e895691ccc0ae05f6bd9ec"
Sep 17 22:25:31.971842 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:31.971805    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"]
Sep 17 22:25:46.991592 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:46.991538    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw" podUID="c71bea14-44ee-45ef-81fb-d8fbbcae9099" containerName="main" containerID="cri-o://c7e795390593893d23babfed8faae45950eb7790240e9436d09a96ec576fb476" gracePeriod=45
Sep 17 22:25:47.246673 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:47.246608    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:47.281794 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:47.281764    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71bea14-44ee-45ef-81fb-d8fbbcae9099-tmp-dir\") pod \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\" (UID: \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\") "
Sep 17 22:25:47.281968 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:47.281809    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c71bea14-44ee-45ef-81fb-d8fbbcae9099-tls-certs\") pod \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\" (UID: \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\") "
Sep 17 22:25:47.281968 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:47.281857    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71bea14-44ee-45ef-81fb-d8fbbcae9099-dshm\") pod \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\" (UID: \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\") "
Sep 17 22:25:47.281968 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:47.281878    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkm99\" (UniqueName: \"kubernetes.io/projected/c71bea14-44ee-45ef-81fb-d8fbbcae9099-kube-api-access-gkm99\") pod \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\" (UID: \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\") "
Sep 17 22:25:47.281968 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:47.281902    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71bea14-44ee-45ef-81fb-d8fbbcae9099-model-cache\") pod \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\" (UID: \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\") "
Sep 17 22:25:47.282311 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:47.281984    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c71bea14-44ee-45ef-81fb-d8fbbcae9099-home\") pod \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\" (UID: \"c71bea14-44ee-45ef-81fb-d8fbbcae9099\") "
Sep 17 22:25:47.282311 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:47.282070    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71bea14-44ee-45ef-81fb-d8fbbcae9099-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c71bea14-44ee-45ef-81fb-d8fbbcae9099" (UID: "c71bea14-44ee-45ef-81fb-d8fbbcae9099"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:25:47.282311 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:47.282300    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71bea14-44ee-45ef-81fb-d8fbbcae9099-tmp-dir\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:25:47.282452 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:47.282291    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71bea14-44ee-45ef-81fb-d8fbbcae9099-model-cache" (OuterVolumeSpecName: "model-cache") pod "c71bea14-44ee-45ef-81fb-d8fbbcae9099" (UID: "c71bea14-44ee-45ef-81fb-d8fbbcae9099"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:25:47.282452 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:47.282320    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71bea14-44ee-45ef-81fb-d8fbbcae9099-home" (OuterVolumeSpecName: "home") pod "c71bea14-44ee-45ef-81fb-d8fbbcae9099" (UID: "c71bea14-44ee-45ef-81fb-d8fbbcae9099"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:25:47.284287 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:47.284266    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71bea14-44ee-45ef-81fb-d8fbbcae9099-dshm" (OuterVolumeSpecName: "dshm") pod "c71bea14-44ee-45ef-81fb-d8fbbcae9099" (UID: "c71bea14-44ee-45ef-81fb-d8fbbcae9099"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:25:47.284375 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:47.284309    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71bea14-44ee-45ef-81fb-d8fbbcae9099-kube-api-access-gkm99" (OuterVolumeSpecName: "kube-api-access-gkm99") pod "c71bea14-44ee-45ef-81fb-d8fbbcae9099" (UID: "c71bea14-44ee-45ef-81fb-d8fbbcae9099"). InnerVolumeSpecName "kube-api-access-gkm99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:25:47.284466 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:47.284452    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c71bea14-44ee-45ef-81fb-d8fbbcae9099-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c71bea14-44ee-45ef-81fb-d8fbbcae9099" (UID: "c71bea14-44ee-45ef-81fb-d8fbbcae9099"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:25:47.383085 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:47.383045    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71bea14-44ee-45ef-81fb-d8fbbcae9099-dshm\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:25:47.383085 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:47.383078    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkm99\" (UniqueName: \"kubernetes.io/projected/c71bea14-44ee-45ef-81fb-d8fbbcae9099-kube-api-access-gkm99\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:25:47.383085 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:47.383088    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71bea14-44ee-45ef-81fb-d8fbbcae9099-model-cache\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:25:47.383085 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:47.383097    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c71bea14-44ee-45ef-81fb-d8fbbcae9099-home\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:25:47.383400 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:47.383105    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c71bea14-44ee-45ef-81fb-d8fbbcae9099-tls-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\""
Sep 17 22:25:48.057017 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:48.056984    2564 generic.go:356] "Generic (PLEG): container finished" podID="c71bea14-44ee-45ef-81fb-d8fbbcae9099" containerID="c7e795390593893d23babfed8faae45950eb7790240e9436d09a96ec576fb476" exitCode=0
Sep 17 22:25:48.057524 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:48.057026    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"
Sep 17 22:25:48.057524 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:48.057068    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw" event={"ID":"c71bea14-44ee-45ef-81fb-d8fbbcae9099","Type":"ContainerDied","Data":"c7e795390593893d23babfed8faae45950eb7790240e9436d09a96ec576fb476"}
Sep 17 22:25:48.057524 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:48.057116    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw" event={"ID":"c71bea14-44ee-45ef-81fb-d8fbbcae9099","Type":"ContainerDied","Data":"7b44e0bc1b2f545f1a5b2b2e76e408c3e70b704ed54c7d5d80175e2e9a24633b"}
Sep 17 22:25:48.057524 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:48.057138    2564 scope.go:117] "RemoveContainer" containerID="c7e795390593893d23babfed8faae45950eb7790240e9436d09a96ec576fb476"
Sep 17 22:25:48.066569 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:48.066546    2564 scope.go:117] "RemoveContainer" containerID="c7e795390593893d23babfed8faae45950eb7790240e9436d09a96ec576fb476"
Sep 17 22:25:48.066919 ip-10-0-129-67 kubenswrapper[2564]: E0917 22:25:48.066882    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7e795390593893d23babfed8faae45950eb7790240e9436d09a96ec576fb476\": container with ID starting with c7e795390593893d23babfed8faae45950eb7790240e9436d09a96ec576fb476 not found: ID does not exist" containerID="c7e795390593893d23babfed8faae45950eb7790240e9436d09a96ec576fb476"
Sep 17 22:25:48.066979 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:48.066928    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7e795390593893d23babfed8faae45950eb7790240e9436d09a96ec576fb476"} err="failed to get container status \"c7e795390593893d23babfed8faae45950eb7790240e9436d09a96ec576fb476\": rpc error: code = NotFound desc = could not find container \"c7e795390593893d23babfed8faae45950eb7790240e9436d09a96ec576fb476\": container with ID starting with c7e795390593893d23babfed8faae45950eb7790240e9436d09a96ec576fb476 not found: ID does not exist"
Sep 17 22:25:48.082213 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:48.082158    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"]
Sep 17 22:25:48.087689 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:48.087666    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-a0433cba-kserve-74f67c4899-7j2lw"]
Sep 17 22:25:48.754062 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:25:48.754024    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c71bea14-44ee-45ef-81fb-d8fbbcae9099" path="/var/lib/kubelet/pods/c71bea14-44ee-45ef-81fb-d8fbbcae9099/volumes"
Sep 17 22:26:24.891829 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:24.891718    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdcnw_6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2/ovn-acl-logging/0.log"
Sep 17 22:26:24.894555 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:24.894082    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdcnw_6ef4bf29-cd42-4cb6-8683-6dd3fdab58d2/ovn-acl-logging/0.log"
Sep 17 22:26:25.474968 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:25.474935    2564 scope.go:117] "RemoveContainer" containerID="1841b97303f994198d50683eb8a0ed8783beb660c3baa8828e21f82269a64ede"
Sep 17 22:26:25.483232 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:25.483207    2564 scope.go:117] "RemoveContainer" containerID="728578aa97acf5bf286744eb5a1b8df5e05a63f118bb6aa46bb903372bcb031b"
Sep 17 22:26:27.749887 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:27.749858    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pxwmt_63db50f1-321e-4505-b616-65e69af22944/manager/0.log"
Sep 17 22:26:27.874825 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:27.874793    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-jjkm7_bc00aec1-f082-4d1f-aef8-78b30b8672e8/manager/0.log"
Sep 17 22:26:28.402376 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:28.402349    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-555cdfd7b4-g9d79_9da019c5-6b53-4b87-ae3a-1fad457500d5/copy-manifests/0.log"
Sep 17 22:26:28.414844 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:28.414820    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-555cdfd7b4-g9d79_9da019c5-6b53-4b87-ae3a-1fad457500d5/manager/0.log"
Sep 17 22:26:29.294134 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:29.294098    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29828040-kxmw9_5e74d340-4b98-4849-a48d-fdc890e79751/cleanup/0.log"
Sep 17 22:26:30.451590 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:30.451561    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-5k9n9_62702f8b-12b0-4adb-9f3f-5a02cb1f31e1/manager/0.log"
Sep 17 22:26:30.577338 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:30.577309    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-sfj9k_307933ba-b117-4062-971b-765cc92e9a1c/manager/0.log"
Sep 17 22:26:30.820069 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:30.819982    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-zgmgg_59d57036-cd0c-40b5-afbd-a0ffa12eae1e/manager/0.log"
Sep 17 22:26:31.085520 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:31.085437    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-f6xcz_c1015b08-0358-412e-b5a1-0910a4de69f5/manager/0.log"
Sep 17 22:26:31.445069 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:31.445036    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj_70308246-88ae-41dd-ad39-6b2bd81f69c8/istio-proxy/0.log"
Sep 17 22:26:31.693043 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:31.693016    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f7577f47d-jfkrw_5169fe15-2965-44fe-888a-ff71d9fb1e1c/kube-auth-proxy/0.log"
Sep 17 22:26:31.932414 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:31.932382    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2h86h_7f47358c-f7a1-4166-9a6c-b723820bc434/istio-proxy/1.log"
Sep 17 22:26:32.056489 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:32.056458    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-sf4lg_1eadf925-d59d-45cf-8d67-e7538c47abd1/payload-pre-processing/0.log"
Sep 17 22:26:32.651428 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:32.651397    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-kcgvs_f178fb0f-cdee-4a0a-b436-1e0cf009c55d/main/0.log"
Sep 17 22:26:33.003601 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:33.003511    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-nfctf_f46e3e76-c17b-44ee-a908-4c096eb7996f/main/0.log"
Sep 17 22:26:33.238678 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:33.238651    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-sll6k_af2e15df-b573-4f4c-8d00-bd04763918d6/main/0.log"
Sep 17 22:26:39.330237 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:39.330191    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jmg5p/must-gather-p5d4d"]
Sep 17 22:26:39.330736 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:39.330549    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c71bea14-44ee-45ef-81fb-d8fbbcae9099" containerName="main"
Sep 17 22:26:39.330736 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:39.330560    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c71bea14-44ee-45ef-81fb-d8fbbcae9099" containerName="main"
Sep 17 22:26:39.330736 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:39.330628    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c71bea14-44ee-45ef-81fb-d8fbbcae9099" containerName="main"
Sep 17 22:26:39.333648 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:39.333631    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jmg5p/must-gather-p5d4d"
Sep 17 22:26:39.337481 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:39.337442    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jmg5p\"/\"openshift-service-ca.crt\""
Sep 17 22:26:39.338103 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:39.338087    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jmg5p\"/\"default-dockercfg-gg4dc\""
Sep 17 22:26:39.338633 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:39.338618    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jmg5p\"/\"kube-root-ca.crt\""
Sep 17 22:26:39.350072 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:39.350046    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jmg5p/must-gather-p5d4d"]
Sep 17 22:26:39.469251 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:39.469213    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c1b4f9ae-92ab-43da-b1db-dabd0c511285-must-gather-output\") pod \"must-gather-p5d4d\" (UID: \"c1b4f9ae-92ab-43da-b1db-dabd0c511285\") " pod="openshift-must-gather-jmg5p/must-gather-p5d4d"
Sep 17 22:26:39.469448 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:39.469301    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx25b\" (UniqueName: \"kubernetes.io/projected/c1b4f9ae-92ab-43da-b1db-dabd0c511285-kube-api-access-jx25b\") pod \"must-gather-p5d4d\" (UID: \"c1b4f9ae-92ab-43da-b1db-dabd0c511285\") " pod="openshift-must-gather-jmg5p/must-gather-p5d4d"
Sep 17 22:26:39.570764 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:39.570721    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c1b4f9ae-92ab-43da-b1db-dabd0c511285-must-gather-output\") pod \"must-gather-p5d4d\" (UID: \"c1b4f9ae-92ab-43da-b1db-dabd0c511285\") " pod="openshift-must-gather-jmg5p/must-gather-p5d4d"
Sep 17 22:26:39.570954 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:39.570780    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx25b\" (UniqueName: \"kubernetes.io/projected/c1b4f9ae-92ab-43da-b1db-dabd0c511285-kube-api-access-jx25b\") pod \"must-gather-p5d4d\" (UID: \"c1b4f9ae-92ab-43da-b1db-dabd0c511285\") " pod="openshift-must-gather-jmg5p/must-gather-p5d4d"
Sep 17 22:26:39.571158 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:39.571133    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c1b4f9ae-92ab-43da-b1db-dabd0c511285-must-gather-output\") pod \"must-gather-p5d4d\" (UID: \"c1b4f9ae-92ab-43da-b1db-dabd0c511285\") " pod="openshift-must-gather-jmg5p/must-gather-p5d4d"
Sep 17 22:26:39.581686 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:39.581620    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx25b\" (UniqueName: \"kubernetes.io/projected/c1b4f9ae-92ab-43da-b1db-dabd0c511285-kube-api-access-jx25b\") pod \"must-gather-p5d4d\" (UID: \"c1b4f9ae-92ab-43da-b1db-dabd0c511285\") " pod="openshift-must-gather-jmg5p/must-gather-p5d4d"
Sep 17 22:26:39.643750 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:39.643709    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jmg5p/must-gather-p5d4d"
Sep 17 22:26:39.928233 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:39.928202    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jmg5p/must-gather-p5d4d"]
Sep 17 22:26:39.931023 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:26:39.930998    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1b4f9ae_92ab_43da_b1db_dabd0c511285.slice/crio-7ea34b0c1280d9e9025a0b4fdd0cd2c212d28a1a7d11af3d5a8352dd2dfae29a WatchSource:0}: Error finding container 7ea34b0c1280d9e9025a0b4fdd0cd2c212d28a1a7d11af3d5a8352dd2dfae29a: Status 404 returned error can't find the container with id 7ea34b0c1280d9e9025a0b4fdd0cd2c212d28a1a7d11af3d5a8352dd2dfae29a
Sep 17 22:26:39.933424 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:39.933405    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:26:40.305114 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:40.305013    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jmg5p/must-gather-p5d4d" event={"ID":"c1b4f9ae-92ab-43da-b1db-dabd0c511285","Type":"ContainerStarted","Data":"7ea34b0c1280d9e9025a0b4fdd0cd2c212d28a1a7d11af3d5a8352dd2dfae29a"}
Sep 17 22:26:41.313302 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:41.313263    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jmg5p/must-gather-p5d4d" event={"ID":"c1b4f9ae-92ab-43da-b1db-dabd0c511285","Type":"ContainerStarted","Data":"e5a347097ffcf2de6ba34d58851c5630a315957a36c1fe7e423fdfed16d1b7a6"}
Sep 17 22:26:41.313778 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:41.313314    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jmg5p/must-gather-p5d4d" event={"ID":"c1b4f9ae-92ab-43da-b1db-dabd0c511285","Type":"ContainerStarted","Data":"7f7bd763f9895b52dbb3d9ee7fdd7fe265cdc152783b33155e9d4ae93bf9b07b"}
Sep 17 22:26:41.333056 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:41.332992    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jmg5p/must-gather-p5d4d" podStartSLOduration=1.648391965 podStartE2EDuration="2.332974413s" totalImagesPullingTime="684.582448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:26:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:26:41.330883448 +0000 UTC m=+2717.149209761" watchObservedRunningTime="2026-09-17 22:26:41.332974413 +0000 UTC m=+2717.151300726"
Sep 17 22:26:42.303549 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:42.303515    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hrbr6_1e76c503-ec71-47a9-b3cd-63d3b12a439a/global-pull-secret-syncer/0.log"
Sep 17 22:26:42.523909 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:42.523874    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7ngpc_d49ed8ac-69ff-4b6e-ad3a-3f45026b84ae/konnectivity-agent/0.log"
Sep 17 22:26:42.596612 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:42.596518    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-67.ec2.internal_c19c959590466136e44fe97bd5d57e26/haproxy/0.log"
Sep 17 22:26:47.488513 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:47.488474    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-5k9n9_62702f8b-12b0-4adb-9f3f-5a02cb1f31e1/manager/0.log"
Sep 17 22:26:47.535053 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:47.535017    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-sfj9k_307933ba-b117-4062-971b-765cc92e9a1c/manager/0.log"
Sep 17 22:26:47.617864 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:47.617831    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-zgmgg_59d57036-cd0c-40b5-afbd-a0ffa12eae1e/manager/0.log"
Sep 17 22:26:47.731521 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:47.731481    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-f6xcz_c1015b08-0358-412e-b5a1-0910a4de69f5/manager/0.log"
Sep 17 22:26:50.088637 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:50.088603    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zpttr_7aacc7a6-e805-4323-a14e-94514b3e1725/node-exporter/0.log"
Sep 17 22:26:50.117877 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:50.117781    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zpttr_7aacc7a6-e805-4323-a14e-94514b3e1725/kube-rbac-proxy/0.log"
Sep 17 22:26:50.152583 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:50.152557    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zpttr_7aacc7a6-e805-4323-a14e-94514b3e1725/init-textfile/0.log"
Sep 17 22:26:52.633764 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:52.633726    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46"]
Sep 17 22:26:52.640074 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:52.640043    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46"
Sep 17 22:26:52.647844 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:52.647806    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46"]
Sep 17 22:26:52.808084 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:52.808035    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/37e56238-ce01-4dac-81db-ef0fffdfddf4-proc\") pod \"perf-node-gather-daemonset-v7w46\" (UID: \"37e56238-ce01-4dac-81db-ef0fffdfddf4\") " pod="openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46"
Sep 17 22:26:52.808084 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:52.808088    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/37e56238-ce01-4dac-81db-ef0fffdfddf4-podres\") pod \"perf-node-gather-daemonset-v7w46\" (UID: \"37e56238-ce01-4dac-81db-ef0fffdfddf4\") " pod="openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46"
Sep 17 22:26:52.808371 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:52.808137    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6p66\" (UniqueName: \"kubernetes.io/projected/37e56238-ce01-4dac-81db-ef0fffdfddf4-kube-api-access-h6p66\") pod \"perf-node-gather-daemonset-v7w46\" (UID: \"37e56238-ce01-4dac-81db-ef0fffdfddf4\") " pod="openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46"
Sep 17 22:26:52.808371 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:52.808188    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37e56238-ce01-4dac-81db-ef0fffdfddf4-lib-modules\") pod \"perf-node-gather-daemonset-v7w46\" (UID: \"37e56238-ce01-4dac-81db-ef0fffdfddf4\") " pod="openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46"
Sep 17 22:26:52.808371 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:52.808334    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37e56238-ce01-4dac-81db-ef0fffdfddf4-sys\") pod \"perf-node-gather-daemonset-v7w46\" (UID: \"37e56238-ce01-4dac-81db-ef0fffdfddf4\") " pod="openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46"
Sep 17 22:26:52.908932 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:52.908894    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/37e56238-ce01-4dac-81db-ef0fffdfddf4-podres\") pod \"perf-node-gather-daemonset-v7w46\" (UID: \"37e56238-ce01-4dac-81db-ef0fffdfddf4\") " pod="openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46"
Sep 17 22:26:52.909136 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:52.908946    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6p66\" (UniqueName: \"kubernetes.io/projected/37e56238-ce01-4dac-81db-ef0fffdfddf4-kube-api-access-h6p66\") pod \"perf-node-gather-daemonset-v7w46\" (UID: \"37e56238-ce01-4dac-81db-ef0fffdfddf4\") " pod="openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46"
Sep 17 22:26:52.909136 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:52.908980    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37e56238-ce01-4dac-81db-ef0fffdfddf4-lib-modules\") pod \"perf-node-gather-daemonset-v7w46\" (UID: \"37e56238-ce01-4dac-81db-ef0fffdfddf4\") " pod="openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46"
Sep 17 22:26:52.909136 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:52.909052    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37e56238-ce01-4dac-81db-ef0fffdfddf4-sys\") pod \"perf-node-gather-daemonset-v7w46\" (UID: \"37e56238-ce01-4dac-81db-ef0fffdfddf4\") " pod="openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46"
Sep 17 22:26:52.909136 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:52.909077    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/37e56238-ce01-4dac-81db-ef0fffdfddf4-podres\") pod \"perf-node-gather-daemonset-v7w46\" (UID: \"37e56238-ce01-4dac-81db-ef0fffdfddf4\") " pod="openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46"
Sep 17 22:26:52.909136 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:52.909086    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/37e56238-ce01-4dac-81db-ef0fffdfddf4-proc\") pod \"perf-node-gather-daemonset-v7w46\" (UID: \"37e56238-ce01-4dac-81db-ef0fffdfddf4\") " pod="openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46"
Sep 17 22:26:52.909423 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:52.909139    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/37e56238-ce01-4dac-81db-ef0fffdfddf4-proc\") pod \"perf-node-gather-daemonset-v7w46\" (UID: \"37e56238-ce01-4dac-81db-ef0fffdfddf4\") " pod="openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46"
Sep 17 22:26:52.909423 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:52.909188    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37e56238-ce01-4dac-81db-ef0fffdfddf4-lib-modules\") pod \"perf-node-gather-daemonset-v7w46\" (UID: \"37e56238-ce01-4dac-81db-ef0fffdfddf4\") " pod="openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46"
Sep 17 22:26:52.909423 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:52.909206    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37e56238-ce01-4dac-81db-ef0fffdfddf4-sys\") pod \"perf-node-gather-daemonset-v7w46\" (UID: \"37e56238-ce01-4dac-81db-ef0fffdfddf4\") " pod="openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46"
Sep 17 22:26:52.920079 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:52.920048    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6p66\" (UniqueName: \"kubernetes.io/projected/37e56238-ce01-4dac-81db-ef0fffdfddf4-kube-api-access-h6p66\") pod \"perf-node-gather-daemonset-v7w46\" (UID: \"37e56238-ce01-4dac-81db-ef0fffdfddf4\") " pod="openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46"
Sep 17 22:26:52.953914 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:52.953867    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46"
Sep 17 22:26:53.251861 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:53.251834    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46"]
Sep 17 22:26:53.254290 ip-10-0-129-67 kubenswrapper[2564]: W0917 22:26:53.254238    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod37e56238_ce01_4dac_81db_ef0fffdfddf4.slice/crio-ca86881d2c6e2ece1bccd2f91d46205ddbf36f6a7eb8d30998f41eca12e177a2 WatchSource:0}: Error finding container ca86881d2c6e2ece1bccd2f91d46205ddbf36f6a7eb8d30998f41eca12e177a2: Status 404 returned error can't find the container with id ca86881d2c6e2ece1bccd2f91d46205ddbf36f6a7eb8d30998f41eca12e177a2
Sep 17 22:26:53.400016 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:53.399974    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46" event={"ID":"37e56238-ce01-4dac-81db-ef0fffdfddf4","Type":"ContainerStarted","Data":"ca86881d2c6e2ece1bccd2f91d46205ddbf36f6a7eb8d30998f41eca12e177a2"}
Sep 17 22:26:54.410305 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:54.410257    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46" event={"ID":"37e56238-ce01-4dac-81db-ef0fffdfddf4","Type":"ContainerStarted","Data":"6b447f516841d0ef0be90514348cf104e98eb816a654c60ec26808e6601eb360"}
Sep 17 22:26:54.411216 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:54.411144    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46"
Sep 17 22:26:54.432755 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:54.432686    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46" podStartSLOduration=2.432658898 podStartE2EDuration="2.432658898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 22:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:26:54.429447812 +0000 UTC m=+2730.247774126" watchObservedRunningTime="2026-09-17 22:26:54.432658898 +0000 UTC m=+2730.250985210"
Sep 17 22:26:55.636263 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:55.636232    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zht2d_3ae24773-6451-4cfd-a768-8548a684e09f/dns/0.log"
Sep 17 22:26:55.662379 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:55.662347    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zht2d_3ae24773-6451-4cfd-a768-8548a684e09f/kube-rbac-proxy/0.log"
Sep 17 22:26:55.719905 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:55.719881    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fdkhb_59115160-02ac-43e8-9a62-0c91775ba337/dns-node-resolver/0.log"
Sep 17 22:26:56.351332 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:56.351293    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5f88f88c4b-r97mr_5c38cf73-dc73-4436-bbc7-0bf58648950a/registry/0.log"
Sep 17 22:26:56.456291 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:56.456263    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rkgql_49f3497d-f458-44e9-8846-92afb8ba26eb/node-ca/0.log"
Sep 17 22:26:57.602420 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:57.602384    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj_70308246-88ae-41dd-ad39-6b2bd81f69c8/istio-proxy/0.log"
Sep 17 22:26:58.554030 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:58.553990    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f7577f47d-jfkrw_5169fe15-2965-44fe-888a-ff71d9fb1e1c/kube-auth-proxy/0.log"
Sep 17 22:26:58.625099 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:58.625072    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2h86h_7f47358c-f7a1-4166-9a6c-b723820bc434/istio-proxy/0.log"
Sep 17 22:26:58.628857 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:58.628827    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2h86h_7f47358c-f7a1-4166-9a6c-b723820bc434/istio-proxy/1.log"
Sep 17 22:26:58.692222 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:58.692193    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-sf4lg_1eadf925-d59d-45cf-8d67-e7538c47abd1/payload-pre-processing/0.log"
Sep 17 22:26:59.850680 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:26:59.850651    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hts6f_d38d2d6b-ead0-4052-ae97-d23439d4b9c9/serve-healthcheck-canary/0.log"
Sep 17 22:27:00.485135 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:27:00.485108    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sdnkh_87a36c1f-0c4f-4875-8d1e-f562ac1a19a2/kube-rbac-proxy/0.log"
Sep 17 22:27:00.511064 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:27:00.511035    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sdnkh_87a36c1f-0c4f-4875-8d1e-f562ac1a19a2/exporter/0.log"
Sep 17 22:27:00.540006 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:27:00.539981    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sdnkh_87a36c1f-0c4f-4875-8d1e-f562ac1a19a2/extractor/0.log"
Sep 17 22:27:01.434144 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:27:01.434117    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jmg5p/perf-node-gather-daemonset-v7w46"
Sep 17 22:27:03.419520 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:27:03.419486    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pxwmt_63db50f1-321e-4505-b616-65e69af22944/manager/0.log"
Sep 17 22:27:03.657200 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:27:03.656698    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj_70308246-88ae-41dd-ad39-6b2bd81f69c8/istio-proxy/0.log"
Sep 17 22:27:03.822878 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:27:03.822825    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-jjkm7_bc00aec1-f082-4d1f-aef8-78b30b8672e8/manager/0.log"
Sep 17 22:27:04.225136 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:27:04.225108    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-555cdfd7b4-g9d79_9da019c5-6b53-4b87-ae3a-1fad457500d5/manager/0.log"
Sep 17 22:27:04.276875 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:27:04.276840    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-555cdfd7b4-g9d79_9da019c5-6b53-4b87-ae3a-1fad457500d5/copy-manifests/0.log"
Sep 17 22:27:04.563401 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:27:04.563271    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f7577f47d-jfkrw_5169fe15-2965-44fe-888a-ff71d9fb1e1c/kube-auth-proxy/0.log"
Sep 17 22:27:04.612698 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:27:04.612663    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2h86h_7f47358c-f7a1-4166-9a6c-b723820bc434/istio-proxy/0.log"
Sep 17 22:27:04.614732 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:27:04.614706    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2h86h_7f47358c-f7a1-4166-9a6c-b723820bc434/istio-proxy/1.log"
Sep 17 22:27:04.642039 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:27:04.642011    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-sf4lg_1eadf925-d59d-45cf-8d67-e7538c47abd1/payload-pre-processing/0.log"
Sep 17 22:27:06.300183 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:27:06.300132    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-c8jfj_70308246-88ae-41dd-ad39-6b2bd81f69c8/istio-proxy/0.log"
Sep 17 22:27:07.175719 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:27:07.175690    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f7577f47d-jfkrw_5169fe15-2965-44fe-888a-ff71d9fb1e1c/kube-auth-proxy/0.log"
Sep 17 22:27:07.222582 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:27:07.222549    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2h86h_7f47358c-f7a1-4166-9a6c-b723820bc434/istio-proxy/0.log"
Sep 17 22:27:07.226251 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:27:07.226220    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2h86h_7f47358c-f7a1-4166-9a6c-b723820bc434/istio-proxy/1.log"
Sep 17 22:27:07.257219 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:27:07.257158    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-sf4lg_1eadf925-d59d-45cf-8d67-e7538c47abd1/payload-pre-processing/0.log"
Sep 17 22:27:11.376979 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:27:11.376946    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-z2x7n_5965f2cf-8d5f-4c57-bd7b-ba79842632d2/migrator/0.log"
Sep 17 22:27:11.404406 ip-10-0-129-67 kubenswrapper[2564]: I0917 22:27:11.404377    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-z2x7n_5965f2cf-8d5f-4c57-bd7b-ba79842632d2/graceful-termination/0.log"